zerodds-endpoint-python 1.0 — Spec Coverage
Source: docs/specs/zerodds-endpoint-python-1.0.md — the ZeroDDS Python
endpoint SDK spec. The marshalling comes from crates/lang-rules, column Language::Python; this document covers the transport.
Implementation:
endpoints/python/zerodds_endpoint.py— XRCE framing (xrce_write_frame/xrce_read_frame), serial framing after Annex C (serial_frame/serial_deframe), syncClient,MemTransport.endpoints/python/zerodds_reliable.py— reliable sender/receiver state machine + HEARTBEAT/ACKNACK wire codec +ReliableWriter.endpoints/python/example_async.py— the asyncioAsyncReaderpattern (§3).crates/endpoint-e2e/tests/python.rs— ping-pong E2E;crates/endpoint-e2e/tests/python_reliable.rs— reliable-stream E2E.
§1 XRCE framing
Spec: §1 — 8-byte XRCE header (session, stream, seq LE, submsg id 0x07
WRITE_DATA/0x09 DATA, flags, len LE) + body, byte-identical to
crates/xrce + endpoints/c; plus serial framing after Annex C.1
(addresses, length, byte-stuffing, FCS-16 of RFC 1662).
Repo: endpoints/python/zerodds_endpoint.py —
xrce_write_frame/xrce_read_frame, fcs16,
serial_frame/serial_deframe, constants XRCE_SESSION_NOKEY (0x80),
XRCE_STREAM_BEST_EFFORT (0x01), XRCE_STREAM_NONE (0x00).
Tests: endpoints/python/test_endpoint.py (python3 test_endpoint.py <golden_dir>) — WRITE_DATA framing, serial framing, DATA receive, serial
deframe round-trip, refusal of a wrong frameCheck and of a 0x7E inside
the frame, HEARTBEAT parse, ACKNACK framing, each checked against
golden_xrce_le.bin/golden_serial_le.bin/golden_data_le.bin/
golden_heartbeat_le.bin/golden_acknack_le.bin, reports ALL OK. The
framing itself is additionally exercised live via
python_endpoint_sync/python_endpoint_async (§4).
Status: done.
§2 Sync Client
Spec: §2 — non-blocking Client: write frames + delivers via the
transport, poll is a single non-blocking receive.
Repo: endpoints/python/zerodds_endpoint.py::Client
(__init__(transport, session, stream)/write/poll, monotonic seq
counter mod 2¹⁶, default session XRCE_SESSION_NOKEY/
XRCE_STREAM_BEST_EFFORT); MemTransport (in-memory FIFO,
deliver/receive) as the reference transport; a transport is pure
duck-typing (deliver(frame)/receive() -> bytes|None), not a formal
interface like Go’s Transport.
Tests: endpoints/python/example_sync.py — 5 typed
Reading(id, value, label) samples via Client.write/.poll(), full field
decode, reports ALL OK; live E2E python_endpoint_sync (§4).
Status: done.
§3 Async (asyncio receive pattern)
Spec: §3 — no dedicated SDK type (unlike Go’s AsyncReader/
AsyncWriter): an async def stream(self) generator polls
transport.receive() non-blockingly and yields the unframed body; the
consumer iterates with async for. No separate AsyncWriter — Client.write
serves sync and async alike since it is already non-blocking.
Repo: endpoints/python/example_async.py::AsyncReader.stream() — the
reference pattern over MemTransport; the same class (inline duplicated,
same semantics) in the E2E app
crates/endpoint-e2e/tests/python.rs::PY_APP::run_async over a real
UdpTransport.
Tests: endpoints/python/example_async.py — 5 Reading samples via
async for, full field decode, reports ALL OK; live E2E
python_endpoint_async (§4).
Status: done.
§4 Ping-pong E2E (live)
Spec: §5.1/§5.2 — a Python app exchanges a typed sample with the shared
Rust XRCE peer over a real UDP socket: once as raw generated codec
(.encode()/.decode()) with no XRCE frame, once as the full stack
(generated dataclasses + endpoint SDK) sync and async.
Repo: crates/endpoint-e2e/tests/python.rs — PY_APP (run_raw raw over
UDP with no XRCE frame, uses only Ping.encode()/Pong.decode(); run_sync
via ze.Client; run_async via the asyncio pattern from §3), mode selected
via sys.argv[1].
Tests: (Linux)
- python_raw_udp — generated Ping/Pong codec directly over a raw UDP
socket, no XRCE framing.
- python_endpoint_sync — full stack via ze.Client.
- python_endpoint_async — full stack via the asyncio receive pattern.
3/3 passed (Linux).
Status: done.
§5 Reliable stream — state machine, wire, async writer
Spec: §4 (references reliable-endpoint v1.0 §3/§4) — XRCE reliable
stream (stream_id 0x80, §8.4.10/§8.4.11), mirroring the reference
crates/xrce/src/reliable.rs: ReliableSender.submit/pending_heartbeat/
recv_acknack/get_in_flight; ReliableReceiver.recv_data/
drain_in_order/pending_acknack/reset. Window 16, receiver buffer 64,
heartbeat 500 ms, payload ≤ 65535, RFC-1982 16-bit sequence numbers
(seq_lt/seq_gt). Alongside it, the async-decoupled ReliableWriter: the
producer enqueue()s onto a queue.Queue, a dedicated drain
threading.Thread holds the ReliableSender state and does all the I/O
(send, heartbeat, ACKNACK-driven retransmit).
Honest note: ReliableWriter.enqueue() is not a wait-free ring-buffer
push like in Rust/C — it is a lock-protected queue.Queue.put(). What is
real is the I/O decoupling: the GIL is released around the drain thread’s
blocking socket.send/recv calls, so enqueue() never blocks on a
syscall — thread + GIL-release-around-syscalls decoupling, not a lock-free
data plane (spec §4 honest note).
Repo: endpoints/python/zerodds_reliable.py —
reliable_write_frame/reliable_unframe, heartbeat_frame/
parse_heartbeat, acknack_frame/parse_acknack; ReliableSender,
ReliableReceiver; ReliableWriter (queue.Queue-based,
enqueue/start/close, drain threading.Thread);
endpoints/python/example_reliable.py (in-process demo with a lossy
receiver thread, UDP sender mode run, latency bench mode bench).
Tests: (Linux)
- python_reliable_unit_and_golden
(crates/endpoint-e2e/tests/python_reliable.rs) runs
endpoints/python/reliable_test.py against goldens generated via
zerodds-xrce — 21 checks (check() calls) across 13 test functions:
monotonic seq + in-flight count (test_submit_assigns_monotonic_seqnrs: 3
checks), payload-too-large (test_submit_rejects_payload_too_large),
window-full (test_submit_rejects_when_window_full), heartbeat
body/silenced/fires-after/none-when-empty (test_pending_heartbeat: 4
checks), ACKNACK partial/full clear
(test_recv_acknack_clears_acked_keeps_missing/
test_recv_acknack_full_clear_when_no_bits_set), receiver
in-order/reorder/dedup/buffer-full (test_recv_data_buffers_in_order/
test_recv_data_reorders_out_of_order: 2 checks/
test_recv_data_drops_duplicates/test_recv_data_rejects_when_buffer_full),
pending-ACKNACK bitmap (test_pending_acknack_marks_missing_slots), reset
(test_reset_clears_state), in-process end-to-end loss recovery
(test_end_to_end_sender_receiver_with_loss_recovery: 3 checks), plus
conditionally (when a golden dir is passed) byte-golden test_byte_golden:
heartbeat_frame(1,1,3,0x80) == the golden_heartbeat_le.bin generated by
zerodds-xrce, acknack_frame(1,1,0,0x80) == the generated
golden_acknack_le.bin, plus round-trip parse — reports ALL OK.
- python_reliable_loss_recovery — peer drops every 3rd sample once; the app
(example_reliable.py run) retransmits on ACKNACK via the ReliableWriter;
all 12 samples delivered gap-free in order.
- python_reliable_no_loss — lossless baseline; 12/12.
- python_reliable_producer_latency — latency bench (§6).
4/4 passed (Linux), 3 of which belong to this section (latency bench in §6).
Status: done.
§6 Latency — enqueue() vs. inline socket.send
Spec: §5.4 — the producer path of ReliableWriter (enqueue() →
queue.Queue.put()) must be measurably below the inline socket.send
syscall — the evidence that syscall latency is decoupled from the producer
path (spec §4 honest note: thread + GIL release, not a wait-free ring).
Repo: endpoints/python/example_reliable.py::run_bench — 20000
iterations of inline sock.send(reliable_write_frame(...)) vs. 20000
iterations of ReliableWriter.enqueue(sample) against an idle-draining sink
socket (no live peer needed, only local dispatch cost under measurement).
Tests: (Linux) python_reliable_producer_latency
(crates/endpoint-e2e/tests/python_reliable.rs) — enqueue 704 ns vs.
inline send 4108 ns (~5.8×).
Honest note: the factor is well below Go’s ~175–220× and Rust’s/C’s
wait-free-ring numbers — expected, since enqueue() itself is already a
queue.Queue.put() (lock + internal deque append, not a bare memory store)
and CPython interpreter overhead dominates both sides of the measurement.
The number evidences decoupling from the syscall, not wait-freedom.
Status: done.
Audit-Status
6 done / 0 partial / 0 open / 0 blocked / 0 n/a (informative).
Test run (Linux, verified): cargo test -p zerodds-endpoint-e2e --test python 3/3 (ping-pong: python_raw_udp/python_endpoint_sync/
python_endpoint_async); --test python_reliable 4/4
(python_reliable_loss_recovery/python_reliable_no_loss/
python_reliable_unit_and_golden — 21 Python unit checks incl. byte-golden/
python_reliable_producer_latency); latency bench enqueue 704 ns / inline
send 4108 ns (~5.8×) — thread + GIL-release decoupling, not a wait-free
ring (see honest note §5/§6).
Open items: none.
zerodds-endpoint-python 1.0 — Spec-Coverage
Quelle: docs/specs/zerodds-endpoint-python-1.0.md — ZeroDDS Python
Endpoint-SDK-Spec. Das Marshalling kommt aus crates/lang-rules, Spalte Language::Python; hier steht der Transport.
Implementation:
endpoints/python/zerodds_endpoint.py— XRCE-Framing (xrce_write_frame/xrce_read_frame), serielles Framing nach Annex C (serial_frame/serial_deframe), syncClient,MemTransport.endpoints/python/zerodds_reliable.py— reliable Sender/Receiver- State-Machine + HEARTBEAT/ACKNACK-Wire-Codec +ReliableWriter.endpoints/python/example_async.py— asyncio-AsyncReader-Muster (§3).crates/endpoint-e2e/tests/python.rs— Ping-Pong-E2E;crates/endpoint-e2e/tests/python_reliable.rs— reliable-Stream-E2E.
§1 XRCE-Framing
Spec: §1 — 8-Byte-XRCE-Header (session, stream, seq LE, submsg id 0x07
WRITE_DATA/0x09 DATA, flags, len LE) + Body, byte-identisch zu
crates/xrce + endpoints/c; dazu serielles Framing nach Annex C.1
(Adressen, Länge, Byte-Stuffing, FCS-16 nach RFC 1662).
Repo: endpoints/python/zerodds_endpoint.py —
xrce_write_frame/xrce_read_frame, fcs16,
serial_frame/serial_deframe, Konstanten XRCE_SESSION_NOKEY (0x80),
XRCE_STREAM_BEST_EFFORT (0x01), XRCE_STREAM_NONE (0x00).
Tests: endpoints/python/test_endpoint.py (python3 test_endpoint.py <golden_dir>) — WRITE_DATA-Framing, Serial-Framing, DATA-Receive,
Serial-Deframe-Roundtrip, Abweisung eines falschen frameCheck und eines
0x7E im Rahmen, HEARTBEAT-Parse, ACKNACK-Framing, je gegen
golden_xrce_le.bin/golden_serial_le.bin/golden_data_le.bin/
golden_heartbeat_le.bin/golden_acknack_le.bin, meldet ALL OK. Framing
selbst wird zusätzlich über python_endpoint_sync/python_endpoint_async
(§4) live geübt.
Status: done.
§2 Sync Client
Spec: §2 — blockierungsfreier Client: write framet + liefert über
den Transport, poll ist ein nicht-blockierender Einzel-Receive.
Repo: endpoints/python/zerodds_endpoint.py::Client
(__init__(transport, session, stream)/write/poll, monotoner
seq-Zähler mod 2¹⁶, Default-Session XRCE_SESSION_NOKEY/
XRCE_STREAM_BEST_EFFORT); MemTransport (In-Memory-FIFO,
deliver/receive) als Referenz-Transport; ein Transport ist reines
Duck-Typing (deliver(frame)/receive() -> bytes|None), kein formales
Interface wie Gos Transport.
Tests: endpoints/python/example_sync.py — 5 typisierte
Reading(id, value, label)-Samples über Client.write/.poll(), voller
Feld-Decode, meldet ALL OK; Live-E2E python_endpoint_sync (§4).
Status: done.
§3 Async (asyncio-Empfangsmuster)
Spec: §3 — kein eigener SDK-Typ (anders als Gos AsyncReader/
AsyncWriter): ein async def stream(self)-Generator pollt
transport.receive() nicht-blockierend und yieldet den entrahmten Body;
Consumer iteriert mit async for. Kein separater AsyncWriter — Client.write
bedient sync wie async, da es bereits nicht-blockierend ist.
Repo: endpoints/python/example_async.py::AsyncReader.stream() — das
Referenzmuster über MemTransport; dieselbe Klasse (inline dupliziert, gleiche
Semantik) in der E2E-App crates/endpoint-e2e/tests/python.rs::PY_APP::run_async
über einen echten UdpTransport.
Tests: endpoints/python/example_async.py — 5 Reading-Samples über
async for, voller Feld-Decode, meldet ALL OK; Live-E2E
python_endpoint_async (§4).
Status: done.
§4 Ping-Pong-E2E (live)
Spec: §5.1/§5.2 — eine Python-App tauscht mit dem geteilten Rust-XRCE-Peer
über einen echten UDP-Socket ein typisiertes Sample aus: einmal roher
generierter Codec (.encode()/.decode()) ohne XRCE-Frame, einmal voller
Stack (generierte Dataclasses + Endpoint-SDK) sync und async.
Repo: crates/endpoint-e2e/tests/python.rs — PY_APP (run_raw roh
über UDP ohne XRCE-Frame, nutzt nur Ping.encode()/Pong.decode();
run_sync über ze.Client; run_async über das asyncio-Muster aus §3),
Modus per sys.argv[1].
Tests: (Linux)
- python_raw_udp — generierter Ping/Pong-Codec direkt über einen rohen
UDP-Socket, ohne XRCE-Framing.
- python_endpoint_sync — voller Stack über ze.Client.
- python_endpoint_async — voller Stack über das asyncio-Empfangsmuster.
3/3 grün (Linux).
Status: done.
§5 Reliable Stream — State-Machine, Wire, Async-Writer
Spec: §4 (verweist auf reliable-endpoint v1.0 §3/§4) — XRCE reliable
Stream (stream_id 0x80, §8.4.10/§8.4.11), spiegelt die Referenz
crates/xrce/src/reliable.rs: ReliableSender.submit/pending_heartbeat/
recv_acknack/get_in_flight; ReliableReceiver.recv_data/
drain_in_order/pending_acknack/reset. Window 16, Receiver-Buffer 64,
Heartbeat 500 ms, Payload ≤ 65535, RFC-1982 16-bit Sequenznummern
(seq_lt/seq_gt). Dazu der async-entkoppelte ReliableWriter: der
Producer enqueue()t in eine queue.Queue, eine dedizierte Drain-
threading.Thread hält den ReliableSender-State und macht die gesamte I/O
(Senden, Heartbeat, ACKNACK-getriebenes Retransmit).
Honest note: ReliableWriter.enqueue() ist kein wait-free
Ringpuffer-Push wie in Rust/C, sondern ein lock-geschützter
queue.Queue.put(). Real ist die I/O-Entkopplung: der GIL wird um die
blockierenden socket.send/recv-Aufrufe des Drain-Threads freigegeben, so
dass enqueue() nie auf einen Syscall wartet — Thread + GIL-Release-um-
Syscalls, keine lock-freie Datenebene (Spec §4, Honest-Note).
Repo: endpoints/python/zerodds_reliable.py —
reliable_write_frame/reliable_unframe, heartbeat_frame/
parse_heartbeat, acknack_frame/parse_acknack; ReliableSender,
ReliableReceiver; ReliableWriter (queue.Queue-basiert,
enqueue/start/close, Drain-threading.Thread);
endpoints/python/example_reliable.py (In-Process-Demo mit lossy
Receiver-Thread, UDP-Sender-Modus run, Latenz-Bench-Modus bench).
Tests: (Linux)
- python_reliable_unit_and_golden
(crates/endpoint-e2e/tests/python_reliable.rs) läuft
endpoints/python/reliable_test.py gegen von zerodds-xrce erzeugte
Goldens — 21 Prüfungen (check()-Aufrufe) über 13 Testfunktionen: monotone
seq + in-flight-Count (test_submit_assigns_monotonic_seqnrs: 3 Checks),
Payload-zu-groß (test_submit_rejects_payload_too_large), Window-full
(test_submit_rejects_when_window_full), Heartbeat
Body/silenced/fires-after/none-when-empty
(test_pending_heartbeat: 4 Checks), ACKNACK Teil-/Voll-Clear
(test_recv_acknack_clears_acked_keeps_missing/
test_recv_acknack_full_clear_when_no_bits_set), Receiver
in-order/Reorder/Dedup/Buffer-full (test_recv_data_buffers_in_order/
test_recv_data_reorders_out_of_order: 2 Checks/
test_recv_data_drops_duplicates/test_recv_data_rejects_when_buffer_full),
Pending-ACKNACK-Bitmap (test_pending_acknack_marks_missing_slots), Reset
(test_reset_clears_state), In-Process-End-to-End-Loss-Recovery
(test_end_to_end_sender_receiver_with_loss_recovery: 3 Checks), plus
bedingt (Golden-Dir übergeben) Byte-Golden test_byte_golden:
heartbeat_frame(1,1,3,0x80) == von zerodds-xrce erzeugtem
golden_heartbeat_le.bin, acknack_frame(1,1,0,0x80) == erzeugtem
golden_acknack_le.bin, plus Roundtrip-Parse — meldet ALL OK.
- python_reliable_loss_recovery — Peer dropt jedes 3. Sample einmalig; die
App (example_reliable.py run) retransmittet auf ACKNACK über den
ReliableWriter; alle 12 Samples lückenlos in Reihenfolge geliefert.
- python_reliable_no_loss — lossless Baseline; 12/12.
- python_reliable_producer_latency — Latenz-Bench (§6).
4/4 grün (Linux), davon 3 in diesem Abschnitt (Latenz-Bench in §6).
Status: done.
§6 Latenz — enqueue() vs. inline socket.send
Spec: §5.4 — der Producer-Pfad des ReliableWriter (enqueue() →
queue.Queue.put()) muss messbar unter dem inline socket.send-Syscall
liegen — der Beleg, dass die Syscall-Latenz aus dem Producer-Pfad entkoppelt
ist (Spec §4 Honest-Note: Thread+GIL-Release, keine wait-free Ring).
Repo: endpoints/python/example_reliable.py::run_bench — 20000
Iterationen inline sock.send(reliable_write_frame(...)) vs. 20000
Iterationen ReliableWriter.enqueue(sample) gegen einen idle-drainenden
Sink-Socket (kein Live-Peer nötig, nur lokale Dispatch-Kosten unter Messung).
Tests: (Linux) python_reliable_producer_latency
(crates/endpoint-e2e/tests/python_reliable.rs) — Enqueue 704 ns vs.
inline send 4108 ns (~5,8×).
Honest note: der Faktor liegt deutlich unter Gos ~175–220× und Rusts/Cs
wait-free-Ring-Werten — erwartbar, weil enqueue() selbst schon ein
queue.Queue.put() (Lock + interner Deque-Append, kein reiner
Speicher-Store) ist und CPython-Interpreter-Overhead beide Seiten der Messung
dominiert. Der Messwert belegt die Entkopplung vom Syscall, nicht
Wait-Freedom.
Status: done.
Audit-Status
6 done / 0 partial / 0 open / 0 blocked / 0 n/a (informative).
Test-Lauf: Linux, verifiziert — cargo test -p zerodds-endpoint-e2e --test python 3/3 (Ping-Pong: python_raw_udp/python_endpoint_sync/
python_endpoint_async); --test python_reliable 4/4
(python_reliable_loss_recovery/python_reliable_no_loss/
python_reliable_unit_and_golden — 21 Python-Unit-Checks inkl. Byte-Golden/
python_reliable_producer_latency); Latenz-Bench Enqueue 704 ns / inline
send 4108 ns (~5,8×) — Thread+GIL-Release-Entkopplung, keine wait-free
Ring (siehe Honest-Note §5/§6).
Offene Punkte: keine.