Back
+30 XP
Concept

Fixed-Header Packet Chunk Reassembler

TCP streams deliver arbitrary byte chunks rather than discrete application messages. A single read may contain a partial packet or multiple concatenated packets. Reassembling packets requires buffering incoming bytes and decoding fixed headers containing magic numbers and declared lengths.

python
import struct

# 4-byte header: MAGIC (2 bytes) + LENGTH (2 bytes)
magic, length = struct.unpack_from('>HH', buffer, 0)

# Fixed-Header Packet Chunk Reassembler A common rookie networking bug is writing `data = sock.recv(1024)` and assuming `data` is one complete message. In TCP, the sender can call `sock.sendall(msg)` once, but your receiver might get it split across 3 tiny chunks, or receive 4 messages glued together in one single chunk. 📦🔀 TCP is a **continuous byte stream**, not a message queue. To extract discrete application messages, protocols prepend a fixed-size header containing a magic constant and payload length. The receiver buffers incoming bytes until the full declared payload length has arrived. ```python import struct MAGIC = 0xABCD # Header: 2 bytes MAGIC + 2 bytes LENGTH magic, length = struct.unpack_from(">HH", buffer, 0) ``` ## Pop Quiz Why can an application layer never assume that one socket `recv()` call corresponds to exactly one sent message in TCP? - [ ] TCP is a continuous byte-stream protocol with no intrinsic packet framing; the OS fragments or merges bytes arbitrarily. - [ ] The Linux kernel drops packet boundary delimiters to save CPU cache space. - [ ] TCP network cards convert variable-length payloads into fixed 1500-byte UDP frames. - [ ] Because Python socket buffers automatically trim whitespace characters. ## Real-World Challenge Implement `StreamReassembler`. - Uses a 4-byte header: `MAGIC = 0xABCD` (uint16 big-endian) + `length` (uint16 big-endian). - Exposes `feed(chunk: bytes) -> list[bytes]`: Appends incoming bytes to an internal buffer, scans for the magic constant, extracts any complete packets, and returns their payloads. Incomplete packets remain buffered awaiting future chunks. <details> <summary>Solution</summary> ```python import struct class StreamReassembler: MAGIC = 0xABCD def __init__(self): self._buffer = bytearray() def feed(self, chunk: bytes) -> list[bytes]: self._buffer.extend(chunk) completed_packets = [] while len(self._buffer) >= 4: magic, length = struct.unpack_from(">HH", self._buffer, 0) # If magic does not match, advance 1 byte to resynchronize stream if magic != self.MAGIC: del self._buffer[0] continue # Check if full payload has arrived total_packet_size = 4 + length if len(self._buffer) < total_packet_size: break # Wait for subsequent chunks payload = bytes(self._buffer[4 : total_packet_size]) del self._buffer[:total_packet_size] completed_packets.append(payload) return completed_packets # Verification reassembler = StreamReassembler() p1 = struct.pack(">HH", 0xABCD, 4) + b"DATA" p2 = struct.pack(">HH", 0xABCD, 5) + b"EVENT" stream = p1 + p2 # Feed split across awkward boundaries out1 = reassembler.feed(stream[:3]) assert out1 == [] out2 = reassembler.feed(stream[3:10]) assert out2 == [b"DATA"] out3 = reassembler.feed(stream[10:]) assert out3 == [b"EVENT"] ``` </details>