Back
+30 XP
Concept

Type-Length-Value (TLV) Binary Stream Parser

Binary protocols prefer Type-Length-Value (TLV) encoding over JSON for speed and compact size. Each frame specifies a 1-byte tag, a 2-byte big-endian length, and raw payload bytes. Parsing streams requires strict boundary checking to prevent buffer over-reads.

python
import struct

tag, length = struct.unpack_from('>BH', data, offset)
value = data[offset+3 : offset+3+length]

# Type-Length-Value (TLV) Binary Stream Parser When you stream 500,000 market order updates or Bluetooth LE beacons every second, formatting records as JSON (`{"type": 1, "len": 4, "val": "..."}`) consumes 10x the bandwidth and wastes CPU cycles parsing curly braces and escaping quotes. 🏎️📦 The **Type-Length-Value (TLV)** format is the backbone of high-performance binary networking (including TLS extensions, GSM, and LDAP). Each record consists of: 1. **Type / Tag** (e.g. 1 byte unsigned integer) 2. **Length** (e.g. 2 bytes unsigned big-endian integer `>H`) 3. **Value** (exactly `length` raw payload bytes) ```python import struct tag, length = struct.unpack_from(">BH", buffer, offset) value = buffer[offset + 3 : offset + 3 + length] ``` ## Pop Quiz Why do low-latency trading and telecommunication networks standardize on TLV rather than JSON or XML? - [ ] TLV allows zero-copy slicing and constant-time length lookups without costly string parsing or character escaping. - [ ] JSON payloads cannot represent integers greater than 255 across TCP networks. - [ ] TLV packets bypass IP routing tables to travel directly through optical fiber switches. - [ ] Because operating system kernels only expose binary serialization drivers. ## Real-World Challenge Implement `parse_tlv_stream(data: bytes) -> list[tuple[int, bytes]]`. It must parse sequential records formatted as `[Tag: uint8, Length: uint16-be, Value: bytes]`. If a record's header is truncated or if the buffer ends before reading the full declared payload length, raise `ValueError`. <details> <summary>Solution</summary> ```python import struct def parse_tlv_stream(data: bytes) -> list[tuple[int, bytes]]: records = [] offset = 0 total_len = len(data) while offset < total_len: if offset + 3 > total_len: raise ValueError(f"Truncated TLV header at offset {offset}") tag, length = struct.unpack_from(">BH", data, offset) offset += 3 if offset + length > total_len: raise ValueError(f"Truncated TLV payload at offset {offset}: expected {length} bytes") payload = data[offset : offset + length] offset += length records.append((tag, payload)) return records # Verification valid_stream = ( struct.pack(">BH", 0x01, 4) + b"PING" + struct.pack(">BH", 0x02, 5) + b"HELLO" + struct.pack(">BH", 0xFF, 0) ) parsed = parse_tlv_stream(valid_stream) assert parsed == [ (1, b"PING"), (2, b"HELLO"), (255, b"") ] # Error handling test truncated_stream = struct.pack(">BH", 0x01, 10) + b"SHORT" has_error = False try: parse_tlv_stream(truncated_stream) except ValueError: has_error = True assert has_error is True ``` </details>