Back
+35 XP
Concept

Module 43 Capstone: Binary RPC Framing & Dispatch Engine

Custom binary RPC frameworks achieve peak throughput by bypassing textual HTTP/JSON serialization. A high-performance binary RPC protocol pairs fixed-size headers, correlation IDs, method strings, and payload framing with dynamic dispatchers to execute remote procedure calls.

python
import struct

# 8-byte header: MAGIC (2B), TYPE (1B), CALL_ID (4B), METHOD_LEN (1B)
header = struct.pack('>HBIB', 0x5250, msg_type, call_id, len(method_bytes))

# Module 43 Capstone: Binary RPC Framing & Dispatch Engine When building real-time gaming engines, high-frequency trading platforms, or internal microservice meshes, JSON-over-HTTP introduces unacceptable latency and memory bloat. You need a native **Binary Remote Procedure Call (RPC)** protocol. 🏆⚡ A production binary RPC frame uses an 8-byte fixed header: - **`MAGIC` (2 bytes)**: Protocol identifier (`0x5250` = ASCII `"RP"`) - **`MSG_TYPE` (1 byte)**: `1` for Request, `2` for Response, `3` for Error - **`CALL_ID` (4 bytes)**: Big-endian `uint32` correlation ID for multiplexing - **`METHOD_LEN` (1 byte)**: Length of the method name string - Followed by `METHOD` bytes, a 4-byte `PAYLOAD_LEN`, and `PAYLOAD` bytes. ```python import struct HEADER_FORMAT = ">HBIB" # Exactly 8 bytes ``` ## Pop Quiz Why does a binary RPC protocol include a correlation ID (`CALL_ID`) in every frame header? - [ ] It enables multiplexing multiple concurrent requests and out-of-order responses over a single shared TCP socket. - [ ] It encrypts the payload using symmetric hardware cipher keys. - [ ] It instructs the network router to prioritize the packet on congested WiFi networks. - [ ] Because Python dictionaries require integer keys for hash lookups. ## Real-World Challenge Build a production `BinaryRPC` framing codec and `RPCDispatcher`: 1. `BinaryRPC.encode_request(call_id: int, method: str, payload: bytes) -> bytes` 2. `BinaryRPC.encode_response(call_id: int, payload: bytes) -> bytes` 3. `BinaryRPC.decode_frame(data: bytes) -> dict` 4. `RPCDispatcher`: Allows registering method callables via `register(method, fn)` and executing raw request frames via `dispatch(frame_bytes) -> bytes`. <details> <summary>Solution</summary> ```python import struct MAGIC_CONSTANT = 0x5250 HEADER_FORMAT = ">HBIB" # 2 + 1 + 4 + 1 = 8 bytes class BinaryRPC: REQ = 1 RES = 2 ERR = 3 @staticmethod def encode_request(call_id: int, method: str, payload: bytes) -> bytes: m_bytes = method.encode("ascii") if len(m_bytes) > 255: raise ValueError("Method name cannot exceed 255 bytes") header = struct.pack(HEADER_FORMAT, MAGIC_CONSTANT, BinaryRPC.REQ, call_id, len(m_bytes)) body = m_bytes + struct.pack(">I", len(payload)) + payload return header + body @staticmethod def encode_response(call_id: int, payload: bytes) -> bytes: header = struct.pack(HEADER_FORMAT, MAGIC_CONSTANT, BinaryRPC.RES, call_id, 0) body = struct.pack(">I", len(payload)) + payload return header + body @staticmethod def decode_frame(data: bytes) -> dict: header_size = struct.calcsize(HEADER_FORMAT) if len(data) < header_size: raise ValueError("Frame smaller than minimum 8-byte header") magic, mtype, call_id, mlen = struct.unpack_from(HEADER_FORMAT, data, 0) if magic != MAGIC_CONSTANT: raise ValueError(f"Invalid RPC magic: expected {hex(MAGIC_CONSTANT)}, got {hex(magic)}") offset = header_size method = "" if mlen > 0: method = data[offset : offset + mlen].decode("ascii") offset += mlen if offset + 4 > len(data): raise ValueError("Truncated payload length field") plen, = struct.unpack_from(">I", data, offset) offset += 4 payload = data[offset : offset + plen] return { "type": mtype, "call_id": call_id, "method": method, "payload": payload } class RPCDispatcher: def __init__(self): self._handlers = {} def register(self, method: str, handler_fn): self._handlers[method] = handler_fn def dispatch(self, request_frame_bytes: bytes) -> bytes: frame = BinaryRPC.decode_frame(request_frame_bytes) call_id = frame["call_id"] method = frame["method"] if method not in self._handlers: return BinaryRPC.encode_response(call_id, b"ERR:UnknownMethod") handler = self._handlers[method] result_payload = handler(frame["payload"]) return BinaryRPC.encode_response(call_id, result_payload) # Verification dispatcher = RPCDispatcher() dispatcher.register("compute_hash", lambda p: b"HASHED_" + p.upper()) # Client encodes request client_call_id = 4040 request_bytes = BinaryRPC.encode_request(client_call_id, "compute_hash", b"raw_payload_data") # Server dispatches and returns response response_bytes = dispatcher.dispatch(request_bytes) # Client decodes response resp_frame = BinaryRPC.decode_frame(response_bytes) assert resp_frame["type"] == BinaryRPC.RES assert resp_frame["call_id"] == client_call_id assert resp_frame["payload"] == b"HASHED_RAW_PAYLOAD_DATA" ``` </details>