Fixed-Width Binary Records & Memory-Mapped Offsets
Variable-length records require parsing every preceding byte. Fixed-width binary structs (`struct.Struct`) enforce a constant byte width $S$. Seeking to row $i$ reduces to trivial math: $\text{offset} = i \times S$, granting instantaneous $O(1)$ disk reads inside `mmap` 📐⚡.
import struct
import mmap
import tempfile
import os
# 16-byte fixed struct: uint32 (4B) + float32 (4B) + 8-byte char (8B)
REC = struct.Struct("!If8s")
with tempfile.NamedTemporaryFile(delete=False) as tmp:
for i in range(3):
tmp.write(REC.pack(i + 1, 80.0 + i, f"U{i}".encode().ljust(8, b"\x00")))
tmp.flush()
filepath = tmp.name
with open(filepath, "rb") as f:
mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
# O(1) jump to index 1:
uid, score, name = REC.unpack_from(mm, 1 * REC.size)
print(f"Record 1: ID={uid}, Name={name.rstrip(b'\x00').decode()} 🚀")
mm.close()
os.remove(filepath)
# 47. Fixed-Width Binary Records & Memory-Mapped Offsets Reading CSV or JSON files forces your CPU to scan every single byte from byte 0, searching for newlines and quotes just to find row 100,000. Under production workloads, this kills performance 🛑. ```python # Fixed width calculation: # Offset = row_index * struct_byte_size # Row 500,000 in a 16-byte record table = offset 8,000,000 # One memory-pointer seek -> Done in 0.000001ms! ``` ### Binary Serialization Architecture 1. **Predictable Memory Footprint**: Every row consumes the exact same physical byte allocation. Unused string space is padded with null bytes (`\x00`). 2. **`unpack_from()`**: Rather than slicing a new `bytes` object (which creates a new heap allocation), `struct.Struct.unpack_from(buffer, offset)` parses directly from the underlying `mmap` or `memoryview` without copying 🏎️. 3. **No Index Table Needed**: If rows are sequential IDs, the record offset is itself the index, eliminating the memory overhead of a separate primary key index. --- ### Interactive Pop-Quiz 🧠 #### Why does fixed-width binary record sizing enable O(1) random row lookups in a multi-gigabyte table? - [ ] The exact byte offset of any record $i$ is calculated directly via $i \times \text{record\_size}$ without scanning prior rows 📐 - [ ] Because fixed-width records bypass CPU L1 instruction cache entirely - [ ] Because the Linux kernel locks memory rows into registers - [ ] Because fixed-width records force the file to be stored on SSDs --- ### Practical Challenge 🎯 Define a `struct.Struct('!I12s')` (16 bytes: uint32 id, 12-byte padded name). Write 3 records to a temp file with IDs 10, 20, 30 and names `'alice'`, `'bob'`, `'charlie'`. Use `mmap` to read index 2 directly (offset `2 * size`), extract ID and cleaned name, remove the file, and print `f'{uid}:{clean_name}'`. <details> <summary>Solution</summary> ```python import struct import mmap import tempfile import os fmt = struct.Struct("!I12s") records = [(10, b"alice"), (20, b"bob"), (30, b"charlie")] with tempfile.NamedTemporaryFile(delete=False) as tmp: for uid, name in records: tmp.write(fmt.pack(uid, name.ljust(12, b"\x00"))) tmp.flush() fname = tmp.name with open(fname, "rb") as f: mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) uid, raw_name = fmt.unpack_from(mm, 2 * fmt.size) clean_name = raw_name.rstrip(b"\x00").decode() mm.close() os.remove(fname) print(f"{uid}:{clean_name}") ``` </details>