Back
+20 XP
Concept

Memory-Mapped Files with mmap

Standard file `read()` copies pages from kernel memory into user-space heap. Python's `mmap` module maps file descriptors directly into virtual address space, letting the OS page table stream disk blocks on demand with zero buffer copies ⚡💾.

python
import mmap
import tempfile
import os

# Create backed disk file:
with tempfile.NamedTemporaryFile(delete=False) as tmp:
    tmp.write(b"HELLO_WORLD_MMAP")
    tmp.flush()
    filepath = tmp.name

with open(filepath, "r+b") as f:
    # Map file into virtual address space:
    mm = mmap.mmap(f.fileno(), 0)
    print(f"Direct virtual read: {mm[:5].decode()} 🚀")

    # Direct in-place byte mutation:
    mm[6:11] = b"SUPER"
    mm.flush()
    mm.close()

os.remove(filepath)

# 46. Memory-Mapped Files with mmap When your database or search engine needs to scan a 50GB file, calling `f.read()` will either crash your server with an Out-Of-Memory exception or kill throughput by endlessly copying gigabytes between kernel space and Python heap memory 🛑. ```python import mmap with open("massive_database.bin", "r+b") as f: # 0 length maps the entire file into virtual memory: mm = mmap.mmap(f.fileno(), 0) # The OS loads only the specific 4KB page touched! header = mm[0:64] ``` ### The Zero-Copy Virtual Memory Mechanism 1. **OS Page Faulting**: Calling `mmap.mmap()` does not read the file into RAM. It assigns a range of virtual addresses. When Python accesses `mm[1000:1010]`, the CPU triggers a minor page fault, paging in that exact 4KB disk block ⚡. 2. **Zero User-Space Buffer Copy**: Bytes are read directly from the OS page cache without copying into Python `str` or heap buffers. 3. **Shared Memory (`MAP_SHARED`)**: Modifications written to the `mmap` slice are flushed back to the underlying disk file when calling `mm.flush()` or closing the map 💾. --- ### Interactive Pop-Quiz 🧠 #### What is the primary architectural performance advantage of mmap over standard file.read()? - [ ] It eliminates intermediate copying between the OS kernel page cache and user-space memory buffers ⚡ - [ ] It compresses file data using lz4 compression on the CPU - [ ] It prevents other processes from viewing file system permissions - [ ] It automatically formats JSON files into SQLite database tables --- ### Practical Challenge 🎯 Using a `NamedTemporaryFile(delete=False)`, write `b'RECORD_AAA_END'`. Open with `'r+b'`, create an `mmap`, slice `mm[7:10]` to read `'AAA'`, overwrite `mm[7:10] = b'BBB'`, flush and close `mm`. Read the file back, verify it equals `b'RECORD_BBB_END'`, remove the file, and print the decoded final content. <details> <summary>Solution</summary> ```python import mmap import tempfile import os with tempfile.NamedTemporaryFile(delete=False) as tmp: tmp.write(b"RECORD_AAA_END") tmp.flush() fname = tmp.name with open(fname, "r+b") as f: mm = mmap.mmap(f.fileno(), 0) assert mm[7:10] == b"AAA" mm[7:10] = b"BBB" mm.flush() mm.close() with open(fname, "rb") as f: data = f.read() os.remove(fname) assert data == b"RECORD_BBB_END" print(data.decode()) ``` </details>

Checkpoint

What is the primary architectural performance advantage of `mmap` over standard `file.read()`? 🧠

Practice

Using a `NamedTemporaryFile(delete=False)`, write `b'RECORD_AAA_END'`. Open with `'r+b'`, create an `mmap`, slice `mm[7:10]` to read `'AAA'`, overwrite `mm[7:10] = b'BBB'`, flush and close `mm`. Read the file back, verify it equals `b'RECORD_BBB_END'`, remove the file, and print the decoded final content. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks