Back
+30 XP
Concept

Profile Memory Churn with tracemalloc

Debugging gradual memory leaks or excessive allocation churn in production requires tracking exact source lines allocating heap memory. Python's built-in tracemalloc library captures snapshots and computes deltas to pinpoint memory bloat down to the file and line number.

python
import tracemalloc

tracemalloc.start()
snap1 = tracemalloc.take_snapshot()
# run workload
snap2 = tracemalloc.take_snapshot()
diffs = snap2.compare_to(snap1, 'lineno')
for stat in diffs[:3]:
    print(stat)

# Profile Memory Churn with tracemalloc When a customer reports that your API microservice starts at 80 MB of RAM but gradually creeps to 2.4 GB over four days, opening `top` or checking Kubernetes pod metrics only tells you *that* memory is increasing—not *where*. 🧠📉 Python’s built-in `tracemalloc` module hooks directly into Python’s memory allocators (`PyMem_Malloc`). By comparing two snapshots (`snap2.compare_to(snap1, 'lineno')`), it highlights the exact line of Python code responsible for allocating heap memory. ```python import tracemalloc tracemalloc.start() snapshot_before = tracemalloc.take_snapshot() # Code under scrutiny data = [str(x) * 10 for x in range(100000)] snapshot_after = tracemalloc.take_snapshot() top_stats = snapshot_after.compare_to(snapshot_before, "lineno") print("Top memory allocation line:", top_stats[0]) ``` ## Pop Quiz Why is `tracemalloc` superior to inspecting general OS process RSS memory with `psutil`? - [ ] It attributes Python heap allocations directly to source code file names and line numbers. - [ ] It automatically compacts virtual memory pages to eliminate swap file thrashing. - [ ] It prevents C extensions from allocating memory outside the Python runtime arena. - [ ] It doubles the maximum memory limit allowed by the Linux cgroups controller. ## Real-World Challenge Write `measure_memory_leak(fn, *args)` using `tracemalloc`. It must track memory allocations during `fn(*args)` execution, compute the net positive difference in bytes across line statistics, stop profiling, and return a tuple `(result, net_bytes_allocated)`. <details> <summary>Solution</summary> ```python import tracemalloc def measure_memory_leak(fn, *args): tracemalloc.start() snap_initial = tracemalloc.take_snapshot() result = fn(*args) snap_final = tracemalloc.take_snapshot() tracemalloc.stop() stats = snap_final.compare_to(snap_initial, "lineno") total_allocated_bytes = sum(stat.size_diff for stat in stats if stat.size_diff > 0) return result, total_allocated_bytes # Verification def allocate_dummy_records(count: int): return [{"id": i, "payload": "x" * 64} for i in range(count)] res, bytes_churned = measure_memory_leak(allocate_dummy_records, 2000) assert len(res) == 2000 assert bytes_churned > 50000, f"Expected > 50KB allocated, got {bytes_churned} bytes" ``` </details>