Subinterpreters & Per-Interpreter GIL: PEP 684
Multi-processing consumes hundreds of megabytes in separate OS processes. Python 3.12+ introduced Per-Interpreter GIL (PEP 684). Subinterpreters (`_interpreters`) run isolated Python execution engines with private GIL locks inside a single OS process space โก๐งต.
import _interpreters as interp
# Create an isolated subinterpreter with its own private GIL:
sub_id = interp.create()
# Execute code within the isolated interpreter state:
interp.exec(sub_id, "result = sum(i for i in range(1000)); print(f'Subinterpreter computed: {result} ๐')")
# Clean teardown:
interp.destroy(sub_id)
print("Subinterpreter lifecycle complete ๐")
# 89. Subinterpreters & Per-Interpreter GIL: PEP 684 For decades, Python developers had to choose between two bad options for multi-core parallelism: 1. **`threading`**: Lightweight, but crippled by a single Global Interpreter Lock (1 CPU core max). 2. **`multiprocessing`**: Multicore scaling, but heavy OS process memory footprints (forking copies memory pages) and slow serialization overhead (`pickle` over IPC pipes) ๐. ```python # The Subinterpreter Revolution (PEP 684 & PEP 734): # Single OS Process # โโโ Main Interpreter (GIL 1) -> Thread 1 (Core 1) # โโโ Subinterpreter A (GIL 2) -> Thread 2 (Core 2) # โโโ Subinterpreter B (GIL 3) -> Thread 3 (Core 3) ``` ### The Subinterpreter Architecture 1. **Isolated State**: Each subinterpreter has its own `sys.modules`, `builtins`, global variables, and garbage collection heap. A crash or mutation in Subinterpreter A cannot pollute Subinterpreter B. 2. **Independent GIL**: Each subinterpreter acquires and releases its own private GIL. Threads running in separate subinterpreters execute simultaneously on distinct CPU cores without locking each other! 3. **Zero-Copy Memory Channels**: Channels (like `CrossInterpreterBufferView`) allow passing memory buffers between subinterpreters without pickling, achieving high-throughput parallel compute pipelines โก๐งต. --- ### Interactive Pop-Quiz ๐ง #### What is the primary memory and operational advantage of Subinterpreters (PEP 684) over traditional multiprocessing? - [ ] They execute with private per-interpreter GILs inside a single process, avoiding heavyweight OS process forks and IPC serialization โก - [ ] They allow Python code to execute without a CPU - [ ] They compress the bytecode of all standard library modules into a single byte - [ ] They automatically synchronize all global variables across threads --- ### Practical Challenge ๐ฏ Import `_interpreters`. Create a subinterpreter using `create()`. Execute a script inside it that computes `val = 25 * 4` and asserts `val == 100`. Verify `interp.list_all()` contains the subinterpreter before destroy, and destroy it cleanly. Print `'SUBINTERPRETER_VERIFIED'`. <details> <summary>Solution</summary> ```python import _interpreters as interp sub_id = interp.create() active_ids = [item[0] for item in interp.list_all()] assert sub_id in active_ids interp.exec(sub_id, "val = 25 * 4; assert val == 100") interp.destroy(sub_id) remaining_ids = [item[0] for item in interp.list_all()] assert sub_id not in remaining_ids print("SUBINTERPRETER_VERIFIED") ``` </details>