Process Pools with ProcessPoolExecutor
Manually spawning processes incurs high OS initialization latency. `concurrent.futures.ProcessPoolExecutor` maintains a pre-warmed worker pool, distributing tasks across CPU cores with unified `Future` workflows 🏭.
from concurrent.futures import ProcessPoolExecutor
def square(x):
return x * x
if __name__ == "__main__":
with ProcessPoolExecutor(max_workers=2) as pool:
results = list(pool.map(square, [1, 2, 3, 4]))
print(f"Squares: {results} 🏭")
# 80. Process Pools with ProcessPoolExecutor Manually spawning processes incurs high OS initialization latency. `concurrent.futures.ProcessPoolExecutor` maintains a pre-warmed worker pool, distributing tasks across CPU cores with unified `Future` workflows 🏭. ```python from concurrent.futures import ProcessPoolExecutor def square(x): return x * x if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as pool: results = list(pool.map(square, [1, 2, 3, 4])) print(f"Squares: {results} 🏭") ``` Starting in Python 3.11, `ProcessPoolExecutor` supports `max_tasks_per_child` to automatically retire and respawn worker processes after $N$ tasks, guarding against creeping memory leaks in third-party C libraries 🛡️! --- ### Question Why is `if __name__ == '__main__':` mandatory when launching processes with `ProcessPoolExecutor`? 🧠 - [ ] a. Without the guard, spawned child processes re-import the entry script, causing an infinite process fork bomb 💀 - [ ] b. It signals the operating system scheduler to assign real-time thread priorities - [ ] c. It triggers Python's JIT bytecode compilation pass - [ ] d. The Linux kernel terminates any process created without an explicit main scope --- ### Challenge Use `ProcessPoolExecutor` to compute `x * 10` for `[1, 2, 3]`. Collect results as a list and print it. 🎯 <details> <summary>Solution</summary> ```python from concurrent.futures import ProcessPoolExecutor def times_ten(x): return x * 10 if __name__ == '__main__': with ProcessPoolExecutor(max_workers=2) as ex: res = list(ex.map(times_ten, [1, 2, 3])) print(res) ``` </details>