Multi-Producer Multi-Consumer Pipeline
Connecting asynchronous data ingestion to heavy downstream processing requires a Multi-Producer Multi-Consumer (MPMC) pipeline. An asyncio.Queue bounds backpressure, while sentinel markers (like None) signal consumers to gracefully drain and terminate.
import asyncio
async def consumer(queue: asyncio.Queue):
while True:
item = await queue.get()
if item is None:
queue.task_done()
break
# Process item
queue.task_done()
# Multi-Producer Multi-Consumer Pipeline When 5 ingest streams dump real-time events that must be parsed, enriched, and saved by 3 consumer workers, you need an **MPMC Pipeline** (Multi-Producer Multi-Consumer). 📦🔄 Without bounded queues, fast producers will hoard millions of unparsed objects into memory until the OS terminates the process with `OOMKilled`. An `asyncio.Queue(maxsize=N)` provides backpressure: producers automatically pause when the queue fills up, keeping memory consumption strictly flat. ```python import asyncio async def pipeline_worker(queue: asyncio.Queue, handler): while True: payload = await queue.get() if payload is None: queue.task_done() break await handler(payload) queue.task_done() ``` ## Pop Quiz Why must a producer push exactly one `None` sentinel per consumer to terminate an MPMC queue gracefully? - [ ] Each sentinel consumes and terminates exactly one consumer task; fewer sentinels leave consumers hanging indefinitely. - [ ] The asyncio.Queue class raises StopIteration when multiple sentinels exist. - [ ] Python allocates thread local memory for every sentinel pushed into the loop. - [ ] Sending multiple None values corrupts the internal FIFO linked list pointers. ## Real-World Challenge Build `mpmc_pipeline(producers_data: list[list[int]], num_consumers: int, process_fn)`. Producers must concurrently stream their lists into a bounded queue (`maxsize=10`), while `num_consumers` pull items, process them, and collect results in a thread-safe manner before cleanly shutting down. <details> <summary>Solution</summary> ```python import asyncio async def mpmc_pipeline(producers_data, num_consumers: int, process_fn): queue = asyncio.Queue(maxsize=10) results = [] lock = asyncio.Lock() async def producer_worker(items): for item in items: await queue.put(item) async def consumer_worker(): while True: item = await queue.get() if item is None: queue.task_done() break processed = await process_fn(item) async with lock: results.append(processed) queue.task_done() # Start consumer tasks consumers = [asyncio.create_task(consumer_worker()) for _ in range(num_consumers)] # Run all producers to completion prod_tasks = [asyncio.create_task(producer_worker(batch)) for batch in producers_data] await asyncio.gather(*prod_tasks) # Wait until all submitted items have been processed await queue.join() # Send poison pills to shut down consumers for _ in range(num_consumers): await queue.put(None) await asyncio.gather(*consumers) return results # Verification async def run_test(): async def triple(x): return x * 3 batches = [[1, 2], [3, 4, 5], [6]] output = await mpmc_pipeline(batches, num_consumers=3, process_fn=triple) assert sorted(output) == [3, 6, 9, 12, 15, 18] asyncio.run(run_test()) ``` </details>