Producer-Consumer Pipelines with asyncio.Queue
Unbounded queues cause memory exhaustion under heavy burst traffic. Specifying `asyncio.Queue(maxsize=N)` provides automatic backpressure: fast producers suspend on `await queue.put()` until slower consumers drain slots with `queue.get()` 🌊📦.
import asyncio
async def producer(queue):
for i in range(3):
await queue.put(f"ITEM_{i}")
await queue.put(None) # Sentinel shutdown token
async def consumer(queue):
while True:
item = await queue.get()
if item is None:
queue.task_done()
break
print(f"Consumed: {item} 🚀")
queue.task_done()
async def main():
q = asyncio.Queue(maxsize=2)
await asyncio.gather(producer(q), consumer(q))
asyncio.run(main())
# 21. Producer-Consumer Pipelines with asyncio.Queue If producers generate jobs faster than consumers can process them and you don't limit queue size, your RAM graph will spike vertically straight into an out-of-memory crash 💀. ```python import asyncio q = asyncio.Queue(maxsize=100) # Enforce physical boundary! ``` ### Backpressure Mechanics 1. **Bounded Queues**: Setting `maxsize=N` turns the queue into a backpressure regulator. The producer cannot run away and hoard memory because it pauses when full 🌊. 2. **Sentinel Tokens**: To shut down background workers cleanly without ugly cancellations, push a sentinel object like `None` into the queue. 3. **Queue Synchronization**: Calling `await q.join()` suspends the caller until every item enqueued has received a corresponding `q.task_done()` call 🎯. --- ### Interactive Pop-Quiz 🧠#### What happens when a producer calls `await queue.put(item)` on an `asyncio.Queue` whose size is equal to `maxsize`? - [ ] The producer's coroutine frame suspends until a consumer extracts an item with `.get()` 🛑 - [ ] The oldest item in the queue is dropped automatically to make room - [ ] An `asyncio.QueueFull` exception is raised immediately - [ ] The queue automatically doubles its memory allocation on the heap --- ### Practical Challenge 🎯 Construct an `asyncio.Queue(maxsize=2)`. Run a producer that pushes integers 1, 2, 3 and then `None`. Run a consumer that pops items, multiplies numbers by 5 into a list `results`, breaks on `None`, and calls `queue.task_done()`. Run both via `asyncio.gather()` and print `results`. <details> <summary>Solution</summary> ```python import asyncio async def producer(q): for num in (1, 2, 3): await q.put(num) await q.put(None) async def consumer(q, results): while True: item = await q.get() if item is None: q.task_done() break results.append(item * 5) q.task_done() async def main(): q = asyncio.Queue(maxsize=2) results = [] await asyncio.gather(producer(q), consumer(q, results)) print(results) asyncio.run(main()) ``` </details>