Back
+20 XP
Concept

Producer-Consumer Pipelines with asyncio.Queue

Unbounded queues cause memory exhaustion under heavy burst traffic. Specifying `asyncio.Queue(maxsize=N)` provides automatic backpressure: fast producers suspend on `await queue.put()` until slower consumers drain slots with `queue.get()` 🌊📦.

python
import asyncio

async def producer(queue):
    for i in range(3):
        await queue.put(f"ITEM_{i}")
    await queue.put(None) # Sentinel shutdown token

async def consumer(queue):
    while True:
        item = await queue.get()
        if item is None:
            queue.task_done()
            break
        print(f"Consumed: {item} 🚀")
        queue.task_done()

async def main():
    q = asyncio.Queue(maxsize=2)
    await asyncio.gather(producer(q), consumer(q))

asyncio.run(main())

# 21. Producer-Consumer Pipelines with asyncio.Queue If producers generate jobs faster than consumers can process them and you don't limit queue size, your RAM graph will spike vertically straight into an out-of-memory crash 💀. ```python import asyncio q = asyncio.Queue(maxsize=100) # Enforce physical boundary! ``` ### Backpressure Mechanics 1. **Bounded Queues**: Setting `maxsize=N` turns the queue into a backpressure regulator. The producer cannot run away and hoard memory because it pauses when full 🌊. 2. **Sentinel Tokens**: To shut down background workers cleanly without ugly cancellations, push a sentinel object like `None` into the queue. 3. **Queue Synchronization**: Calling `await q.join()` suspends the caller until every item enqueued has received a corresponding `q.task_done()` call 🎯. --- ### Interactive Pop-Quiz 🧠 #### What happens when a producer calls `await queue.put(item)` on an `asyncio.Queue` whose size is equal to `maxsize`? - [ ] The producer's coroutine frame suspends until a consumer extracts an item with `.get()` 🛑 - [ ] The oldest item in the queue is dropped automatically to make room - [ ] An `asyncio.QueueFull` exception is raised immediately - [ ] The queue automatically doubles its memory allocation on the heap --- ### Practical Challenge 🎯 Construct an `asyncio.Queue(maxsize=2)`. Run a producer that pushes integers 1, 2, 3 and then `None`. Run a consumer that pops items, multiplies numbers by 5 into a list `results`, breaks on `None`, and calls `queue.task_done()`. Run both via `asyncio.gather()` and print `results`. <details> <summary>Solution</summary> ```python import asyncio async def producer(q): for num in (1, 2, 3): await q.put(num) await q.put(None) async def consumer(q, results): while True: item = await q.get() if item is None: q.task_done() break results.append(item * 5) q.task_done() async def main(): q = asyncio.Queue(maxsize=2) results = [] await asyncio.gather(producer(q), consumer(q, results)) print(results) asyncio.run(main()) ``` </details>

Checkpoint

What happens when a producer calls `await queue.put(item)` on an `asyncio.Queue` whose size is equal to `maxsize`? 🧠

Practice

Construct an `asyncio.Queue(maxsize=2)`. Run a producer that pushes integers 1, 2, 3 and then `None`. Run a consumer that pops items, multiplies numbers by 5 into a list `results`, breaks on `None`, and calls `queue.task_done()`. Run both via `asyncio.gather()` and print `results`. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks