Asynchronous Batching Aggregator (Micro-Batcher)
Issuing a distinct database write or model inference call for every single user request crushes throughput. A micro-batcher buffers incoming coroutine calls, flushing a bulk execution when either a batch size limit is met or a timeout window elapses.
import asyncio
class MicroBatcher:
def __init__(self, size=10, timeout=0.05):
self.size = size
self.timeout = timeout
self.buffer = []
# Asynchronous Batching Aggregator (Micro-Batcher) Running a separate SQL `INSERT` or GPU inference call for every incoming HTTP request generates immense network overhead. However, asking users to wait for 100 requests to accumulate before processing means low-traffic periods hang indefinitely. 🏎️📦 A **Micro-Batcher** balances latency and throughput: 1. **Size-triggered flush**: If 10 requests arrive instantly, flush them as a single batch immediately. 2. **Time-triggered flush**: If only 2 requests arrive, flush them anyway after 50 milliseconds so those users never wait too long. ```python import asyncio class MicroBatcher: def __init__(self, max_batch_size: int, max_wait: float, processor): self.max_batch_size = max_batch_size self.max_wait = max_wait self.processor = processor ``` ## Pop Quiz Why does a micro-batcher require both a size limit and a timeout window? - [ ] The size limit bounds throughput latency under heavy load, while the timeout prevents starvation during low traffic. - [ ] The timeout is required to purge unreferenced python objects from the generational GC. - [ ] Without both parameters, the TCP socket connection automatically resets to half-duplex. - [ ] To prevent thread deadlocks between the asyncio loop and the operating system kernel. ## Real-World Challenge Implement `MicroBatcher(max_batch_size: int, max_wait: float, batch_processor)`. - `await add(item)`: Appends an item to the buffer and returns a future that resolves with the item's individual result once the batch processes. - When `max_batch_size` is reached, flush immediately. - If fewer items arrive, flush after `max_wait` seconds. <details> <summary>Solution</summary> ```python import asyncio class MicroBatcher: def __init__(self, max_batch_size: int, max_wait: float, batch_processor): self.max_batch_size = max_batch_size self.max_wait = max_wait self.batch_processor = batch_processor self.queue = [] self.timer_task = None self.lock = asyncio.Lock() async def add(self, item): fut = asyncio.get_running_loop().create_future() async with self.lock: self.queue.append((item, fut)) if len(self.queue) >= self.max_batch_size: if self.timer_task and not self.timer_task.done(): self.timer_task.cancel() self._dispatch() elif len(self.queue) == 1: self.timer_task = asyncio.create_task(self._delayed_flush()) return await fut async def _delayed_flush(self): await asyncio.sleep(self.max_wait) async with self.lock: if self.queue: self._dispatch() def _dispatch(self): batch = list(self.queue) self.queue.clear() asyncio.create_task(self._process_batch(batch)) async def _process_batch(self, batch): items = [item for item, _ in batch] results = await self.batch_processor(items) for (_, fut), res in zip(batch, results): fut.set_result(res) # Verification async def run_test(): call_count = 0 async def bulk_calc(items): nonlocal call_count call_count += 1 return [x * 2 for x in items] batcher = MicroBatcher(max_batch_size=3, max_wait=0.05, batch_processor=bulk_calc) # Size-triggered test t1 = asyncio.create_task(batcher.add(10)) t2 = asyncio.create_task(batcher.add(20)) t3 = asyncio.create_task(batcher.add(30)) res1 = await asyncio.gather(t1, t2, t3) assert res1 == [20, 40, 60] assert call_count == 1 # Time-triggered test t4 = asyncio.create_task(batcher.add(5)) await asyncio.sleep(0.06) r4 = await t4 assert r4 == 10 assert call_count == 2 asyncio.run(run_test()) ``` </details>