Continuous Batching Explained: Eliminating Idle GPU Slots in LLM Serving | Chinombe Simbaremuteuro

Introduction: Static Batching and Wasted GPU Compute

[0:00]Hi, I am Simba. Say a batch of eight requests starts decoding together. One of them finishes after 20 tokens. Another needs 400. Static batching holds all eight slots until the slowest one is done. So, for the last few hundred tokens, most of those slots are doing nothing useful. What if a finished slot could just be handed to the next request waiting in line? Here's the idea. After every single decoding step, a scheduler looks at the batch. Anything that just finished gets evicted right then. Whatever's waiting in the queue gets the freed slot on the next step. No request

The Iteration-Level Scheduling Loop: Step-by-Step Batch Deciding Worked Example: 4-Slot Scheduler and Immediate Request Admission

[0:36]Waits for the rest of the batch to catch up. The batch itself is a different group of sequences almost every step. Static batching, the baseline, eight requests start together and the batch doesn't release until every single one is done. The ones that finish early don't leave. They sit there, padded, holding a slot for nothing, while new requests queue outside and wait for the whole group to drain. Two ways to run a batch. Static, a fixed group for its whole lifetime, a finished sequence still takes up a slot, padded, and new requests wait for every slot to clear. Continuous, the scheduler redecides who's in the batch every single step. A finished sequence is evicted right away and a new request can join on the next one. Mechanically, it's a loop. Decode one step for every sequence currently in the batch. Check

Whether anything just hit its end token or its length limit. If something finished, evict it, free the slot, and pull the next request off the queue into that slot. Whatever the batch looks like now, decode one more step and check again. The batch's membership is a decision made fresh every single step. Here's a small worked example. Four slots. Request B finishes at step three, a short request. Slot two doesn't sit idle. Request C gets admitted right at step four, starting with its own profile step while the others keep decoding. Slot four stays empty until step six, not because the scheduler is slow, but because the queue was actually empty until then. Continuous batching fills a slot the moment there's something to fill it with, nothing more. This only works cheaply if the KV cache is managed per sequence, allocated and freed

The Memory Requirement: Why Paged KV Cache Management is Essential

[2:15]Independently, not as one fixed block for the whole batch. That's exactly the paged block-based memory layout from the paged attention episode. Without it, admitting a new request mid-batch means reshaping a contiguous buffer on every eviction, expensive enough to undo the whole point. This next number is invented purely to show the shape of it, not a measurement. Picture a batch where request lengths vary a lot. With static batching, the padded, already finished slots mean a chunk of GPU time buys nothing. With continuous batching, those slots go to waiting requests instead of sitting padded. The real number depends entirely on how varied your request lengths are and how deep your queue is. Three real knobs, max batch size, bounded by how

Three Production Knobs: Max Batch Size, Admission Policy, and Prefill Spikes

[3:02]Much KV cache memory you have, not by compute. Admission policy, plain first-come, first-served, or something that accounts for how long a request has waited. And how a new request's first step gets handled. Prefill is a heavier, more compute-bound step than an ongoing decode, so admitting several requests at once can spike that step's cost. What actually caps the gain isn't the scheduling idea, it's memory. A clever scheduler staring at a full KV cache still has nowhere to put a new request. And every admission mixes a prefill step into what was a steady decode pattern. That changes the cost of the step itself, not just how many slots are filled. The tradeoff is real. You gain no slot sits padded for the slowest sequence, and a short request doesn't wait behind a long one just to

Engineering Trade-offs: Latency Dependencies and Memory Limits

[3:51]Start. You take on a scheduling decision every single step instead of one decision at the start, and your latency now depends partly on who else happened to get admitted around you. So, continuous batching reclaims the GPU time static batching spends on padded, already finished slots by redesigning the batch every step instead of once. What it costs is a scheduler running continuously and KV cache memory managed per sequence to make that cheap. And it's worth saying plainly what it isn't. It's not a way around a full cache. If there's no memory for new sequence, no scheduler changes that. If you're serving variable length requests, check

Summary: Reclaiming Idle GPU Slots in LLM Inference Servers

[4:30]Whether your server still forms one fixed batch per inference call. If it does, GPU time is probably going to padded, already finished slots, and that's exactly the gap continuous batching is built to close. Continuous batching, Simba for Humanitarian's AI.