Simbaremuteuro explains: Prefill vs. Decode and GPU Serving Bottlenecks

Two Jobs in One LLM Response

[0:00]Hi, I am Simba. Every time a language model answers you, it's actually doing two completely different jobs. First, it reads your entire prompt all at once. Then it writes the reply one word at a time. Those two jobs load the GPU in opposite ways, and mixing them badly is one of the biggest reasons a server ends up slow. Prefill reads your whole prompt in a single parallel pass across every

Prefill: Compute-Bound Parallel Prompt Reading

[0:25]Input token. Decode writes the reply one token at a time, each one waiting on the last. Same model, same GPU, two completely different kinds of work. Prefill looks at every token in your prompt at the same time. 100 tokens or 1,000, it's one forward pass either way, computed as one big matrix multiplication. Every input token feeds into that single

Decode: Memory-Bound Sequential Token Generation

[0:50]Pass at once, and it fans back out into two things: a KV cache entry for every token, and the first token of the reply. That shape is what decides the bottleneck. Prefill multiplies a big batch of tokens against the same weights at once. Lots of math per byte of memory it touches, so the GPU's compute cores stay busy. Decode does the opposite. One token's worth of math, but it still has

Workload Collision: Time to First Token vs. Time per Output Token

[1:16]To pull in every model weight and the entire key value cache from memory to do it. Almost nothing to compute, a lot to read. That's why prefill is called compute bound and decode is called memory bound. Decode's loop never changes shape. Take the last token and the cache built so far, run one forward pass, get the next token out, append it to the cache, and do it again.

Architectural Fixes: Chunking vs. Disaggregated GPU Pools

[1:40]Every step depends on the step before it. There's no way to skip ahead or run two steps at once for the same reply. And the cache it reads from grows by exactly one block every single time around. Servers measure these two phases separately. Time to first token is how long prefill takes. It grows with how long your prompt is. Time per output token is how long each decode step takes. It barely changes with prompt

Your Turn: Diagnosing Which Clock is Stuck on Your Stack

[2:06]Length, but it's set by how much has to be read from memory every single step. Put both phases on the same GPU without care, and they fight. A big prefill batch can hog the GPU for the time it takes to finish, and every decode step waiting behind it in the queue stalls until it's done. A new prompt showing up mid-conversation can freeze everyone else's reply. That's not a bug in one phase, it's the two bottlenecks colliding. One fix keeps them on the same GPU, but schedules around the collision. Sarath serves splits a big prefill into small chunks, and slots each chunk between ongoing decode steps instead of running it all at once. No decode ever waits for a whole prefill to clear, and that alone bought 2.6 to 5.6 times the serving capacity of vLLM, depending on the model. The other fix doesn't share the GPU at all. Dist

Serve runs prefill and decode on separate pools of GPUs entirely. Each one tuned to its own bottleneck, instead of splitting one GPU's time between two different jobs. Against the same latency bar, that served 7.4 times more requests while hitting a 12.6 times tighter deadline on over 90% of them. So, every reply is two jobs, not one. Prefill reads your prompt in one compute-bound pass. Decode writes the reply memory-bound, one token at a time. Left alone on shared hardware, they collide, and two different fixes, chunking the prefill or splitting the GPUs entirely, both turn that collision into serving capacity because they stopped treating one bottleneck like the other. Your turn. If your own serving stack feels slow, check which clock is actually stuck, time to first token or time per output token? The fix for one is not the fix for the other. Prefill versus decode. Simba for humanitarian's AI.