Inference

The two phases have different performance characteristics, and knowing which one you are in explains most latency behaviour. Prefill is compute-bound and parallel across the prompt, so doubling prompt length increases time-to-first-token but does not double the total wall clock in the way people expect. Decode is sequential and memory-bandwidth-bound: each new token requires reading the model weights and the accumulated key-value cache, which is why output length usually dominates end-to-end latency and why streaming makes such a difference to perceived speed.

Serving systems optimise around those facts. Continuous batching interleaves requests so the GPU is not idle while one sequence finishes; paged key-value cache management prevents memory fragmentation when many sequences of different lengths share a device; prompt caching reuses the prefill work for a stable prefix across requests. Each of these is invisible from the API but shows up directly in cost per request and in tail latency.

The misconception is that inference cost tracks model size alone. It tracks tokens — in and out — multiplied by how efficiently your traffic batches. A modest model called with bloated prompts on unbatchable traffic can cost more to run than a larger model with disciplined context management, and the first place to look when a bill grows is almost always prompt size, not model choice.