Streaming

The mechanism exploits the sequential nature of decoding: tokens exist one at a time anyway, so there is no reason to hold them. Transport is usually server-sent events or a chunked HTTP response, with the server forwarding provider chunks as they arrive. The number that changes is time-to-first-token, and it changes the experience disproportionately — a response that takes twelve seconds to finish feels responsive if it starts in four hundred milliseconds and unbearable if the screen is blank for twelve seconds.

Streaming complicates everything downstream of the text. Markdown may be half-parsed mid-stream, code fences arrive unclosed, and structured output cannot be validated until it is complete. Interfaces therefore need a rendering strategy that tolerates incomplete input, a cancellation path that actually aborts the upstream request rather than just hiding the output, and an error story for a stream that fails after partial delivery — the user has already read text that may now be retracted.

The misconception is that streaming is a performance optimisation. It optimises perception, not throughput: the same tokens take the same time, and streaming a response the user must wait for in full anyway — because the next action depends on the complete result — adds complexity for nothing. Stream where the output is read progressively; buffer where it is consumed atomically.