Transformer

Input tokens are converted to embeddings and combined with positional information, since attention itself has no inherent notion of order. Each block then lets every position attend to the others, producing a representation that is contextual rather than fixed — the vector for a word depends on the words around it. A feed-forward network transforms each position independently, and residual connections plus normalisation keep gradients healthy through many stacked layers. Depth and width scale this pattern up; the pattern itself does not change.

Two consequences follow for anyone building on top. Training parallelises across the sequence, which is what makes large-scale pretraining feasible, but generation does not — decoding is inherently one token at a time, and no amount of hardware removes that serialisation. And the cost of attention grows quadratically with sequence length, which is the root reason context windows have limits, why long prompts are disproportionately expensive, and why so much serving engineering is devoted to managing the key-value cache.

The misconception is that architecture is what distinguishes today’s models from each other. Most share the same fundamental design; the differences that matter in practice come from training data, scale, post-training alignment and inference-time technique. Knowing the architecture explains why models behave as they do at a mechanical level — it does not predict which one will be better at your task.