Distillation

The teacher supplies a richer training signal than a hard label does. Its full output distribution encodes which alternatives were nearly as good, and that relative information transfers structure the student would otherwise have to discover for itself. In practice this may mean training on the teacher’s probability distributions, on its generated responses, or on its reasoning traces for tasks where the intermediate steps carry the value.

The motivation is almost always operational. A distilled model serves at a fraction of the cost, responds faster, and can run on hardware the teacher cannot fit on — which matters most for high-volume, narrow tasks: classification, extraction, routing, moderation, the first stage of a pipeline that escalates hard cases to a larger model. Distillation is also how many small open-weight releases acquire capabilities that would be implausible for their size.

The misconception is that a distilled model is a smaller version of the teacher. It is a specialist shaped by the distribution it was trained on. Inside that distribution it can be close to indistinguishable; outside it, quality falls off in ways that general benchmarks may not reveal. Distillation is a narrowing as much as a compression, and it is worth confirming that the narrowing matches the traffic you actually serve. Licensing is a second real constraint: many providers restrict using their outputs to train competing models.