Quantization

The method maps a range of high-precision values onto a smaller set of discrete levels, storing a scale factor so the values can be reconstructed approximately. Post-training quantization applies this to a finished model, sometimes using a small calibration set to choose ranges sensibly; quantization-aware training instead exposes the model to the rounding during training so it adapts. Not every layer is treated equally — implementations commonly keep sensitive components at higher precision because the accuracy cost of quantising them is disproportionate.

This is the technique that puts capable open-weight models on consumer hardware. A model that will not fit in available memory at full precision may fit comfortably at four bits, and local runtimes distribute quantised variants by default for exactly that reason. The trade is real but often modest: some loss of fidelity, most visible on tasks with tight precision requirements, in exchange for the model running at all.

The misconception is that quantization is free below some magic bit width. Degradation is gradual, uneven across tasks, and does not announce itself — a heavily quantised model can look fine in casual use and fail specifically on the long-tail reasoning or formatting cases your application depends on. The only way to choose a level responsibly is to run your own evaluation set against each candidate rather than trusting a general rule.