What is quantization?
The reason a language model fits on the hardware you already own. A compression step with a small, honest cost.
Quantization stores a model's numbers in fewer bits than they were trained with. A model is a very large collection of numbers, and rounding them from 16 bit to 4 bit precision makes the file several times smaller, which is what lets a modest GPU hold one at all. The cost is a small loss of precision that rarely changes the quality of an answer; the benefit is that serious models run on ordinary hardware.
A model is a very large list of numbers.
Everything a language model knows lives in its parameters, the values tuned during training. At full precision each one takes 16 or 32 bits of storage, and the counts run into the billions, so the memory bill is enormous: a 4 billion parameter model wants roughly 8 GB before it has answered a word. Quantization rounds each parameter into fewer bits, typically 4 or fewer. The rounded values are not identical to the originals, but they are close enough that the model's behavior survives, the way a photo survives being saved as a JPEG.
Why local AI leans on it.
The compression is what makes the local story arithmetic that works. At 4 bit precision a parameter takes about half a byte, so a 2 billion parameter model fits near 1.5 GB and a 4 billion parameter model near 2.5 GB, which is what lets a GTX 1650's 4 GB of VRAM carry a model and still leave room for the context window around it. Without quantization, local research tools would need workstation hardware, and the GPU question would have a very different answer.
The trade, honestly.
Quantization does cost something, and pretending otherwise would be the wrong way to earn trust. Measured benchmarks dip slightly at 4 bit, and the loss grows if you go more aggressive than that. For grounded question answering the loss is dominated by other factors anyway: the relevance of the retrieved passages and whether the answer is verified against them matter far more than the storage precision of the model reading them. The honest summary is that quantization trades a sliver of theoretical accuracy for the ability to run at all on hardware people actually own, and in practice that trade is almost always worth it.