What is model distillation?
How a large model's behavior gets packed into a small one, and why so many of the models that run on your own hardware were built that way.
Model distillation trains a small model to behave like a larger one: the big model generates outputs, and the small model learns to reproduce them, so it inherits much of the teacher's behavior at a fraction of the size. The student is not smarter, it is cheaper. That trade is why so many capable assistants run on models that fit in a few gigabytes of memory.
The teacher writes, the student learns.
Ordinary training teaches a model from collected text; what training data is covers that raw-material side. Distillation changes the material. A strong teacher model is prompted to produce answers on a huge range of inputs, and those outputs become the small model's examples. The advantage is that the examples are already in the shape of good responses, so the student learns the form of a good answer and not only the statistics of text. The student can even train on the teacher's probability distributions, which carries information about what the teacher considered plausible, not just what it finally chose.
What the student keeps, and what it loses.
A distilled model reads like its teacher: similar tone, similar habits, similar competence on everyday questions. It does not inherit everything. Knowledge lives in the parameters, and what a smaller parameter count means is a smaller place to store it, so the tail of rare and specialized knowledge thins out first. Students also tend to be narrower: they do well inside the distribution their teacher covered and drift outside it. That is why a distilled model can feel as good as a much larger one on ordinary questions and noticeably weaker at the edges.
Why local AI leans on it.
The consumer sweet spot sits in models that answer well while staying small enough to hold in modest memory, and distillation is one of the main ways such models get made. The training happens long before download, so it changes nothing at run time; the file that lands on your machine is just the finished student, and running it works the same as any other model. Size still has to fit the hardware, which is where quantization takes over after training, and the same arithmetic explains why model size dominates speed on a laptop.