What does Q4_K_M mean?
The code stamped on a compressed model file, naming exactly how its numbers were squeezed. Once you can read it, model downloads stop being a lottery.
Q4_K_M is a quantization recipe label, mostly seen in GGUF file names. The Q4 means weights are stored at about 4 bits each, down from 16. The K names a family of recipes that mixes block scales for better accuracy, and the M says medium: within that family, a middle setting between quality and size. Read the code and you know a file's size, speed and quality trade before downloading it.
Reading the code piece by piece.
The number after the Q is the headline: bits per weight, and the main dial on file size. Each halving roughly halves the download and the memory bill. The letters after it describe how the rounding is done. K series recipes store weights in blocks, each with its own scale factor, which keeps precision where it is needed instead of flattening it everywhere. The trailing letter grades the recipe: L for large, M for medium, S for small. Q4_K_M therefore reads as: four bits, K-style block scales, the medium variant of that family.
Why the mixing matters.
Not all weights are equally important, and a good recipe spends its bits accordingly. K-family quantizations keep the most sensitive parts of the model, such as attention and output layers, at higher precision while compressing the rest harder. That is why Q4_K_M consistently tests close to the full-precision model on ordinary tasks, while a naive uniform 4 bit scheme loses more. The same logic explains the ladder: every step from Q4_K_S to Q5_K_M to Q6_K buys a little accuracy back at the cost of size, and your memory budget picks where you stop.
Which one to pick.
The honest rule is: take the highest quality that fits in your memory with room to spare, and do not chase lower sizes than you need. Q4_K_M earned its popularity as the balance point, small enough for a 4 to 8 GB card and accurate enough that most users never notice the compression. Below 4 bits, quality degrades enough to notice in careful reading, which matters in research work where a subtly degraded model can misread a passage. If a tool ships a fixed roster, someone made this choice for you already; that is one less dial to learn.