What is llama.cpp?
The engine underneath most local AI. A model file is the cargo; this is what actually does the computing on your machine.
llama.cpp is an open-source engine that runs language models on ordinary hardware, written from scratch in C++ to perform inference with no cloud, no Python stack and no GPU requirement. It reads GGUF files, runs on CPUs and GPUs alike, and powers most local AI tools either directly or through runtimes built on it. When a desktop app answers from a model on your machine, this engine is often what is doing the computing.
The engine under the model file.
A model file, even a well-made one, does nothing by itself; something has to load its weights and compute. That something is an inference engine, and llama.cpp is the one most local tools inherit from. When you ask a question, the engine reads the model's parameters, runs the step-by-step computation that produces an answer, and hands back one token at a time for the app to display. The app is the part you see; the engine is the part doing the arithmetic, and the split is why different desktop tools can run the same model file with the same results.
Built to fit in the memory you have.
llama.cpp was written to run models on the hardware people actually own, which is why quantization is native to it rather than an afterthought: the engine reads 4-bit weights directly and does its arithmetic accordingly. That is the pairing that makes the whole local stack work, a compact file format and an engine that loads it anywhere. The format is GGUF, and because the engine reads it natively, one downloaded file runs on a CPU-only laptop, an entry-level GPU, or a faster card, with the hardware deciding the speed rather than the possibility.
Why your tools sit on it.
Most desktop AI applications do not write their own inference code. They embed llama.cpp or a runtime built from it, which is why the model files you download work across so many different apps, and why improvements to the engine quietly speed up every tool that uses it. For a researcher, the practical consequence is trust that travels well: an open engine, open model weights, and a computation that happens on your machine, with no service in the middle and nothing about the process hidden. That stack is why local inference can be audited at all.