What is an inference engine?
The software that turns a model's weights into answers, and the reason the same file runs at different speeds in different tools.
An inference engine is the software that turns a model's weights into answers. The model file stores what the network learned; the engine loads it, runs the arithmetic that generates each token, manages memory and context, and serves the result to whatever application sits above. The same file runs faster or slower depending on the engine. This is what makes a model portable: one GGUF file, many engines, and the hardware beneath deciding the speed.
What the engine actually does.
Inference itself is the process of generating an answer from a model; the engine is the program that carries it out. It loads the weights from the model file, converts your text into tokens, runs the network's arithmetic over each step, picks the next token, and repeats until the answer is complete. Around that core loop it manages the memory budget, keeps the conversation context ordered, and exposes the whole thing through an interface an application can call. None of that changes what the model knows; it decides how well the model's knowledge gets used.
Why engines decide speed.
Generation is dominated by a single cost, streaming the model's weights from memory into the processor on every token. How well an engine does that is engineering, not mathematics: kernel code tuned to the processor, native support for compressed weights, and smart placement of layers in fast GPU memory. This is why two tools running the identical model file can differ noticeably on tokens per second, and why engine updates sometimes speed up every tool built on them without any model changing.
The engine under your tools.
Most local AI applications do not write their own engine; they embed one, most often llama.cpp or a runtime derived from it. That division of labor is why a GGUF file works across so many different apps, and why the application you see is really a research layer above a shared running layer. When an answer feels slow, the application's settings usually reach down into engine territory, GPU offload and context size chief among them, which is why knowing the layer exists makes the knobs readable.