What is an inference engine?
The software that turns a model's weights into answers, and the reason the same file runs at different speeds in different tools.
An inference engine is the software that turns a model's weights into answers. The model file stores what the network learned; the engine loads it, runs the arithmetic that generates each token, manages memory and context, and serves the result to whatever application sits above. The same file runs faster or slower depending on the engine. This is what makes a model portable: one GGUF file, many engines, and the hardware beneath deciding the speed.
What the engine actually does.
Inference itself is the process of generating an answer from a model; the engine is the program that carries it out. It loads the weights from the model file, converts your text into tokens, runs the network's arithmetic over each step, picks the next token, and repeats until the answer is complete. Around that core loop it manages the memory budget, keeps the conversation context ordered, and exposes the whole thing through an interface an application can call. None of that changes what the model knows; it decides how well the model's knowledge gets used.
Why engines decide speed.
Generation is dominated by a single cost, streaming the model's weights from memory into the processor on every token. How well an engine does that is engineering, not mathematics: kernel code tuned to the processor, native support for compressed weights, and smart placement of layers in fast GPU memory. This is why two tools running the identical model file can differ noticeably on tokens per second, and why engine updates sometimes speed up every tool built on them without any model changing.
The engine under your tools.
Most local AI applications do not write their own engine; they embed one, most often llama.cpp or a runtime derived from it. That division of labor is why a GGUF file works across so many different apps, and why the application you see is really a research layer above a shared running layer. When an answer feels slow, the application's settings usually reach down into engine territory, GPU offload and context size chief among them, which is why knowing the layer exists makes the knobs readable.
Where the engineering shows.
The same weights, the same question, three tools, three speeds, and the differences trace to concrete decisions. Kernel code tuned for a specific processor architecture can double the throughput of the arithmetic itself. Support for a quantization format, when native, avoids dequantizing weights in the hot path. Layer placement decides how much of the model lives in the fast memory versus system RAM. Even batching, how the engine schedules tokens when a server serves several requests at once, changes perceived latency. None of this is visible in the model file, which is why identical files genuinely perform differently.
The settings that reach the engine.
Application settings usually expose engine decisions directly, and knowing which is which makes them readable. GPU offload counts the layers handed to the graphics card; the remainder runs on the CPU, so the setting is a memory-budget dial rather than a speed toggle. Context size reserves the working memory for conversation and documents. Sampling settings, temperature and its neighbors, steer token choice and belong to the engine's output stage rather than its memory management. When a model runs slowly, working through these in order, fit first, context second, sampling last, resolves most cases before downloading anything new.