What is a local LLM?
A language model that runs on your own hardware instead of a company's servers: what that means in practice, what it can and cannot do, and why privacy follows from the design.
A local LLM is a language model that runs on your own hardware instead of a company's servers. The model's weights are files on your disk, your questions are answered by your own GPU or CPU, and nothing is sent anywhere to generate an answer. The trade is size: a local model is far smaller than a frontier cloud model, and gives up general range for privacy, offline use, and independence from any account.
What "local" actually means.
A language model is, at bottom, a large file of numbers, its weights, plus the code that runs those numbers over your text. In a cloud product the file lives on the company's servers and your words travel to it. In a local LLM the file lives on your disk, and the computation happens on your silicon. The question never leaves the machine, because the machine is where the model is.
That physical fact is what privacy rests on. There is no upload to weigh, no policy to trust, no retention window to wonder about. A local LLM does not promise to handle your words well; it makes the journey impossible.
What local models can and cannot do.
The honest limit first: frontier cloud models are far larger, and no model that fits on consumer hardware matches their general reasoning and world knowledge. If you need the broadest possible mind, the cloud wins and it is not close.
What local models do well is narrower work. Answering questions over retrieved documents, summarizing a paper, drafting with sources attached, checking a claim against a passage: these jobs are bounded, and a small model grounded in the right context performs them usefully. Around the model, tooling matters as much as raw size, which is why a grounded research loop can produce answers worth checking even on modest hardware.
Local also wins on availability. It works offline, on a plane, in a dead zone, on a machine with no account, and it stops working only when your hardware does. There is no rate limit, no subscription, and no one to change the terms.
What it takes to run one.
Model size is a memory question. The weights must fit in VRAM if you want GPU speed, or in ordinary RAM if you accept CPU speed. Quantization, storing each number in fewer bits, shrinks the file: a 2 to 4 billion parameter model at 3 to 4 bit precision fits in about 4 GB, which is a GTX 1650 class GPU, and each step up in capability costs memory in proportion. On a CPU alone, almost any small model runs, just slowly; whether you need a GPU at all depends on how long your answers tend to be.
Getting started is a download, not a deployment. Tools such as GPT4All put a model gallery on nearly any machine, and research tools such as Istor ship a preconfigured roster so there is nothing to pick. Either way the model arrives as a file, and your documents never go the other way.