How to run an LLM locally
From nothing to a first answer on your own machine: the three decisions that matter, and the hardware bar you actually need to clear.
Running an LLM locally takes three things: a tool that loads and runs models, a model file that fits your hardware, and enough memory to hold it while it works. The process is a download, not a deployment. Pick a desktop app, pick a model file sized to your machine, and the first answer is usually minutes away. The hardware bar is lower than most people expect, because quantized files shrink models dramatically.
Three things, in order.
First, the tool. A local model needs software that can load its weights and generate from them, and today that means a desktop application or a command-line runner rather than any deployment of your own. Second, the file. The model arrives in most cases as a GGUF file, chosen for your hardware. Third, memory. The file must fit while the model runs, which makes the RAM arithmetic the one calculation worth doing up front. Take those three in that order and the rest is waiting for the download to finish.
Choosing the file.
Model names hide the decision, but the decision is simple. The parameter count sets the capability tier, and the quantization code sets the size, so a file's full name is really a spec sheet. Start small: a 2 to 4 billion parameter model at 4-bit precision runs on nearly any modern machine and answers real questions. Bigger files cost proportionally more memory and time, and the honest way to choose is to check the model card for what the model was built for, then let your hardware pick the bit depth.
The first answer.
Speed is the honest variable. On a CPU, short answers arrive in reasonable time and long ones test patience; on even a modest GPU the difference is dramatic, and tokens per second is the number to compare across setups. The threshold is low: a GPU is not required, though a GTX 1650 class card changes the whole experience, as covered on running research AI on a GTX 1650. One habit worth forming from the first session: the model holds nothing between conversations, so anything worth keeping belongs in your notes, not in the chat.