What is tokens per second?
The speed a model writes, measured in the pieces it thinks in. It is the number that decides whether local AI feels instant or sluggish.
Tokens per second is the rate a model generates text, counted in tokens, the word pieces a model reads and writes. The measure exists because generation is one piece at a time: during inference, each piece is computed from everything before it. Faster rates make answers feel instant; slower ones make you watch an answer arrive. It is a speed number only, and it says nothing about whether the text is right.
The rate you actually feel.
A person reads a few hundred words per minute, which lands somewhere near five to ten tokens per second. A model sustaining twenty or thirty is streaming faster than you can read, so the answer feels immediate even though the computation is still unfolding. Below roughly ten, long answers start to drag; below five, the tool feels broken even when nothing is wrong. That reading-speed yardstick is why hardware pages obsess over this number, and why an entry-level GPU like a GTX 1650 can feel perfectly usable: what matters is clearing the reading bar, not competing with a datacenter.
What moves the number.
Three things dominate. Model size, because each piece requires running the whole model once, so a bigger parameter count costs speed at the same hardware. Quantization, since a smaller file means less arithmetic per step; the same model at 4 bits runs quicker than at 8. And memory bandwidth, which is why a GPU's speed shows up here more than its raw compute does. As an answer grows, the rate usually drifts down, because each new piece attends to everything written so far. If the number sits below your reading bar, the hardware reasons behind it are on why local AI runs slow on a laptop.
Speed is not quality.
The same model with the same input produces the same text whether it streams at thirty tokens per second or three. Slow generation never makes an answer worse, and fast generation never makes it better; correctness lives elsewhere, in grounding and verification structure. What speed really changes is which jobs feel practical: a fast setup invites long documents and full-length answers, while a slow one quietly trains you to ask smaller questions. That is also why retrieval matters locally, since reading your library is a separate and much faster computation than writing the answer out.