What is retrieval-augmented generation?
The technique behind "chat with your documents", explained plainly: what retrieval contributes, what the model still gets wrong, and why the local version is different.
Retrieval-augmented generation (RAG) is a method where a language model answers from documents found for the question instead of from its training alone. The question is matched against a document collection, the most relevant passages are placed into the prompt, and the answer is written from those passages, ideally with a citation back to each one. A model that has never seen your files can answer questions about them.
The three steps.
First, retrieval. Your documents are indexed ahead of time, usually as embeddings, which are number representations of meaning that let a question match passages by sense rather than by exact words. When you ask, the system finds the passages that bear on the question. Good systems use hybrid retrieval, combining that semantic match with plain keyword search, because each catches what the other misses.
Second, assembly. The retrieved passages are placed into the prompt alongside your question, so the model reads them as context. What goes in is what the answer can draw on, which is why retrieval quality bounds answer quality.
Third, generation. The model writes an answer grounded in those passages. Done well, every claim points to the passage it came from, so the reader can check it in a click.
What RAG fixes, and what it does not.
What it fixes is access. A model's training data stops at a date and never included your private files; retrieval lets it answer from what is actually in the room. It is the difference between asking a well-read stranger and asking someone holding your documents.
What it does not fix is faithfulness. Retrieval finds the passages; it cannot force the model to stick to them. A model can still write a claim the passages do not support, blend two sources into one, or answer past what was retrieved from memory of its training. This is the quiet failure of a lot of RAG products: the citations look convincing whether or not they hold. Why a chatbot invents citations at all is its own question, and it explains why retrieval alone leaves the problem half-solved.
The fix is verification, not more retrieval. A grounded system checks each claim against the passage it cites before the answer is shown, and refuses when the sources do not contain the answer. Refusal is the feature: in research work, "your sources do not say" is a finding, and an invented answer is a liability.
The local version.
Nothing in the RAG loop needs a server. The index, the embeddings, the retrieval and the model can all run on one machine, and when they do, your documents are never uploaded to be processed. That is what "local RAG" means: the same technique, with the trust question removed, because there is nobody to send anything to. What actually leaves the machine is the next question to ask of any such setup. A GTX 1650 class GPU runs it comfortably, and it works offline the whole way through.