What is training data?
The text a model learns from, and what becomes of it afterwards. Less is kept than most people assume, and what is kept has no sources attached.
Training data is the text a model learns from: the examples whose statistical patterns become the model's weights. Once training ends, the data itself is gone from the process; the model carries compressed tendencies of it, not copies of the pages. That is why facts learned this way cannot be cited, updated or removed, and why research tools put knowledge in a retrievable library instead, where every claim can point back to its source.
The examples the weights come from.
Training a language model means showing it an enormous amount of text and adjusting its parameters so that its predictions of what comes next get better; what perplexity is covers the standard measurement of that prediction skill. The text is the training data: web pages, books, articles, documentation, code, and curated sets mixed in for quality; the code in that mix is why small local models can write code at all. Nothing in the process stores the examples the way a database would; the result is the settled weights that run when you ask a question, and the data itself plays no further part. What the model knows, it knows as patterns absorbed from that mix. Distillation is a variation that changes the material: a teacher model's own outputs become the examples, and what model distillation is covers how that yields a smaller student with the teacher's habits.
What the model keeps, and what it loses.
What survives training is compression, not storage: the statistics of how text tends to continue. That is the mechanism behind confident fabrication, since a model can produce the shape of a fact, a citation or a date with the same ease as a correct one, and nothing in the output announces which is which. Heavily repeated text can survive as near-verbatim recall, but even then the model cannot attach a source to what it offers, because the source was never part of what it kept. Facts with provenance are a property of documents, not of weights.
Why knowledge belongs in the library instead.
This is the practical consequence for research work. Knowledge inside weights is frozen at training time, unverifiable at answer time, and immune to correction. Knowledge in documents stays current, and retrieval hands the model exactly the passages an answer should rest on, with the source one click away. That division, behavior in the weights and knowledge in the library, is the design behind retrieval-augmented generation, and it is also what makes structural verification possible: you can only check a claim against material the tool can actually open.