What is tokenization?
The step before everything else. How text becomes pieces a model can read, and why the pieces matter more than people expect.
Tokenization is how a model breaks text into pieces before reading it. Each token is a short chunk, usually a few characters, and the model never sees letters or words directly, only the tokens they became. Every limit that governs an answer, from the size of the context window to the speed of generation, is measured in tokens, which makes this invisible step the unit of account for everything a model does.
Pieces, not words.
A model cannot read. It reads numbers, and tokenization is the mapping from text to those numbers. Each token is a short chunk of text, a common word in one piece, a rarer word split into parts, punctuation and spaces folded in as the scheme allows. The mapping is fixed per model: the same text always becomes the same tokens. A rough rule of thumb says one token is about three quarters of an English word, so a thousand words cost around thirteen hundred tokens, and a long research paper is a few thousand of them. This is also the unit underneath the context window, which is measured in tokens, not words or pages.
Why not just use words?
Splitting on whole words leaves holes a research tool cannot afford. Names, numbers, chemical formulas, citations in other languages: a word-level scheme would meet each one as an unknown, or would need an unmanageable dictionary. Splitting on single characters goes the other way, stretching every sentence into long sequences that take longer to process. Tokens sit in between and get the useful property: anything written can be represented, common words cost one piece, and the rare or foreign text still costs only a few pieces rather than failing outright. For a library of PDFs, where terminology is dense and multilingual sources are normal, that coverage is the difference between text that can be indexed and text that cannot.
What it costs you.
Tokens are the currency of every limit a model has. The context window holds a fixed number of them, generation produces a few dozen per second on local hardware, and the memory needed grows with the count. Two practical consequences follow. First, long documents do not fit whole, which is why summaries of long PDFs work in sections rather than in one pass. Second, text that tokenizes inefficiently, including many non-English languages, dense technical notation, pays more tokens for the same content, so the effective window is smaller than its label suggests. None of this is a reason to worry; it is a reason the budget exists.