Every page on this site.

All 75 pages about local AI on istor.fyi, grouped by the kind of question each one answers. Every title and every line under it is the page's own, read from the page.

What it is (41)

The vocabulary: one page per term, each defined plainly and then placed against the thing it is usually confused with.

What does 7B mean in a model name?Seven billion parameters, the learned numbers that make up the model. What the count predicts about quality and memory, and why 7B to 8B is the sweet spot on consumer hardware.
What does Q4_K_M mean?The label on a quantized model file that names its compression recipe: 4 bits per weight, with a mix of precisions chosen to protect quality. How to read the code and which one to pick.
What formats can a local AI read?The common research formats, read on your machine: PDF including scans, DOCX, EPUB, HTML, Markdown and plain text. Why the format list is also a privacy list.
What is a benchmark in AI?A fixed set of tasks with a scoring rule for comparing models. Why benchmark scores are useful, why they get gamed, and why none of them describe how a model handles your documents.
What is a chat template?The text wrapper a model was trained to expect around a conversation. Why a model behaves strangely without it, where it lives in the model file, and how tools apply it for you.
What is a context window?A model's context window is everything it can read at once: your question, the passages retrieved, and the answer being written. What it bounds, how retrieval works around it, and why bigger is not automatically better.
What is a GGUF file?The single-file format local models ship in: weights and metadata in one download, quantization baked in, loadable by CPU and GPU runtimes. What is inside and why it won.
What is a knowledge base?An organized collection of information a tool can search and answer from. How a knowledge base differs from notes and from a chatbot's memory, and what makes one useful for AI answers.
What is a local LLM?A plain-language definition of a local LLM: a language model that runs on your own hardware, what it can and cannot do compared to cloud models, and why privacy follows from the design.
What is a local-first app?An app that keeps your data and its core work on your machine, with the network as an option rather than a requirement. What the pattern means, and why AI tools revived it.
What is a model card?The spec sheet a model ships with: what it is, what it was trained for, its limits and its license. What a model card tells you before you download, and what to check first.
What is a retrieval query?The text a search system actually runs against your library, rewritten from the question you asked. How queries are built, why wording matters to a machine, and what a good one earns.
What is a system prompt?The standing instructions a model reads before anything you type. What a system prompt does, how it differs from your messages, and why it shapes behavior more than facts.
What is a token limit?The cap on how much text a model can read and write in one request, counted in tokens. Where the limit comes from, what happens when you hit it, and how tools work around it.
What is a vector database?The index that lets search match by meaning instead of exact words: what vectors are stored in one, how similarity search works, and why your own files need no cloud service.
What is a vision language model?A language model that can also look at images. How a vision language model reads page images directly, where it beats plain OCR, and why its reading still needs checking.
What is AI hallucination?Confident output with no basis in fact, produced by a system built to continue patterns rather than check them. What hallucination is, why it happens, and whether it can be fixed.
What is an AI agent?A model arranged to plan, call tools and loop until a job is done, instead of answering in one pass. What makes an agent, what it costs, and where local fits.
What is an embedding?An embedding turns a passage of text into a list of numbers that captures its meaning, so search can find passages by sense instead of by exact words. What embeddings do, why retrieval needs them, and why they run fine on your own machine.
What is an inference engine?The software that turns a model's weights into answers: what an inference engine loads, computes and manages, why it decides speed, and how it sits under every local AI tool.
What is an open-weight model?The trained numbers are downloadable, so the model runs wherever you copy it. What open weights give you, what they still do not, and why the term is not the same as open source.
What is fine-tuning?Continuing a model's training on examples to change how it behaves. What it can adapt, what it cannot fix, and why a document library beats it for knowledge.
What is Hugging Face?The platform where the machine learning community publishes models, datasets and demos, mostly free to download. What it hosts, where local models come from, and what to check first.
What is hybrid search?Running keyword search and vector search together, then merging the results. Why each method fails alone, how the merge works, and why it runs fine locally.
What is inference?The act of running a trained model on your input to produce an answer. How inference differs from training, what it costs, and why it runs fine on your own machine.
What is llama.cpp?The open-source engine that runs language models on ordinary hardware: what it does, why it reads GGUF files, and why it sits underneath so many local AI tools.
What is MCP?The Model Context Protocol, the open standard connecting AI models to external tools. What it does, why it matters for local AI, and what it does not fix.
What is model distillation?Training a small model to imitate a larger one, so it inherits the teacher's behavior at a fraction of the size. How distillation works, what the student loses, and why so many local models get built this way.
What is OCR?Optical character recognition: how text is read out of images of pages. What OCR does, where it fails, and why it can run entirely on your own machine instead of an upload service.
What is Ollama?The free tool that runs language models on your own machine with a single command. What it bundles, what it is good at, and where you outgrow it.
What is perplexity in AI?The standard score for how well a language model predicts text. What a lower number means, why it never says whether an answer is right, and how to read it on a model card.
What is prompt engineering?Writing instructions a model follows well: what the practice actually involves, the habits that matter more than magic phrases, and where prompting ends and structure begins.
What is quantization?Storing a model's numbers in fewer bits so it fits in less memory. Why nearly every local model is quantized, what the accuracy trade actually is, and why the file size decides which GPU can run it.
What is reranking?A second pass over search results, where a model reorders candidates by true relevance. Why search is fast and rough, what a reranker reads, and what better ordering buys.
What is retrieval-augmented generation (RAG)?A plain-language definition of RAG: how retrieval-augmented generation works, what it fixes, what it does not, and how it runs entirely on your own machine.
What is semantic search?Search that matches meaning instead of wording. How it works, where it fails, and why it is the retrieval step that lets a local AI answer from your documents.
What is temperature in AI?The setting that decides how adventurous a model is when it picks each next word. What temperature does, where low and high settings belong, and why it is not creativity.
What is tokenization?How a language model breaks text into pieces before reading it: what a token is, why models count in them instead of words, and what that means for speed, limits and long documents.
What is tokens per second?The speed a model generates text, measured in the pieces it reads and writes. What the number means, what moves it up and down, and why it decides whether local AI feels usable.
What is training data?The text a model learns from, and what happens to it afterwards. Why the model carries compressed tendencies of its examples rather than copies, and why knowledge wants a library instead.
What is VRAM?The memory built into a graphics card, separate from system RAM, and where a local model wants to live. Why it decides generation speed, and what happens when the model does not fit.

How it works, and why it behaves that way (7)

The machinery behind an answer, and the behaviour that surprises people when a model gets something wrong.

How does document chunking work?The step that cuts long documents into the passages retrieval can find: why chunk size matters, what good boundaries preserve, and how it stays part of a local pipeline.
How much RAM does a local AI need?Less than most people expect: a quantized 7B model fits in about 5 gigabytes, so 16 GB of system memory is a comfortable setup. Where the numbers come from and what actually limits a machine.
Why do AI chatbots invent citations?Because a language model writes text that looks like a citation, it does not look one up. Why retrieval alone does not fix it, and what a real verification gate does instead.
Why does AI forget between chats?Because a language model has no memory, only a context window that empties when the conversation ends. What remembering actually is, and where the notes-on-you approach differs.
Why does AI forget the beginning of long documents?A model's attention is not uniform across a long context: the start and middle get less weight as text grows. Why it happens, what a bigger window does and does not fix.
Why does AI give different answers to the same question?Sampling: language models pick each next word by chance, weighted by likelihood. Why answers vary between runs, why that is a feature rather than a bug, and what makes answers repeatable when it matters.
Why is local AI slow on my laptop?The honest reasons a local model runs slow: model size, memory bandwidth, and one-token-at-a-time generation. What actually helps, what does not, and why slow never makes answers worse.

How to do it (3)

Tasks, start to finish.

How do you cite an AI answer?Treat the answer as a conversation with a tool, cite the sources you actually verified, and disclose what you asked. The principles that outlast any style guide's current rules.
How do you stop an AI from making things up?Grounded retrieval, citations that open passages, and a verification gate that refuses when sources lack the answer. What actually reduces AI hallucination, in order.
How to run an LLM locallyRunning a language model on your own machine takes three decisions: a tool, a model file that fits the hardware, and memory to hold it. What each one involves, and what the first answer feels like.

Whether it can (7)

Capability questions, answered with the limits attached rather than the headline alone.

Can a local AI read handwriting?Neat handwriting, mostly yes; hurried cursive, with real limits. What local OCR and vision models manage on your machine, and where human checking stays necessary.
Can a local AI read spreadsheets?Yes, with caveats: CSV reads as plain text, Excel needs its cells extracted first. What survives the conversion, what does not, and how to ask questions of a sheet.
Can a local AI summarize PDFs?Yes: a local model summarizes PDFs on your machine. The interesting part is what a summary should be: grounded, checkable against the passages it came from, and honest about length limits.
Can a local AI translate documents?Yes: translation is a core language task, and a model that runs on your machine does it without any upload. What quality depends on, which language pairs hold up, and what stays private.
Can a local AI write code?Yes, within limits. Small open models write working code from plain descriptions, with the caveats that matter: quality scales with size, review stays essential, and your code never leaves the machine.
Does local AI need a GPU?No: small models run on a plain CPU, just slower. What a GPU actually buys, where CPU-only is fine, and where research work wants even an entry-level card.
Is local RAG private?Local RAG keeps embeddings, retrieval and inference on your machine. What can still leave it, and the questions to ask of any tool before you trust it with sources.

Using it for your own work (9)

What it is like in a literature review, a lecture hall, a field notebook, or a PDF you would rather not upload.

Can a local AI do a literature review?Yes, and here is what that looks like: importing sources, asking grounded questions across the whole library, checking every claim against its passage, and keeping notes where the evidence lives.
Can a local AI read scanned PDFs?Yes: local OCR and a vision model can read scanned PDFs entirely on your machine, no upload. What works, what the limits are, and how scans join the same research loop as born-digital documents.
Can an AI do research with no internet?Yes, if the whole loop runs on your machine. What a local research AI can do with the network unplugged, what it honestly cannot, and when offline is the whole point.
Can you run a research AI on a GTX 1650?Yes. Quantized 2B to 4B models fit in 4 GB of VRAM, and a grounded research loop runs on that class of GPU. What fits, and how Istor uses it.
Is local AI more private than cloud AI?Yes, structurally: cloud AI must ship your text to a server, local AI runs on your machine. What the trust models are, what local gives up, and how to compare any tool.
Local AI for field research, offlineField research happens where the internet is unreliable and the material is sensitive. Here is how a local research notebook carries interviews, observations and photos through the whole analysis loop offline.
Local AI for lecture notesA semester of slides, transcripts and readings becomes one searchable library that answers with citations. What a local AI does for lecture notes, and what stays yours to learn.
What is the most private way to chat with PDFs?A checklist for asking questions of PDFs without uploading them: local inference, local embeddings, no telemetry, user-triggered network only. And how Istor satisfies each item.
Writing with sources on a local AIFor a thesis, a nonfiction book or a long report, the hard part is writing against sources without losing track of them. Here is how a local research notebook keeps the library beside the draft, with citations as you write.

Compared with other tools (7)

Side-by-side tables, including where the other tool wins.

Istor and AnythingLLM, comparedA factual comparison of Istor and AnythingLLM: two tools that run AI on your own documents locally, and the different bets they make.
Istor and ChatGPT, comparedA factual comparison of Istor and ChatGPT: a general-purpose cloud assistant with frontier-scale models, and a local research notebook where documents stay on your machine and every claim is checked.
Istor and GPT4All, comparedA factual comparison of Istor and GPT4All: both fully local, one built to chat on any CPU, the other built to ground research answers on a modest GPU.
Istor and Mendeley, comparedMendeley is a reference manager in the Elsevier ecosystem: it collects and formats sources. Istor is a local research notebook: it reads them and answers with citations. What each does, and why they are not rivals.
Istor and NotebookLM, comparedA factual comparison of Istor and NotebookLM: where your documents are processed, what hardware you need, and how citations work in each.
Istor and Obsidian, comparedObsidian is a local-first knowledge base of notes you write; Istor is a local research notebook that reads your sources and answers with citations. Two different local bets, and how they complement.
Istor and Zotero, comparedZotero is a reference manager: it collects and formats sources. Istor is a local research notebook: it reads them and answers with citations. Why the two are complementary, not rivals.

The project log (1)

What has shipped, newest first.

Istor project logWhat has shipped on Istor, newest first, dated from the public repository history. Versioned releases will be added here once they are cut.