Local LLMs: a guide from scratch

When someone uses ChatGPT, Gemini or Claude, the text they type leaves their computer, travels over the internet to a company's data center, gets processed there, and the answer comes back. A local LLM does the same job without that trip: the model is stored on your own computer and runs there, even with no internet connection. This article explains, from scratch, what that means, when it is worth it, which models are the best known and which one to pick for each task — including what I learned using a local model in a real project, EduData.

§1What an LLM is, without jargon

LLM stands for Large Language Model. In practice, it is a program that has learned to continue text. It was trained by reading an enormous amount of text — books, websites, code, papers — and in the process learned patterns: which words tend to follow which, how a question is usually answered, how code is usually written.

When you type “The capital of France is”, the model works out which continuations are most likely and picks “Paris”. A whole chatbot conversation is that, repeated many times: the model generates the answer one piece at a time. Those pieces are called tokens — something between a syllable and a word. A page of English text is roughly 500 to 700 tokens.

Three ideas help with everything else in this article:

• Parameters. The “adjustable numbers” inside the model — what it learned is stored in them. A 7B model has 7 billion parameters; a 70B one, 70 billion. Broadly, more parameters = more knowledge and better reasoning, but also more memory and slower answers.
• Weights. The file that holds those parameters. “Downloading a model” means downloading that file, which can be anywhere from 1 to over 100 GB.
• Context. How much text the model can “keep in mind” at once: the conversation so far, the document you pasted, the instructions. Measured in tokens (e.g. 8k, 32k, 128k).

A useful analogy: the model is like a student who read an entire library and then had the library taken away. They remember a lot, write well and reason, but they do not look things up — so sometimes they remember wrongly, with complete confidence. That has a name: hallucination, and it applies to both local and cloud models.

§2Local vs cloud: what is the difference

The famous assistants (ChatGPT, Gemini, Claude) run in the cloud: on huge machines with graphics cards that each cost as much as a car. A local LLM runs on your laptop, your desktop or your company's server. That is only possible because several companies and labs publish their models' weights for anyone to download — so-called open models (or open-weight models).

In short: the cloud wins on peak quality and convenience; local wins on privacy, predictable cost, offline use and control. It is not a team sport — many people use both, each for what it does best.

§3What it is for, and when to use it

A local LLM does almost everything a chatbot does: answer questions, summarize, translate, rewrite, proofread, explain code, write code, pull information out of documents and classify text. The right question is not “what can it do” but when is it the best choice.

Local is worth it when:

• The data is sensitive. Medical records, contracts, customer data, internal documents, anything covered by privacy law. If the text cannot leave the organization, the cloud is already out.
• The volume is high and repetitive. Classifying 50,000 comments, extracting fields from thousands of PDFs, summarizing an archive of meeting minutes. In the cloud every call costs money; locally, the cost is electricity.
• There is no reliable internet — in the field, on a plane, on a closed corporate network.
• You need predictability. A system that depends on a model should give the same answer tomorrow. With the model file stored locally, it does not change on its own.
• You want to learn. Running a local model is the best way to understand, hands-on, how these tools work and where they fail.

The cloud is better when:

• The task needs the strongest reasoning available: hard math, complex programming across large projects, long and subtle analysis.
• You need up-to-date information and built-in web search.
• Your computer is modest and the task is occasional — paying per use is cheaper than buying hardware.
• Many people will use it at once and nobody wants to run a server.

§4What you need to run one

The deciding factor is memory. The whole model has to fit in memory to run at a reasonable speed — ideally in the graphics card's memory (VRAM), which is much faster; if it does not fit, it falls back to regular memory (RAM) and the processor, and gets much slower.

This is where the trick that made all of this possible comes in: quantization. Originally each parameter takes 16 bits. Quantizing stores it with less precision — 8, 5 or 4 bits — a bit like saving a photo as JPEG instead of RAW: the file gets much smaller and the difference is barely noticeable. A 7B model that would take about 14 GB drops to around 4–5 GB at 4 bits. The names you will see, such as Q4_K_M or Q8_0, give the level: Q4 is the most common balance; Q8 is nearly identical to the original but twice the size.

A rule of thumb: at 4 bits, each 1 billion parameters takes about 0.6 GB, plus 1–2 GB of headroom for the conversation's context.

Two practical notes. Apple-silicon Macs (M1 onwards) are surprisingly good at this, because memory is shared between processor and graphics — a 32 GB Mac runs models that would need an expensive graphics card on a PC. And you can run without a graphics card: in EduData, the 7B model runs on the processor alone. It works, but each answer takes seconds instead of fractions of a second.

§5The tools: Ollama and friends

The model is just a file. To talk to it you need a program that loads and runs it. The best known:

• Ollama — the most popular and the simplest way to start. It installs like any program (Windows, Mac or Linux) and downloads models with one command. It also starts a small server on your computer (at localhost:11434), so other programs can use the model — that is how EduData talks to Qwen. It has its own chat window too.
• LM Studio — the friendliest option if you would rather avoid the terminal. It has a graphical interface, built-in model search, and tells you whether a model fits your computer before you download it.
• llama.cpp — the “engine” behind many of these tools (Ollama started on top of it). More technical, but the most flexible and efficient. It popularized the GGUF file format, the standard for quantized models.
• Jan and GPT4All — open-source desktop apps focused on privacy and simplicity.
• Open WebUI — a ChatGPT-like interface that runs in the browser and connects to Ollama. Good for a family or a small team.
• vLLM — for servers: serves many people at once efficiently. It is what companies use to put an open model into production.

Getting started with Ollama takes a few minutes. Once installed, in a terminal:

ollama pull qwen3:8b      # download the model (~5 GB)
ollama run qwen3:8b       # open a chat in the terminal
ollama list               # list downloaded models
ollama rm qwen3:8b        # delete a model to free up space

What comes after the colon is the tag: the size and sometimes the quantization. qwen3:8b is Qwen3 with 8 billion parameters; qwen3:4b would be the smaller version. The full library is at ollama.com/library.

§6The best-known model families

Open models come in families: a company releases several versions in different sizes, sometimes with specialized variants (for code, images or reasoning). The main ones:

• Qwen (Alibaba). Probably the most complete open family today. It has everything: tiny models (0.6B) and huge ones, versions for code (Qwen-Coder), for images (Qwen-VL) and for embeddings. It is strong in many languages, including Portuguese, and very good at following strict formats such as JSON. Qwen2.5 established the family; Qwen3 added a “think before answering” mode that can be switched on and off. It is what I use in EduData.
• Llama (Meta). The family that made open models popular, in 2023. Llama 3 (8B and 70B, then versions 3.1–3.3) became a reference point with a huge ecosystem of community fine-tunes. Llama 4 brought larger models, heavier for home computers.
• Gemma (Google). Open, compact versions of the technology behind Gemini. Gemma 3 (1B to 27B) understands images and does very well for its size; there are also variants built for phones.
• Mistral (Mistral AI, France). Known for efficient models. Mistral Small (about 24B) is a great middle ground, and Mistral Nemo (12B) was popular for its quality in European languages. There is also Codestral, for code.
• DeepSeek (China). Became famous with DeepSeek-R1, a reasoning model that competed with the best in the world. The original is far too large for home use, but there are “distilled” versions — smaller models trained to imitate it — that run in Ollama.
• gpt-oss (OpenAI). OpenAI's first open-weight models since GPT-2. The 20B version runs on a computer with about 16 GB of memory and is good at reasoning and tool use.
• Phi (Microsoft). Small models trained on carefully selected data. Phi-4 (14B) and Phi-4-mini stand out in reasoning and math for their size, but are weaker in general knowledge and in languages other than English.

Beyond these, there are models that do not chat but are essential building blocks: embedding models (such as nomic-embed-text, bge-m3 and qwen3-embedding), which turn text into lists of numbers so you can search by meaning, and Whisper (OpenAI), which transcribes audio into text and also runs locally.

§7Which one is good for what

The table below is a starting point. The best choice always depends on your computer: start with the largest model that runs comfortably on your machine and test it on your own tasks.

Some terms that show up in model names:

• instruct or chat: tuned to converse and follow instructions — almost always what you want. The base model only continues text and is meant for people who will train it further.
• coder, vl/vision: specialized in code or images.
• MoE (mixture of experts), as in qwen3:30b-a3b: the model has 30B in total but uses only 3B per token. It takes the memory of a big model but answers at the speed of a small one.
• distill: a smaller model trained to imitate a larger one.

§8In practice: the EduData case

In EduData, a query engine over Brazil's INEP higher-education statistics, I tested whether a local model could understand questions in Portuguese and turn them into a structured query. The model never computes anything: it only returns a JSON saying what was asked, and the code validates that JSON and fetches the numbers from the database.

On 40 unseen questions, Qwen2.5 7B got 27 right; Qwen2.5 3B, 21. Neither matched the hand-written rules. So the model stayed optional, with a warning telling the user to check the interpretation. Three lessons that apply to any project:

• Measure before you trust. “Looks good” in a chat is not the same as being right 9 times out of 10. Build a small set of questions with correct answers and test.
• Size matters more than speed when mistakes are costly: the 7B made far fewer errors than the 3B.
• Let the model interpret and the code compute. LLMs are great with language and bad at arithmetic and exact facts. Using them only for what they do well — and checking their output — is what makes a system reliable.

§9Limitations and caveats

• They make things up. Hallucination happens in every model, more often in small ones. Do not take an answer as fact without checking, especially numbers, names, laws and citations.
• Frozen knowledge. The model only knows what was in its training data, up to a certain date. It does not know what happened yesterday unless you put that information in the context.
• Context has a cost. Many models advertise 128k tokens of context, but every extra token takes memory. Ollama's default is much smaller and may need raising for long documents.
• Licenses vary. “Open” does not mean “no rules”. Qwen and Mistral usually ship under Apache 2.0 (very permissive); Llama and Gemma have their own licenses with conditions. For commercial use, read the license.
• Download from trusted sources. Prefer Ollama's official library or the labs' official Hugging Face pages.
• Privacy depends on the whole stack. The model is local, but the app around it may not be. Check that the interface you use does not send data elsewhere.

§10How to start today

• 1. Check your memory. On Windows: Task Manager → Performance (RAM and, under “GPU”, the graphics card's dedicated memory).
• 2. Install Ollama (ollama.com) or, if you prefer something fully visual, LM Studio.
• 3. Choose by size: 8 GB RAM → qwen3:4b or gemma3:4b; 16 GB → qwen3:8b; a 12 GB+ graphics card → qwen3:14b or gemma3:12b.
• 4. Test with your own real tasks — an email to rewrite, a text to summarize, some code to explain — and compare with a cloud service.
• 5. Swap and compare. Downloading another model is one command. The best way to choose is to test two or three on the same questions.

§11Takeaways

• A local LLM is a language model that runs on your computer: data never leaves it and it works without internet.
• It is worth it when there is sensitive data, high volume, no internet or a need for control; the cloud is still better for the hardest reasoning and for people who just want to open it and go.
• Memory decides what runs; quantization (Q4) makes a 7–8B model fit in about 5 GB.
• Ollama is the simplest way in; Qwen, Llama, Gemma, Mistral, DeepSeek, gpt-oss and Phi are the main families, and Qwen is the most versatile choice today, including for Portuguese.
• No model removes the need to check: test on your own tasks, measure accuracy, and leave anything that must be exact to code.

References: Ollama library; llama.cpp; LM Studio; Qwen on Hugging Face. EduData results: decision D17 in the project's _arquitetura/02_decisoes.md. Memory and sizes are approximations for 4-bit quantization.