Before you start
Open a notes file and write two lines. The first is the job, such as "summarize 20-page PDFs" or "answer questions about our support tickets". The second is your limit: the memory your computer has, which the hardware lesson shows how to find, or the server you can use.
The model names below are the ones the cards used on 6 November 2026. Makers release new versions every few months. Read the names as examples, and look at the publisher's page for what is current.
Choose a model in six steps
Each step removes candidates, so you spend your testing time on two. The evals lesson shows how to grade the answers in step 5.
- Write the job in one sentence, with the longest input you will give it. "Summarize a 40-page PDF" tells you more than "chat".
- Filter on Hugging Face. The Hub lets you filter models by license, and the tag
library=gguflists models stored as GGUF files, a format Ollama can pull from Hugging Face. - Open each card and note four things: total parameters, active parameters, context window and license. Lesson 2 shows how to read the license.
- Estimate the memory for the weights from the total parameters, then leave room for the context. The hardware lesson has the arithmetic, and lesson 4 covers the quantization that shrinks it.
- Write five prompts from your own work, with one that is awkward or long. Run them on the two models that fit, with the same settings for both.
- Pick the smaller model that does the job, and write down its name, version and the date. The bigger one can wait until a prompt fails.
Total parameters set memory, active ones set speed
Six of the seven models in the chart below are mixture-of-experts models. Hugging Face's post on the design explains that sparse layers replace the dense layers, and a router decides which expert receives each token. A token is a piece of a word, the unit a model reads and writes.
What you give up is memory. The post says Mixtral 8x7B needs enough VRAM to hold a dense 47B model, yet it runs with the compute of a 12B one. Every expert has to be loaded. Memory stays high while the work per token drops.
Cards give both numbers. Gemma 4 26B A4B has 25.2B parameters and uses 3.8B per token, and its card says it runs almost as fast as a 4B model. Qwen3.8-27B is dense, which means all 27B parameters work on every token.
Plan memory from the total and expect speed from the active count. Llama 4 Scout needs room for 109B parameters but computes with 17B per token, fewer than the 27B that a dense model such as Qwen3.8-27B uses each time. A low active count does not mean a small download.
Sources: Hugging Face: openai/gpt-oss-20b model card (read November 6, 2026), Hugging Face: google/gemma-4-26B-A4B-it model card (read November 6, 2026), Hugging Face: deepseek-ai/DeepSeek-V4.1-Flash model card (read November 6, 2026), GitHub: Meta Llama 4 model card (read November 6, 2026), Hugging Face: Qwen/Qwen3.8-27B model card (read November 6, 2026), Hugging Face: mistralai/Mistral-Large-4-1T-A52B model card (read November 6, 2026)
Read the context length as a ceiling
The context length is how many tokens a model can hold at once, counting your prompt, the documents you add and its own answer. The table lists what five cards said on 6 November 2026.
A big number is a ceiling, not a promise. Holding a long context takes memory on top of the weights, and accuracy can fall well before the window is full. The context rot lesson measures that drop, so test with the longest input your job produces.
| Model | Context length on its card | Note |
|---|---|---|
| Qwen3.8-27B | 262,144 tokens | The native length. The card says it can be extended to 1,000,000. |
| Gemma 4 12B | 256K tokens | The E2B and E4B sizes list 128K. |
| DeepSeek-V4.1-Flash | 1M tokens | The card says it supports contexts of up to one million tokens. |
| Mistral Large 4 | 1M tokens | The card lists 1M context length. |
| Llama 4 Scout | 10M tokens | Llama 4 Maverick lists 1M. |
A rank is a shortlist, not a verdict
Hugging Face retired its Open LLM Leaderboard on 13 March 2025, after two years and more than 13K evaluated models. The reason is quoted below.
It also says over 200 community-led leaderboards exist on Hugging Face, so rankings are still there to find candidates, as long as you ask which task each one measures. A model card has the same limit, because the maker picks which benchmarks to show. Mistral Large 4's card cites the Artificial Analysis Coding Agent Index, where it scores 48%, tied with GLM-5.3, and the Intelligence Index v5, where it scores 26. Those are scores on tests Mistral chose, and whether they match your job is for you to find out.
we feel it could encourage people to hill climb irrelevant directions in the field.
Common mistakes
- Sizing memory from the active parameters. The weights of every expert load, so the total is the number that has to fit.
- Testing with a short prompt only. A model that handles a paragraph can still lose a fact in a 40-page input.
- Trusting the biggest context number. Meta lists 10M tokens for Scout, and no card number says how well a model uses a long input.
- Changing two things at once. Give both models the same prompts and the same settings, so a difference comes from the model.
- Choosing before you read the license. A model that wins your test can still be one you may not use, and lesson 2 has the five questions.
Try it yourself
- Write the task and the memory your computer has, in two lines.
- Open the cards for Qwen3.8-27B and Gemma 4 26B A4B and note the total parameters, active parameters, context length and license of each.
- Use the hardware lesson's arithmetic to see which of the two fits in your memory.
- Collect five prompts you have sent to a chatbot and keep them in a file, since lesson 4 asks you to test a quantized file against them.
Next: quantization explained
A model that looks too big for your memory is not always out. Lesson 4 shows how a file name such as Q4_K_M shrinks it, and what the smaller file costs you.
Check yourself
Llama 4 Scout lists 109B total and 17B active parameters. Which number tells you how much memory its weights need?
Show answer
C. The 109B total number Every expert loads into memory, so the total sets the size of the weights. The active count sets how much work each token takes.
A model card shows strong benchmark scores for a model you like. What should you do next?
Show answer
A. Run five prompts from your own work on it and on one rival Makers choose which benchmarks to show, and the Open LLM Leaderboard team called its own ranking "slowly becoming obsolete".
Llama 4 Scout's card lists a 10M context length. What does that number tell you?
Show answer
D. The most tokens it can hold at once, which you still have to test It is a ceiling. Long inputs take memory beyond the weights, and accuracy can fall well before the window is full.
What does Hugging Face's mixture-of-experts post say Mixtral 8x7B needs?
Show answer
B. VRAM to hold a dense 47B model, with the compute of a 12B one The post says the rest of the model parameters are shared, so the total is 47B and not 56B, and each token computes like a 12B model.
Frequently asked questions
Is a bigger open model always better?
No. A bigger model needs more memory, and a dense one is slower on every token. Whether it also answers your prompts better is something only your own test can show.
How often should I repeat the choice?
When a maker publishes a new version, since names and numbers change every few months. Keep the five prompts in a file and rerun them. Each card lists its license and size, so you can check those first.
What do names like 17Bx16E and 26B A4B mean?
In Meta's name 17Bx16E, the card lists 17B activated parameters and 16 experts. In Gemma 4's 26B A4B, the A stands for active parameters. The first number in a name is rarely the one that sets speed.
Can I trust the benchmark scores on a model card?
They are the maker's own picks, so use them to build a shortlist. Run your own five prompts before you decide, since a benchmark can measure something your job never does.
Can I run a model that is bigger than my memory?
Sometimes. The hardware lesson covers offloading, which runs part of a model from system RAM, and a smaller quantization shrinks the file too. Both have limits, so check the fit before you download.
Sources
- Hugging Face blog: Mixture of Experts explained https://huggingface.co/blog/moe
- Hugging Face: Open LLM Leaderboard discussion 1135 (the end of the leaderboard) https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/1135
- Hugging Face: openai/gpt-oss-20b model card https://huggingface.co/openai/gpt-oss-20b
- Hugging Face: google/gemma-4-26B-A4B-it model card https://huggingface.co/google/gemma-4-26B-A4B-it
- Hugging Face: google/gemma-4-12B-it model card https://huggingface.co/google/gemma-4-12B-it
- Hugging Face: Qwen/Qwen3.8-27B model card https://huggingface.co/Qwen/Qwen3.8-27B
- Hugging Face: deepseek-ai/DeepSeek-V4.1-Flash model card https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
- Hugging Face: mistralai/Mistral-Large-4-1T-A52B model card https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B
- GitHub: Meta Llama 4 model card https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md
- Hugging Face Docs: Model Cards https://huggingface.co/docs/hub/model-cards
- Hugging Face Docs: GGUF https://huggingface.co/docs/hub/gguf
- Hugging Face Docs: Use Ollama with any GGUF model https://huggingface.co/docs/hub/ollama