Before you start: the numbers that decide it
Local LLM hardware requirements come down to memory more than anything else. A model has to sit in memory while it answers, and the fastest place for it is VRAM, the memory on a graphics card. How much VRAM for an LLM depends on two numbers in the model's name and one setting you choose.
Find out what your computer has before you pick a download. On Windows, the Performance tab of Task Manager lists each GPU with its dedicated memory. On a Mac, About This Mac shows the memory, which the CPU and GPU share. On Linux with an NVIDIA card, nvidia-smi prints the card's memory. Write the numbers down.
If you followed lesson 1 or lesson 3, you already met these numbers as tags and download options. This lesson turns them into gigabytes.
- Parameters: the learned numbers inside a model, counted in billions. The 8B in a name means about 8 billion.
- Bits per weight: how many bits store each parameter. A 16-bit file uses 16, and the common 4-bit files use a little under 5.
- Context length: the maximum number of tokens the model holds in memory, in the words of Ollama's docs.
- Unified memory: one pool of RAM that the CPU and GPU share on Apple silicon Macs.
- Offloading: running part of a model on the CPU, from system RAM, when it does not fit in VRAM.
Work out how much memory a model needs
The weights set the floor. Multiply the parameter count by the bits per weight and divide by eight, because eight bits make a byte. For an 8B model at Q4_K_M, that is 8 billion times 4.89 bits over eight, a little under five billion bytes.
The quantize README from llama.cpp gives the same answer for Meta's Llama 3.1 at three sizes, in the table below. It also says that memory and disk requirements are the same, because the whole model loads into memory. That makes the download size next to each option a fair first guess at the memory it takes.
The conversation needs room too. Every token in the context keeps data in a KV cache, short for key-value cache, which stores what the model already worked out about earlier tokens. Ollama's context length page says a larger context increases the memory a model needs. By default Ollama picks the context from your VRAM: 4k tokens under 24 GiB, 32k from 24 to 48 GiB, and 256k above that.
That cache can shrink. Ollama's FAQ lets you store it at 8 bits with OLLAMA_KV_CACHE_TYPE=q8_0, about half the memory of the default f16, or at 4 bits with q4_0, about a quarter. Both need Flash Attention, a way of computing attention that saves memory as the context grows. Ollama turns it on by itself when the hardware supports it.
Parallel requests multiply the cache. The FAQ says required RAM scales with the number of parallel requests times the context length. Its example is a 2K context with 4 parallel requests, which becomes an 8K context.
Then leave a margin. Hugging Face's memory estimator page says its numbers cover loading only, and that inference can add up to 20% more, a figure it credits to EleutherAI. Its tool, accelerate estimate-memory, reads models in the Transformers library format rather than GGUF files. The margin still applies to both.
LM Studio can run the sum for you. Its lms load command takes --estimate-only, which prints a memory estimate and exits without loading the model.
# memory for the weights alone: parameters x bits per weight / 8
def weights_gb(params_billion, bits_per_weight):
return params_billion * bits_per_weight / 8
# bits per weight from llama.cpp's quantize README (Llama 3.1 8B)
for name, bpw in [("F16", 16.0005), ("Q8_0", 8.5008), ("Q4_K_M", 4.8944)]:
print(name, round(weights_gb(8, bpw), 1), "GB before context")- Read the parameter count from the model name, such as 8B or 70B.
- Pick a quantization and note its bits per weight, such as 4.89 for Q4_K_M.
- Multiply the two and divide by eight to get the weights in gigabytes.
- Add room for the context you plan to use, plus a margin for inference.
- Compare the total with your VRAM, or your unified memory on a Mac. If it does not fit, step down one quantization or one model size.
| Model size | Original size | Size at Q4_K_M |
|---|---|---|
| 8B | 32.1 GB | 4.9 GB |
| 70B | 280.9 GB | 43.1 GB |
| 405B | 1,625.1 GB | 249.1 GB |
Quantization: GGUF Q4, Q8 and the levels between
Quantization stores each weight with fewer bits. The llama.cpp README describes it as reducing the precision of the weights, for example from 32-bit floats to 4-bit integers. GGUF files carry the result, and the label in a file name, such as Q4_K_M or Q8_0, names the method.
The number after Q is the rough bit width. A K marks the k-quant family, which groups weights into super-blocks that share their scales. Hugging Face's GGUF page lists Q4_K at 4.5 bits per weight and Q6_K at 6.5625. It files Q8_0 and Q4_0 under legacy types, which round each weight to the nearest step. Names that start with IQ use an importance matrix, data on which weights matter most.
Real files land a little above the nominal width, because the scales take space too. Check the file size, not the label. The S, M and L versions of one level differ slightly in size. The chart below shows llama.cpp's figures for that 8B model, in GiB. A GiB is 1,073,741,824 bytes, a little more than a GB.
Fewer bits can cost quality. The README says quantization may introduce some accuracy loss, usually measured with perplexity or KL divergence. Perplexity scores how well a model predicts real text. KL divergence measures how far the quantized model's predictions drift from the original's. The README adds that a suitable importance matrix file can reduce the loss, and it gives no such scores for these types.
Both apps start you near the middle. LM Studio's download page advises a 4-bit option or higher when your machine can run it. When Ollama pulls a GGUF repository from Hugging Face with no tag, Hugging Face's Ollama page says it uses Q4_K_M if the repository has one.
The README lists speeds for each type too, but without naming the hardware, so plan with the sizes.
Source: llama.cpp: quantize README (read November 5, 2026)
CPU, GPU or Apple silicon
Where the model runs decides how fast it answers. A GPU works through the model's math in parallel, so a model that fits entirely in VRAM replies fastest. Ollama's context length page says that for best performance you should avoid offloading the model to the CPU.
When a model does not fit, both tools split it. The llama.cpp README calls this CPU+GPU hybrid inference, which partially accelerates models larger than the total VRAM. Ollama's quickstart says that with less VRAM it can use system RAM, but responses may be slower. Run ollama ps and read the PROCESSOR column: 100% GPU, 100% CPU, or a mix such as 48%/52% CPU/GPU.
A computer with no supported GPU still runs models, from system RAM on the CPU. llama.cpp supports the AVX, AVX2, AVX512 and AMX instruction sets on x86 processors. LM Studio requires AVX2 on x64 Windows machines. The same arithmetic applies, with system RAM as the limit.
Apple silicon Macs work differently. The CPU and GPU share unified memory, so a Mac's memory size plays the part that VRAM plays on a PC. That is why Ollama's quickstart asks for 8 GB of available VRAM, or unified memory on a Mac. llama.cpp calls Apple silicon a first-class citizen, optimized through Metal, Apple's graphics interface. Ollama runs on the CPU and GPU of M-series Macs and on the CPU alone on Intel Macs.
On a PC, check that your card is supported. Ollama's GPU page lists NVIDIA cards with compute capability 5.0 or higher and driver 550 or newer, though the oldest of them, up to 6.2, need driver 570. AMD cards need the ROCm v7 driver. Vulkan, a graphics interface that works across vendors, covers other GPUs on Windows and Linux.
LM Studio's requirements give a floor for its app: 16GB of RAM and 4GB of dedicated VRAM recommended on Windows, and 16GB or more on a Mac.
| Hardware | Ollama | llama.cpp backend |
|---|---|---|
| NVIDIA GPU | Compute capability 5.0+ and a recent driver | CUDA |
| AMD GPU | ROCm v7 driver | HIP |
| Apple silicon Mac | CPU and GPU, through Metal | Metal |
| Intel Mac | CPU only | CPU |
| Other GPUs on Windows or Linux | Vulkan | Vulkan |
| No supported GPU | CPU and system RAM, slower | CPU, with AVX up to AMX on x86 |
Why Dan Woods's team went back from Q2 to Q4
Dan Woods is Vice President of AI Platforms at CVS Health. In a reply on X in March, he said tool calling was the metric that took them from 2-bit quantization back to 4-bit. At Q2 the model was great in conversation, he wrote, but too unpredictable for work that needed a set result.
Paul Gauthier made the same point on X in November 2024, and the test he linked still works as a method. On Aider's blog, Qwen 2.5 Coder 32B finished 71.4% of a code editing benchmark at fp16 through Ollama, 66.9% at q4_K_M and 61.7% at q2_K. At fp16 with Ollama's default context of the time, 2k tokens, it fell to 51.9%, because longer prompts were cut without an error.
So size a model with the arithmetic above, then test the file on your own tasks. If an agent calls tools or edits code, run a few of those jobs at Q4_K_M and at Q8_0 before you settle on the smaller one. Set the context high enough that no prompt gets cut.
"tool calling" became the metric that got us back to Q4.
Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. Select one to open its post on X.
Common mistakes
- Reading the download size as the whole memory bill. The context and the runtime need room on top of the weights.
- Raising the context length to the maximum on a small card. A longer context grows the KV cache and can push the model onto the CPU.
- Comparing GB and GiB figures as if they were one unit. A GiB is about 7% larger.
- Turning up parallel requests without checking memory. Each parallel slot adds a full context to the cache.
- Picking the smallest quantization of a big model without checking its answers. The README warns that quantization may cost accuracy, so try it on your own questions first.
- Expecting an Intel Mac to use its GPU. Ollama runs on the CPU there.
Try it yourself
- Write down your VRAM and RAM, then work out which of the three Llama sizes in the table fit at Q4_K_M.
- Run the Python helper for a model you want, at Q8_0 and at Q4_K_M.
- Load a model in Ollama and read the PROCESSOR column in
ollama ps. - Start Ollama with
OLLAMA_CONTEXT_LENGTH=64000 ollama serve, the size its docs suggest for agents and coding, and compareollama pswith the default. - In LM Studio, load one model with
lms loadand--gpu 0.5, then with--gpu max, and compare how fast it replies. - Run
lms loadwith--estimate-onlyon a model you downloaded, and compare its estimate with your own sum.
Next: the Hugging Face course
Every file you sized in this course came from one place. The Hugging Face course starts with the Hub, where models, GGUF files and their model cards live.
Check yourself
About how much memory do the weights of an 8B model at Q4_K_M take?
Show answer
D. About 5 GB 8 billion weights at about 4.9 bits each, divided by eight, comes to just under 5 GB, which matches llama.cpp's table.
What does 100% CPU in the PROCESSOR column of ollama ps mean?
Show answer
A. The model was loaded entirely into system memory 100% CPU means the model sits entirely in system memory, which answers more slowly than the GPU.
Which setting cuts the memory of Ollama's KV cache to about half?
Show answer
C. OLLAMA_KV_CACHE_TYPE=q8_0 An 8-bit KV cache uses about half the memory of the default f16, and it needs Flash Attention.
Why is the download size not the full memory a model needs?
Show answer
B. The context and the runtime need room on top of the weights The weights set the floor, and the context, parallel requests and inference overhead all add memory.
Frequently asked questions
How much VRAM do I need to run an LLM?
Enough for the weights and the context together. Ollama recommends 8 GB of VRAM for its quickstart model, a download of about 7.2 GB. A 70B model at 4 bits is around 43 GB, which takes a card or a Mac with that much memory, or a split onto the CPU.
Is a 4-bit model much worse than a 16-bit one?
The sources give no single score. llama.cpp says quantization may cost some accuracy, and LM Studio suggests 4-bit or higher. In llama.cpp's table the 4-bit 8B file was under a third the size of the 16-bit one, so compare both on your own prompts if you have the memory.
Can I run an LLM without a GPU?
Yes. Ollama and llama.cpp both run models on the processor from system RAM, and ollama ps shows 100% CPU when that happens. Replies come slower, so start with a small model and a 4-bit file.
Does unified memory on a Mac count as VRAM?
For sizing, yes. The M-series GPU reads the same memory as the processor, which is why Ollama treats a Mac's unified memory like VRAM. macOS and your open apps use part of it, so leave a margin. On an Intel Mac, Ollama uses the processor only.
Sources
- llama.cpp README https://github.com/ggml-org/llama.cpp
- llama.cpp: quantize README https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
- Hugging Face Docs: GGUF https://huggingface.co/docs/hub/gguf
- Hugging Face Docs: Use Ollama with any GGUF model on Hugging Face Hub https://huggingface.co/docs/hub/ollama
- Hugging Face Accelerate: Model memory estimator https://huggingface.co/docs/accelerate/usage_guides/model_size_estimator
- Ollama Docs: Quickstart https://docs.ollama.com/quickstart
- Ollama Docs: Context length https://docs.ollama.com/context-length
- Ollama Docs: FAQ https://docs.ollama.com/faq
- Ollama Docs: Hardware support https://docs.ollama.com/gpu
- Ollama Docs: macOS https://docs.ollama.com/macos
- Ollama library: llama3.1 https://ollama.com/library/llama3.1
- LM Studio Docs: System requirements https://lmstudio.ai/docs/app/system-requirements
- LM Studio Docs: Download an LLM https://lmstudio.ai/docs/app/basics/download-model
- LM Studio Docs: lms load https://lmstudio.ai/docs/cli/local-models/load
- Dan Woods on X: tool calling got us back to Q4 https://x.com/danveloper/status/2034686509748462022
- Paul Gauthier on X: pay attention to how your open models are quantized https://x.com/paulgauthier/status/1859684310204473349
- Aider blog: Details matter with open source models https://aider.chat/2024/11/21/quantization.html