Llama-3.2-3B-Instruct-GGUF

Llama-3.2-3B-Instruct-GGUF

GGUF quantizations of Meta's Llama-3.2-3B-Instruct, made by bartowski with an importance matrix, from about 1.8 GB to 6.4 GB. For running a small chat model locally with llama.cpp or LM Studio.

Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF quantizations of Meta's Llama-3.2-3B-Instruct, a 3.2B-parameter chat model. Bartowski made the files with llama.cpp and an importance matrix. Many quant levels are offered, so you can match the file to your memory.

Who it suits: People who want a small chat model on a laptop or desktop with no cloud account. The page lists English, German, French, Italian, Portuguese, Hindi, Spanish and Thai.

Industry | Who uses it | How they use it | Example
Local AI | Hobbyists | Chat with a small model on their own computer | Load a mid-size quant in LM Studio and ask questions
Software | Developers | Run a light instruction model inside a local app | Serve it with llama.cpp for a prototype
Mobile and ARM | Edge developers | Use quants made for ARM chips | Pick the ARM-tuned file for a board that supports it
Education | Students | Study how quantization changes size and quality | Compare Q8_0 and Q4_K_M on the same prompt

You can skip it if you need long reasoning or a large model. A 3B model has limits that a larger one does not.

Where to use it:

• LM Studio, which the card names for running the files
• llama.cpp and other tools that read GGUF
• Windows, macOS and Linux with Python, for downloading

Use case | Who | What they get back
Local chat | Hobbyists | Text replies from a 3B model on your machine
Low-memory use | People with little RAM | A small quant that still runs
ARM devices | Edge developers | A quant tuned for ARM inference

How to set it up: Download one file, not the whole branch, as the card advises.

• Python and the Hugging Face tool, or the download button in LM Studio
• A program that reads GGUF, such as LM Studio or llama.cpp
• Memory for the quant you choose, from about 1.8 GB to 6.4 GB

1. Choose a quant from the file list below.
2. Download that one file.
3. Open it in LM Studio or llama.cpp.
4. Use the prompt format shown for the model.

pip install -U huggingface_hub
hf download bartowski/Llama-3.2-3B-Instruct-GGUF

How to use it: Load a file in LM Studio and chat, or run it in llama.cpp with the prompt format below. The command above fetches the whole repository, so narrow it to one file.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Quant | Size and role
f16 | 6.43 GB, the full F16 weights
Q8_0 | 3.42 GB, the largest quant, rarely needed
Q6_K_L | 2.74 GB, Q8_0 for embedding and output weights, near-perfect quality
Q6_K | 2.64 GB, near-perfect quality and recommended
Q5_K_M | 2.32 GB, high quality and recommended
Q4_K_L | 2.11 GB, Q8_0 for embedding and output weights, good quality
Q4_K_M | 2.02 GB, the default size for most uses
Q4_K_S | 1.93 GB, a little lower quality with more savings
Q4_0_4_4 | 1.92 GB, tuned for ARM chips and the safe ARM pick
Q3_K_XL | 1.91 GB, lower quality but usable on low RAM
IQ4_XS | 1.83 GB, decent quality and smaller than Q4_K_S

Example: You download the Q4_K_M file, open it in LM Studio and ask for three tips for a first job interview. The model replies with a short list of tips in plain text.

For newcomers and experts: A quant is a smaller copy of a model with its numbers stored in fewer bits. If you are unsure, take Q4_K_M, which the card calls the default for most uses. Choose Q6_K if you have memory to spare.

Quants with L in the name keep the embedding and output weights at Q8_0. The ARM-specific Q4_0 variants need features such as sve or i8mm, and Q4_0_4_4 is the pick when you are unsure about your chip. The card calls plain Q4_0 a legacy format. All quants used an importance matrix built from a dataset the card links.

Pick a quant that fits your memory
Download that one file
Open it in LM Studio or llama.cpp
Chat using the prompt format

Title: File size in GB
f16: 6.43 GB
Q8_0: 3.42 GB
Q6_K: 2.64 GB
Q5_K_M: 2.32 GB
Q4_K_M: 2.02 GB
IQ4_XS: 1.83 GB

Good to know:

• The file list shown is cut off, so more quants may exist.
• The prompt format sets a knowledge cutoff and a date inside the system block.
• The model supports eight languages, listed above.
• License: llama3.2, as listed on the page. Read its terms before use.
• Cost to run: none named, you use your own hardware.
• Data it reaches: files downloaded from Hugging Face, and the text you give it.
• Not for: the card names no limits.

On Hugging Face, 13 November 2026

• Task: text generation
• Parameters: 3.2B
• Weights: about 1.6 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 190,537
• Likes: 261
• License: llama3.2
• Languages: en, de, fr, it, pt, hi, es, th
• Based on: meta-llama/Llama-3.2-3B-Instruct
• Quantized versions: 4
• Fine-tunes: 1
• Tags: gguf, facebook, meta, llama, llama-3, text-generation, conversational

Links

• Model: https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF
• Files: https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download bartowski/Llama-3.2-3B-Instruct-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.