Meta-Llama-3.1-8B-Instruct-GGUF

Meta-Llama-3.1-8B-Instruct-GGUF

Quantized GGUF files of Meta's Llama 3.1 8B Instruct for local chat, such as 4.92 GB for Q4_K_M and 8.54 GB for Q8_0. Made by bartowski with an importance matrix. Covers 8 languages and runs in LM Studio.

Made by bartowski on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This repository holds GGUF quantizations of Meta's Llama 3.1 8B Instruct chat model, made by bartowski. He built them with llama.cpp release b3472 and an importance matrix, which helps keep quality as files shrink. The model has 8B parameters and Hugging Face lists English, German, French, Italian, Portuguese, Hindi, Spanish and Thai. The card points to LM Studio for running the files.

Who it suits: It suits people who want to chat with Llama 3.1 8B on their own computer and need a file size that fits their memory.

Industry | Who uses it | How they use it | Example
Local AI | People running chat models offline | Load a quantized file in LM Studio | Chat with the 4.92 GB Q4_K_M file
Software | Developers testing local models | Try the same prompts on several file sizes | Compare Q6_K and Q5_K_M answers
Research | Hobbyists studying quantization | Read the card's list of quant types and sizes | Check how the file size falls with each level

You can skip it if you want the original Meta weights for training, since these are compressed files made for running the model.

Where to use it:

• LM Studio, which the card names
• Other GGUF runners such as llama.cpp
• Windows, macOS and Linux with Python, to download the files

Use case | Who | What they get back
Local chat | Hobbyists | Replies from an 8B model on their own machine
Multilingual prompts | Users in eight languages | Answers in the languages Hugging Face lists
Size choice | People with limited memory | A file matched to the memory they have

How to set it up: You need these first:

• Python, to use Hugging Face's download client
• Disk space and memory that fit the file you pick
• LM Studio or another GGUF runner

1. Install or update the Hugging Face download client.
2. Choose one file from the card's table, since the card says not to fetch the whole branch.
3. Download that file to your computer.
4. Load it in your runner and use the prompt format below.

pip install -U huggingface_hub
hf download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

How to use it: You load one file in LM Studio and chat with the model. If your runner does not apply the chat template for you, the card gives this prompt format.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Tool | What it does
Q4_K_M | A 4.92 GB file that the card names as the default size for most uses
Q6_K | A 6.60 GB file with higher quality
Q8_0 | The largest quantized file, at 8.54 GB
Q3_K_XL | A 4.78 GB file with lower quality, for low memory
Q4_0_8_8 | A 4.66 GB file for ARM and AVX CPUs, which the card says not to use on a Mac

Example: You pick the 4.92 GB Q4_K_M file and load it in LM Studio. You then send a prompt in English or Spanish and read the reply. The card suggests this size for most people.

For newcomers and experts: Quantizing shrinks a model so it fits in less memory, and the suffix tells how much was shrunk. A good first try is the Q4_K_M file, then a larger or smaller one if it does not fit.

For readers who know the field, the files marked L keep the embedding and output weights at Q8_0. The Q4_0_8_8 and Q4_0_4_8 files are tuned for ARM and AVX CPUs. They need sve or i8mm support on ARM, and the card says to avoid them on a Mac.

Pick a file that fits your memory
Download that single file
Open it in LM Studio
Apply the prompt format
Chat with the model

Good to know:

• Single file: the card says to download one file, not the whole branch, and the repository holds many large files.
• Update: on 30 July 2024 the chat template was fixed for a small bug with tool use.
• Mac: the Q4_0_8_8 and Q4_0_4_8 files are not meant for a Mac.
• Cut: the file table in the text available here stops after 14 rows, so the smaller files are missing.
• Counts: Hugging Face lists 8B parameters and about 4 GB at 4-bit as an estimate.
• Versions: Hugging Face lists 2 quantized versions made by others.

Title: File size by quantization, from the card's table
f32: 32.13 GB
Q8_0: 8.54 GB
Q6_K: 6.60 GB
Q5_K_M: 5.73 GB
Q4_K_M: 4.92 GB
Q3_K_XL: 4.78 GB
IQ4_NL: 4.68 GB

• License: llama3.1, Meta's own license for Llama 3.1, so check its terms before use.
• Cost to run: none named. You need memory that fits the file.
• Data it reaches: the prompts you give it, on your own machine. The text available here names no network use.
• Not for: the Q4_0_8_8 and Q4_0_4_8 files on a Mac.

On Hugging Face, 13 November 2026

• Task: text generation
• Parameters: 8B
• Weights: about 4 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 322,920
• Likes: 433
• License: llama3.1
• Languages: en, de, fr, it, pt, hi, es, th
• Based on: meta-llama/Meta-Llama-3.1-8B-Instruct
• Quantized versions: 2
• Tags: gguf, facebook, meta, pytorch, llama, llama-3, text-generation, conversational

Links

• Model: https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
• Files: https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

Then follow the usage in its model card. Nothing in this kit runs by itself.