Meta-Llama-3.1-8B-Instruct-AWQ-INT4

Meta-Llama-3.1-8B-Instruct-AWQ-INT4

Runs Meta's Llama 3.1 8B Instruct chat model in 4-bit AWQ form, so it fits on a GPU with about 4 GiB of free memory for the weights. For developers who want a compact multilingual chat model.

Made by hugging-quants on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a community 4-bit AWQ copy of Meta Llama 3.1 8B Instruct, the instruction-tuned chat model. The quantization used AutoAWQ with GEMM kernels, zero-point quantization and a group size of 128. Meta's original is the BF16 version, and this copy is not an official Meta release.

Who it suits: It suits developers who want to run an 8B chat model on a modest GPU, in several languages, through transformers or a serving stack.

Industry | Who uses it | How they use it | Example
Software | App developers | Add a local chat model to a product | A support assistant that runs on one GPU
Research | NLP researchers | Test an instruction model at lower memory cost | Comparing 4-bit output with the original
Operations | Platform engineers | Serve a quantized model for internal tools | Running it behind text-generation-inference
Education | Students and educators | Experiment with a language model on a single graphics card | A classroom demo of chat generation

Skip it if you have no NVIDIA GPU, since the card's example moves the model to CUDA.

Where to use it:

• Python with the transformers library, which the card recommends
• AutoAWQ, as an alternative loader
• Text Generation Inference with Marlin kernels, run through Docker
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Local chat | Developers | Replies from an 8B instruction model on a smaller GPU
Multilingual dialogue | Product teams | Chat in the listed languages, including German, French and Hindi
Serving | Platform teams | A quantized model behind an optimized inference server

How to set it up: You need these first.

• Python with pip
• A GPU with a bit over 4 GiB of free VRAM, since the card says that much is needed only for loading the checkpoint
• The transformers, autoawq and accelerate packages

1. Install the three Python packages with pip.
2. Load the model by its Hugging Face name, with an AWQ config set to 4 bits.
3. Build a chat prompt and call generate.

pip install -q --upgrade transformers autoawq accelerate

pip install -U huggingface_hub
hf download hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4

How to use it: Load the model with the AWQ config, apply the chat template to your messages and generate. The card's script sets a system message and a user question, then prints the reply.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4"

quantization_config = AwqConfig(
    bits=4,
    fuse_max_seq_len=512, # Note: Update this as per your use-case
    do_fuse=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.float16,
  low_cpu_mem_usage=True,
  device_map="auto",
  quantization_config=quantization_config
)

prompt = [
  {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
  {"role": "user", "content": "What's Deep Learning?"},
]
inputs = tokenizer.apply_chat_template(
  prompt,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors="pt",
  return_dict=True,
).to("cuda")

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
print(tokenizer.batch_decode(outputs[:, inputs['input_ids'].shape[1]:], skip_special_tokens=True)[0])

Tool | What it does
transformers | The card's recommended way to load and run the model
autoawq | An alternative loader built on top of transformers
text-generation-inference | Serves the model with Marlin kernels and needs Docker
fuse_max_seq_len | A fusion length setting that you change to fit your use

Example: Give the model a system message telling it to answer as a pirate, then ask it what deep learning is. It writes a reply of up to 256 new tokens, in that voice. The wording differs on each run because sampling is switched on.

For newcomers and experts: If you are new, quantization shrinks a model so it fits on smaller hardware. Run the card's Python script once to see a reply, then change the user question to something of your own.

If you know the field, this is INT4 AWQ with GEMM kernels and zero-point quantization at group size 128. The card notes the 4 GiB figure excludes the KV cache and CUDA graphs, so leave headroom. It lists TGI with Marlin kernels for faster inference.

Meta's Llama 3.1 8B Instruct weights
AutoAWQ quantizes them to INT4
You load them with an AwqConfig
Chat template builds the prompt
Generate returns the reply

Good to know:

• Languages: English, German, French, Italian, Portuguese, Hindi, Spanish and Thai are listed
• Origin: a community quantization, not an official Meta release
• Memory: about 4 GiB of VRAM for the checkpoint alone, plus room for the KV cache
• Other builds: other people have made 16 fine-tuned versions of it

• License: Llama 3.1 license, as listed on the record, inherited from Meta's model
• Cost to run: none named, the source lists no API key or paid service, though the TGI route needs Docker and a Hugging Face login
• Data it reaches: the prompts you give your own runtime, and the source names no web or account access beyond the optional login
• Not for: machines without a CUDA GPU, since the card's example runs on CUDA

On Hugging Face, 14 November 2026

• Task: text generation
• Library: transformers
• Parameters: 8B
• Weights: about 32.1 GB in I32
• Formats: Safetensors
• Downloads in the last 30 days: 319,739
• Likes: 96
• License: llama3.1
• Languages: en, de, fr, it, pt, hi, es, th
• Fine-tunes: 16
• Tags: transformers, safetensors, llama, text-generation, llama-3.1, meta, autoawq, conversational, 4-bit

Links

• Model: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4
• Files: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4

Then follow the usage in its model card. Nothing in this kit runs by itself.