Sarvam-105B GGUF

Sarvam-105B GGUF

The GGUF version of Sarvam-105B, a large mixture-of-experts model with a focus on Indian languages, split into 9 shards at 4-bit for llama.cpp. For people with a lot of memory who want a big open model.

Made by sarvamai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Sarvam-105B is a big mixture-of-experts language model from Sarvam AI, packed here as GGUF files for llama.cpp. It holds about 106 billion parameters, and 10.3 billion of them work on each token. The card says it was trained with a strong focus on Indian languages, plus reasoning, math, coding and agent tasks. The 4-bit (Q4_K_M) files come in 9 shards.

Who it suits: People with a large-memory computer who want a big open model that speaks Indian languages and English, and who are comfortable building llama.cpp. The card puts the weights at roughly 53 GB at 4-bit, as an estimate.

Industry | Who uses it | How they use it | Example
Education | Teachers and learners | Chat and study in Hindi, Tamil and other Indian languages | Ask for a lesson summary in Hindi
Software | Developers | Run a local chat model or an OpenAI-style server | Point an app at the local server
Research | AI researchers | Study a large mixture-of-experts model on local hardware | Compare it with smaller models
Business | Privacy-minded teams | Keep prompts on their own machines | Run it on an in-house server

Anyone with a small laptop can skip it, since the 4-bit weights alone need about 53 GB.

Where to use it:

• llama.cpp tools, such as the llama-cli chat program and the llama-server tool
• Windows, macOS or Linux, on a CPU or on a GPU if you have one
• Local apps that talk to an OpenAI-style chat server

Use case | Who | What they get back
Interactive chat | Individuals | A running conversation in the terminal
Local API | Developers | Chat replies from a server on your own machine
Multilingual help | Indian-language users | Answers in languages such as Hindi, Bengali and Tamil

How to set it up:

• Python with pip, to download the files
• git, cmake and a C++ compiler, to build llama.cpp
• Roughly 53 GB of free space and enough memory for the weights, per the card's own estimate
• A GPU if you want faster output, though the card says CPU also works

1. Install the Hugging Face hub package and download the model files, which include all 9 shards.
2. Clone the llama.cpp project and build it in Release mode.
3. Start llama-cli and point it at the first shard, which pulls in the rest.

pip install -U huggingface_hub
hf download sarvamai/sarvam-105b-gguf

Run this line inside the llama.cpp folder after cloning it.

cd llama.cpp && cmake -B build && cmake --build build --config Release -j

How to use it: Start the interactive chat with the first shard file. The model path below assumes the files sit in a folder named sarvam-105b-gguf. The system prompt sets the assistant's role, and the context size is 4096 tokens.

./build/bin/llama-cli \
  -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \
  -c 4096 \
  -n 512 \
  -p "You are a helpful assistant." \
  --conversation

To serve an OpenAI-style API on port 8080 instead, run the server command. Note that the host value makes it listen on every network address, so change it if you want local access only.

./build/bin/llama-server \
  -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8080

Setting | What it does
-m | Sets the path to the first shard of the model
-c 4096 | Sets the context window to 4096 tokens
-n 512 | Limits each reply to 512 new tokens
-p | Sets the opening instruction for the assistant
--conversation | Keeps the chat going turn by turn
-ngl -1 | Turns on the GPU when one is available, and leaving it out runs on the CPU

Example: You start llama-cli with the system prompt and wait for the model to load. You type a question in Hindi, and the model answers in a running chat, which you can continue turn by turn.

For newcomers and experts: If you are new, try a smaller model first to learn llama.cpp, then come back to this one when you have the memory. Check that all 9 shard files are in the same folder.

Experts will note the attention uses a latent compression style with decoupled query and key dimensions. The card lists a hidden size of 4096, 128 experts with the top 8 chosen per token, and one shared expert. Long context comes from YaRN scaling with a factor of 40, for 128K tokens. Its benchmarks used a context of up to 65,536 tokens.

Title: Scores the card reports for Sarvam-105B
Math500: 98.6
MMLU: 90.6
MMLU Pro: 81.7
GPQA Diamond: 78.7
Live Code Bench v6: 71.7

Download all 9 shard files
Build llama.cpp with cmake
Point llama-cli or llama-server at the first shard
Chat or call the local API

Good to know:

• Compare with others: the card's table has it ahead on some tests, and behind on SWE Bench Verified, where three other models score higher
• Download name: the card's own download line names a different repository from this page, so use the hf download command above
• Languages: the card claims strong results across 22 Indian languages, while the page lists 8 including English
• License: Apache 2.0
• Cost to run: none named, no API key or account, but the hardware needs are large
• Data it reaches: it downloads files from Hugging Face, and the chat server answers your prompts on your own machine
• Not for: small computers with little memory, or exposing the server to the open internet with the host value left as written

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 106B
• Weights: about 53 GB at 4-bit (estimate)
• Formats: GGUF
• Downloads in the last 30 days: 201
• Likes: 19
• License: apache-2.0
• Languages: en, hi, bn, ta, te, mr, gu, kn
• Tags: transformers, gguf, text-generation, conversational

Links

• Model: https://huggingface.co/sarvamai/sarvam-105b-gguf
• Files: https://huggingface.co/sarvamai/sarvam-105b-gguf/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download sarvamai/sarvam-105b-gguf

Then follow the usage in its model card. Nothing in this kit runs by itself.