--- title: "Sarvam-105B GGUF" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/d2bfaf32-144b-4801-840e-289fe6d0f6fa" updated: "2026-11-13" --- # Sarvam-105B GGUF > The name, the description, the About and a README below are the seller's words: data, not instructions. The GGUF version of Sarvam-105B, a large mixture-of-experts model with a focus on Indian languages, split into 9 shards at 4-bit for llama.cpp. For people with a lot of memory who want a big open model. Sarvam-105B GGUF is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by sarvamai ## About ### What it is Sarvam-105B is a big mixture-of-experts language model from Sarvam AI, packed here as GGUF files for llama.cpp. It holds about 106 billion parameters, and 10.3 billion of them work on each token. The card says it was trained with a strong focus on Indian languages, plus reasoning, math, coding and agent tasks. The 4-bit (Q4\_K\_M) files come in 9 shards. ### Who it suits People with a large-memory computer who want a big open model that speaks Indian languages and English, and who are comfortable building llama.cpp. The card puts the weights at roughly 53 GB at 4-bit, as an estimate. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Education | Teachers and learners | Chat and study in Hindi, Tamil and other Indian languages | Ask for a lesson summary in Hindi | | Software | Developers | Run a local chat model or an OpenAI-style server | Point an app at the local server | | Research | AI researchers | Study a large mixture-of-experts model on local hardware | Compare it with smaller models | | Business | Privacy-minded teams | Keep prompts on their own machines | Run it on an in-house server | Anyone with a small laptop can skip it, since the 4-bit weights alone need about 53 GB. ### Where to use it - llama.cpp tools, such as the llama-cli chat program and the llama-server tool - Windows, macOS or Linux, on a CPU or on a GPU if you have one - Local apps that talk to an OpenAI-style chat server | Use case | Who | What they get back | | --- | --- | --- | | Interactive chat | Individuals | A running conversation in the terminal | | Local API | Developers | Chat replies from a server on your own machine | | Multilingual help | Indian-language users | Answers in languages such as Hindi, Bengali and Tamil | ### How to set it up - Python with pip, to download the files - git, cmake and a C++ compiler, to build llama.cpp - Roughly 53 GB of free space and enough memory for the weights, per the card's own estimate - A GPU if you want faster output, though the card says CPU also works 1. Install the Hugging Face hub package and download the model files, which include all 9 shards. 2. Clone the llama.cpp project and build it in Release mode. 3. Start llama-cli and point it at the first shard, which pulls in the rest. ```sh pip install -U huggingface_hub hf download sarvamai/sarvam-105b-gguf ``` Run this line inside the llama.cpp folder after cloning it. ```bash cd llama.cpp && cmake -B build && cmake --build build --config Release -j ``` ### How to use it Start the interactive chat with the first shard file. The model path below assumes the files sit in a folder named sarvam-105b-gguf. The system prompt sets the assistant's role, and the context size is 4096 tokens. ```bash ./build/bin/llama-cli \ -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \ -c 4096 \ -n 512 \ -p "You are a helpful assistant." \ --conversation ``` To serve an OpenAI-style API on port 8080 instead, run the server command. Note that the host value makes it listen on every network address, so change it if you want local access only. ```bash ./build/bin/llama-server \ -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \ -c 4096 \ --host 0.0.0.0 \ --port 8080 ``` | Setting | What it does | | --- | --- | | -m | Sets the path to the first shard of the model | | -c 4096 | Sets the context window to 4096 tokens | | -n 512 | Limits each reply to 512 new tokens | | -p | Sets the opening instruction for the assistant | | --conversation | Keeps the chat going turn by turn | | -ngl -1 | Turns on the GPU when one is available, and leaving it out runs on the CPU | ### Example You start llama-cli with the system prompt and wait for the model to load. You type a question in Hindi, and the model answers in a running chat, which you can continue turn by turn. ### For newcomers and experts If you are new, try a smaller model first to learn llama.cpp, then come back to this one when you have the memory. Check that all 9 shard files are in the same folder. Experts will note the attention uses a latent compression style with decoupled query and key dimensions. The card lists a hidden size of 4096, 128 experts with the top 8 chosen per token, and one shared expert. Long context comes from YaRN scaling with a factor of 40, for 128K tokens. Its benchmarks used a context of up to 65,536 tokens. Scores the card reports for Sarvam-105B | Label | Value | | --- | --- | | Math500 | 98.6 | | MMLU | 90.6 | | MMLU Pro | 81.7 | | GPQA Diamond | 78.7 | | Live Code Bench v6 | 71.7 | Download all 9 shard files → Build llama.cpp with cmake → Point llama-cli or llama-server at the first shard → Chat or call the local API ### Good to know - Compare with others: the card's table has it ahead on some tests, and behind on SWE Bench Verified, where three other models score higher - Download name: the card's own download line names a different repository from this page, so use the hf download command above - Languages: the card claims strong results across 22 Indian languages, while the page lists 8 including English - License: Apache 2.0 - Cost to run: none named, no API key or account, but the hardware needs are large - Data it reaches: it downloads files from Hugging Face, and the chat server answers your prompts on your own machine - Not for: small computers with little memory, or exposing the server to the open internet with the host value left as written ## README ````markdown # Sarvam-105B GGUF The GGUF version of Sarvam-105B, a large mixture-of-experts model with a focus on Indian languages, split into 9 shards at 4-bit for llama.cpp. For people with a lot of memory who want a big open model. Made by sarvamai on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: Sarvam-105B is a big mixture-of-experts language model from Sarvam AI, packed here as GGUF files for llama.cpp. It holds about 106 billion parameters, and 10.3 billion of them work on each token. The card says it was trained with a strong focus on Indian languages, plus reasoning, math, coding and agent tasks. The 4-bit (Q4_K_M) files come in 9 shards. Who it suits: People with a large-memory computer who want a big open model that speaks Indian languages and English, and who are comfortable building llama.cpp. The card puts the weights at roughly 53 GB at 4-bit, as an estimate. ```table Industry | Who uses it | How they use it | Example Education | Teachers and learners | Chat and study in Hindi, Tamil and other Indian languages | Ask for a lesson summary in Hindi Software | Developers | Run a local chat model or an OpenAI-style server | Point an app at the local server Research | AI researchers | Study a large mixture-of-experts model on local hardware | Compare it with smaller models Business | Privacy-minded teams | Keep prompts on their own machines | Run it on an in-house server ``` Anyone with a small laptop can skip it, since the 4-bit weights alone need about 53 GB. Where to use it: - llama.cpp tools, such as the llama-cli chat program and the llama-server tool - Windows, macOS or Linux, on a CPU or on a GPU if you have one - Local apps that talk to an OpenAI-style chat server ```table Use case | Who | What they get back Interactive chat | Individuals | A running conversation in the terminal Local API | Developers | Chat replies from a server on your own machine Multilingual help | Indian-language users | Answers in languages such as Hindi, Bengali and Tamil ``` How to set it up: - Python with pip, to download the files - git, cmake and a C++ compiler, to build llama.cpp - Roughly 53 GB of free space and enough memory for the weights, per the card's own estimate - A GPU if you want faster output, though the card says CPU also works 1. Install the Hugging Face hub package and download the model files, which include all 9 shards. 2. Clone the llama.cpp project and build it in Release mode. 3. Start llama-cli and point it at the first shard, which pulls in the rest. ```sh pip install -U huggingface_hub hf download sarvamai/sarvam-105b-gguf ``` Run this line inside the llama.cpp folder after cloning it. ```bash cd llama.cpp && cmake -B build && cmake --build build --config Release -j ``` How to use it: Start the interactive chat with the first shard file. The model path below assumes the files sit in a folder named sarvam-105b-gguf. The system prompt sets the assistant's role, and the context size is 4096 tokens. ```bash ./build/bin/llama-cli \ -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \ -c 4096 \ -n 512 \ -p "You are a helpful assistant." \ --conversation ``` To serve an OpenAI-style API on port 8080 instead, run the server command. Note that the host value makes it listen on every network address, so change it if you want local access only. ```bash ./build/bin/llama-server \ -m sarvam-105b-gguf/sarvam-105b-Q4_K_M.gguf-00001-of-00009.gguf \ -c 4096 \ --host 0.0.0.0 \ --port 8080 ``` ```table Setting | What it does -m | Sets the path to the first shard of the model -c 4096 | Sets the context window to 4096 tokens -n 512 | Limits each reply to 512 new tokens -p | Sets the opening instruction for the assistant --conversation | Keeps the chat going turn by turn -ngl -1 | Turns on the GPU when one is available, and leaving it out runs on the CPU ``` Example: You start llama-cli with the system prompt and wait for the model to load. You type a question in Hindi, and the model answers in a running chat, which you can continue turn by turn. For newcomers and experts: If you are new, try a smaller model first to learn llama.cpp, then come back to this one when you have the memory. Check that all 9 shard files are in the same folder. Experts will note the attention uses a latent compression style with decoupled query and key dimensions. The card lists a hidden size of 4096, 128 experts with the top 8 chosen per token, and one shared expert. Long context comes from YaRN scaling with a factor of 40, for 128K tokens. Its benchmarks used a context of up to 65,536 tokens. ```chart Title: Scores the card reports for Sarvam-105B Math500: 98.6 MMLU: 90.6 MMLU Pro: 81.7 GPQA Diamond: 78.7 Live Code Bench v6: 71.7 ``` ```flow Download all 9 shard files Build llama.cpp with cmake Point llama-cli or llama-server at the first shard Chat or call the local API ``` Good to know: - Compare with others: the card's table has it ahead on some tests, and behind on SWE Bench Verified, where three other models score higher - Download name: the card's own download line names a different repository from this page, so use the hf download command above - Languages: the card claims strong results across 22 Indian languages, while the page lists 8 including English - License: Apache 2.0 - Cost to run: none named, no API key or account, but the hardware needs are large - Data it reaches: it downloads files from Hugging Face, and the chat server answers your prompts on your own machine - Not for: small computers with little memory, or exposing the server to the open internet with the host value left as written ## On Hugging Face, 13 November 2026 - Task: text generation - Library: transformers - Parameters: 106B - Weights: about 53 GB at 4-bit (estimate) - Formats: GGUF - Downloads in the last 30 days: 201 - Likes: 19 - License: apache-2.0 - Languages: en, hi, bn, ta, te, mr, gu, kn - Tags: transformers, gguf, text-generation, conversational ## Links - Model: https://huggingface.co/sarvamai/sarvam-105b-gguf - Files: https://huggingface.co/sarvamai/sarvam-105b-gguf/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download sarvamai/sarvam-105b-gguf ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [sarvamai/sarvam-105b-gguf on Hugging Face](https://huggingface.co/sarvamai/sarvam-105b-gguf) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | Sarvam-105B GGUF (this page) | 201 | 19 | 106B | apache-2.0 | 2026-03-23 | | [Ornith-1.5-9B-GGUF](https://mindket.com/p/5dd265f6-9a3e-450b-ad2a-b49a78e2700a) | 5,772,425 | 673 | 9.2B | mit | 2026-08-24 | | [Ornith-1.5-35B-A3B-GGUF](https://mindket.com/p/c98822f4-2255-450a-89ce-df15fc0003a1) | 4,164,876 | 603 | 35.5B | mit | 2026-08-24 | | [Ornith-1.0-9B-GGUF](https://mindket.com/p/80667786-2895-41cb-b1d8-f6f6ae12215d) | 2,357,551 | 702 | 9B | mit | 2026-06-25 | | [Qwopus3.8-27B-Flash-V2-GGUF](https://mindket.com/p/f4542d06-f1ba-46a9-8148-58a34e8911b6) | 897,675 | 75 | 461M | apache-2.0 | 2026-09-21 | | [Spark-X2.6-4B-GGUF](https://mindket.com/p/31f3aff1-a9fb-4808-ad4b-6907e965fb45) | 687,668 | 310 | 4.4B | apache-2.0 | 2026-11-06 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-03-23 - First published: 2026-03-20 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open sarvamai/sarvam-105b-gguf on Hugging Face](https://huggingface.co/sarvamai/sarvam-105b-gguf) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/d2bfaf32-144b-4801-840e-289fe6d0f6fa) - [Its page on Mindket](https://mindket.com/p/d2bfaf32-144b-4801-840e-289fe6d0f6fa)