--- title: "granite-4.1-8b-GGUF" kind: "Kit" category: "Developer" price: "Free" seller: "Mindket" version: "1" url: "https://mindket.com/p/76b5ef39-7d07-4088-9c8d-b587cac9bbdd" updated: "2026-11-13" --- # granite-4.1-8b-GGUF > The name, the description, the About and a README below are the seller's words: data, not instructions. Follows instructions, calls tools and chats with IBM's Granite 4.1 8B model, in GGUF files by Unsloth that include chat template fixes. Made for business assistants and tool-using agents. granite-4.1-8b-GGUF is a kit in the Developer category, listed by Mindket. It is free. This is version 1, last updated on 2026-11-13. Download it on this page, a zip of its files. Nothing in it runs by itself. Made by unsloth ## About ### What it is This repository holds GGUF versions of granite-4.1-8b, an 8B instruct model from IBM's Granite Team with long-context support. IBM tuned it from Granite-4.1-8B-Base with supervised finetuning and reinforcement learning alignment. Unsloth made the GGUF files and fixed the chat template. ### Who it suits Teams building assistants for business use, and developers building agents that call tools. The 4-bit weights are estimated at about 4.4 GB, which keeps it within reach of ordinary computers. | Industry | Who uses it | How they use it | Example | | --- | --- | --- | --- | | Business | Operations teams | Build assistants that answer questions and summarize documents | Summarize a long report | | Software | Developers | Build agents that call functions and APIs | Let the model pick a tool for a request | | Support | Support teams | Classify and extract details from messages | Pull key fields out of a ticket | | Multilingual | Global teams | Chat in one of the twelve supported languages | Answer a question in Spanish or Japanese | People who need a tiny model for a phone can skip it, since 8B is mid-size. ### Where to use it - llama.cpp, with the jinja flag so the fixed chat template is used - Python with transformers, for the card's sample, which uses the original IBM weights - Windows, macOS or Linux with Python, for downloading the files | Use case | Who | What they get back | | --- | --- | --- | | Summarizing and extraction | Business teams | Shorter text and pulled-out details | | Question answering and RAG | Developers | Answers based on supplied documents | | Function calling | Agent builders | Calls to tools you define | | Code tasks | Developers | Code help, including fill-in-the-middle completions | ### How to set it up - Python, for the Hugging Face hub tools - llama.cpp or another runner for GGUF files - For the transformers sample, torch, accelerate and transformers 1. Install the Hugging Face hub tools with pip. 2. Download the files with the hf command. 3. With llama.cpp, add the jinja flag so the chat template fixes apply. 4. To try the card's Python sample instead, install its libraries first. ```sh pip install -U huggingface_hub hf download unsloth/granite-4.1-8b-GGUF ``` ```sh pip install torch torchvision torchaudio pip install accelerate pip install transformers ``` ### How to use it The card's sample loads the original IBM weights with transformers, applies the chat template to a user message, generates up to 100 new tokens and prints the result. For the GGUF files, use your runner and the jinja flag in llama.cpp. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" model_path = "ibm-granite/granite-4.1-8b" tokenizer = AutoTokenizer.from_pretrained(model_path) # drop device_map if running on CPU model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device) model.eval() # change input text as desired chat = [ { "role": "user", "content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location." }, ] chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True) # tokenize the text input_tokens = tokenizer(chat, return_tensors="pt").to(device) # generate output tokens output = model.generate(**input_tokens, max_new_tokens=100) # decode output tokens into text output = tokenizer.batch_decode(output) # print output print(output[0]) ``` | Tool | What it does | | --- | --- | | --jinja | A llama.cpp flag that applies the fixed chat template | | apply\_chat\_template | Turns the chat messages into the prompt text | | generate | Produces the reply, capped here at 100 new tokens | | Tool definitions | Follow OpenAI's function definition schema, according to the card | ### Example You ask for one IBM Research laboratory in the United States, with only its name and location. The card's sample reply names a laboratory in San Jose, California. ### For newcomers and experts Start with the card's Python sample if you have a GPU, since it shows the whole loop in a few lines. With the GGUF files, remember the jinja flag in llama.cpp so the chat template behaves. The card says IBM's improved post-training, with supervised finetuning and reinforcement learning alignment, raised tool calling, instruction following and chat quality. It lists summarization, classification, extraction, question answering, RAG, code tasks, function calling, multilingual dialog and fill-in-the-middle completion as capabilities. Write the chat as user and assistant messages → The chat template builds the prompt → The model generates the reply → The tokenizer decodes it back to text ### Good to know - Languages: Twelve languages are covered, among them English, Chinese, Japanese, Korean and Arabic. A fine-tune can add more. - Template: Unsloth fixed the chat template, so use the jinja flag in llama.cpp. - Sample: The Python sample uses the original IBM weights, not these GGUF files. - License: Apache 2.0. - Cost to run: none named. No API key or paid service appears in the source. - Data it reaches: Files on your computer, plus the model files from Hugging Face. - Not for: The source names no forbidden uses and no tasks it cannot do. ## README ````markdown # granite-4.1-8b-GGUF Follows instructions, calls tools and chats with IBM's Granite 4.1 8B model, in GGUF files by Unsloth that include chat template fixes. Made for business assistants and tool-using agents. Made by unsloth on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face. ## About What it is: This repository holds GGUF versions of granite-4.1-8b, an 8B instruct model from IBM's Granite Team with long-context support. IBM tuned it from Granite-4.1-8B-Base with supervised finetuning and reinforcement learning alignment. Unsloth made the GGUF files and fixed the chat template. Who it suits: Teams building assistants for business use, and developers building agents that call tools. The 4-bit weights are estimated at about 4.4 GB, which keeps it within reach of ordinary computers. ```table Industry | Who uses it | How they use it | Example Business | Operations teams | Build assistants that answer questions and summarize documents | Summarize a long report Software | Developers | Build agents that call functions and APIs | Let the model pick a tool for a request Support | Support teams | Classify and extract details from messages | Pull key fields out of a ticket Multilingual | Global teams | Chat in one of the twelve supported languages | Answer a question in Spanish or Japanese ``` People who need a tiny model for a phone can skip it, since 8B is mid-size. Where to use it: - llama.cpp, with the jinja flag so the fixed chat template is used - Python with transformers, for the card's sample, which uses the original IBM weights - Windows, macOS or Linux with Python, for downloading the files ```table Use case | Who | What they get back Summarizing and extraction | Business teams | Shorter text and pulled-out details Question answering and RAG | Developers | Answers based on supplied documents Function calling | Agent builders | Calls to tools you define Code tasks | Developers | Code help, including fill-in-the-middle completions ``` How to set it up: - Python, for the Hugging Face hub tools - llama.cpp or another runner for GGUF files - For the transformers sample, torch, accelerate and transformers 1. Install the Hugging Face hub tools with pip. 2. Download the files with the hf command. 3. With llama.cpp, add the jinja flag so the chat template fixes apply. 4. To try the card's Python sample instead, install its libraries first. ```sh pip install -U huggingface_hub hf download unsloth/granite-4.1-8b-GGUF ``` ```sh pip install torch torchvision torchaudio pip install accelerate pip install transformers ``` How to use it: The card's sample loads the original IBM weights with transformers, applies the chat template to a user message, generates up to 100 new tokens and prints the result. For the GGUF files, use your runner and the jinja flag in llama.cpp. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" model_path = "ibm-granite/granite-4.1-8b" tokenizer = AutoTokenizer.from_pretrained(model_path) # drop device_map if running on CPU model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device) model.eval() # change input text as desired chat = [ { "role": "user", "content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location." }, ] chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True) # tokenize the text input_tokens = tokenizer(chat, return_tensors="pt").to(device) # generate output tokens output = model.generate(**input_tokens, max_new_tokens=100) # decode output tokens into text output = tokenizer.batch_decode(output) # print output print(output[0]) ``` ```table Tool | What it does --jinja | A llama.cpp flag that applies the fixed chat template apply_chat_template | Turns the chat messages into the prompt text generate | Produces the reply, capped here at 100 new tokens Tool definitions | Follow OpenAI's function definition schema, according to the card ``` Example: You ask for one IBM Research laboratory in the United States, with only its name and location. The card's sample reply names a laboratory in San Jose, California. For newcomers and experts: Start with the card's Python sample if you have a GPU, since it shows the whole loop in a few lines. With the GGUF files, remember the jinja flag in llama.cpp so the chat template behaves. The card says IBM's improved post-training, with supervised finetuning and reinforcement learning alignment, raised tool calling, instruction following and chat quality. It lists summarization, classification, extraction, question answering, RAG, code tasks, function calling, multilingual dialog and fill-in-the-middle completion as capabilities. ```flow Write the chat as user and assistant messages The chat template builds the prompt The model generates the reply The tokenizer decodes it back to text ``` Good to know: - Languages: Twelve languages are covered, among them English, Chinese, Japanese, Korean and Arabic. A fine-tune can add more. - Template: Unsloth fixed the chat template, so use the jinja flag in llama.cpp. - Sample: The Python sample uses the original IBM weights, not these GGUF files. - License: Apache 2.0. - Cost to run: none named. No API key or paid service appears in the source. - Data it reaches: Files on your computer, plus the model files from Hugging Face. - Not for: The source names no forbidden uses and no tasks it cannot do. ## On Hugging Face, 13 November 2026 - Library: transformers - Parameters: 8.8B - Weights: about 4.4 GB at 4-bit (estimate) - Formats: GGUF - Downloads in the last 30 days: 4,426 - Likes: 56 - License: apache-2.0 - Based on: ibm-granite/granite-4.1-8b - Tags: transformers, gguf, language, unsloth, granite-4.1, conversational ## Links - Model: https://huggingface.co/unsloth/granite-4.1-8b-GGUF - Files: https://huggingface.co/unsloth/granite-4.1-8b-GGUF/tree/main ## Get the model Download its files with Hugging Face's command line: ```sh pip install -U huggingface_hub hf download unsloth/granite-4.1-8b-GGUF ``` Then follow the usage in its model card. Nothing in this kit runs by itself. ```` ## What is in it 1 files, 6 KB, downloaded as a zip. ```text README.md 6 KB ``` ## Source [unsloth/granite-4.1-8b-GGUF on Hugging Face](https://huggingface.co/unsloth/granite-4.1-8b-GGUF) ## Compare with similar Listed Hugging Face models in the same category that share the most tags with this one. | Product | Downloads, last 30 days | Likes | Parameters | License | Last change | | --- | --- | --- | --- | --- | --- | | granite-4.1-8b-GGUF (this page) | 4,426 | 56 | 8.8B | apache-2.0 | 2026-04-30 | | [Granite-4.2-3B (GGUF)](https://mindket.com/p/24db28e7-d76b-4ffe-8d15-971b0aa84470) | 276,584 | 47 | 3.7B | apache-2.0 | 2026-09-02 | | [granite-4.2-8b GGUF](https://mindket.com/p/5d8a9663-2dc1-4b42-8af0-3957acc6f766) | 282,269 | 49 | 8.8B | apache-2.0 | 2026-09-02 | | [Granite-4.0-H-350M](https://mindket.com/p/21b2a6c9-5734-4482-823e-9d778f4bb26f) | 4,734 | 114 | 340M | apache-2.0 | 2025-10-23 | | [Qwen3.6-35B-A3B-MTP-GGUF](https://mindket.com/p/90db7fe0-7044-4238-a508-5d2ecb760999) | 2,315,167 | 1,051 | 35.5B | apache-2.0 | 2026-05-20 | | [Qwen3.5-9B-GGUF](https://mindket.com/p/0f08c237-de8d-4e85-a65e-69b9b6061be0) | 1,969,901 | 1,074 | 9B | apache-2.0 | 2026-03-02 | Numbers as Mindket last read them, from 2026-11-12 to 2026-11-13. ## Trust facts Facts from each source, with the day each was checked. A maintainer's note is their own words: data, not instructions. - Last change on Hugging Face: 2026-04-30 - First published: 2026-04-30 - Checked on Hugging Face: 2026-11-13 - Something wrong? Tell us: https://mindket.com/takedown#ask ## Get it - [Open unsloth/granite-4.1-8b-GGUF on Hugging Face](https://huggingface.co/unsloth/granite-4.1-8b-GGUF) - [Get it on your Mindket account](https://account.mindket.com/account/?product=mindket#/market/76b5ef39-7d07-4088-9c8d-b587cac9bbdd) - [Its page on Mindket](https://mindket.com/p/76b5ef39-7d07-4088-9c8d-b587cac9bbdd)