Granite-4.0-H-350M

Granite-4.0-H-350M

Handles summaries, classification, extraction, question answering, RAG, code tasks and tool calling in 12 languages with a 340M-parameter instruct model from IBM. For developers who need a compact on-device model.

Made by ibm-granite on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Granite-4.0-H-350M is a small instruct model from IBM's Granite Team. It was fine-tuned from the Granite-4.0-H-350M-Base model on permissively licensed open instruction data and IBM's own synthetic data. The methods included supervised fine-tuning, reinforcement learning and model merging. It targets on-device use and research.

Who it suits: It suits developers who want a compact multilingual model for everyday language tasks, and teams that plan to fine-tune a small model on their own domain without large compute.

Industry | Who uses it | How they use it | Example
Software | App developers | Run summaries and extraction on the user's device | Pulling fields out of a pasted email
Support | Help desk tool builders | Classify tickets and answer from retrieved text | Routing a ticket to the right queue
Engineering | Developer tool teams | Use code tasks and fill-in-the-middle completion | Completing a missing function body
Research | ML researchers | Fine-tune a small model for a specialized field | Adapting it to legal or medical wording

Skip it if you need the depth of a large model, since 340M parameters limits what it can reason through.

Where to use it:

• Python with the transformers library, on a GPU or on CPU
• On-device and research setups, which the card names as the intended use
• Windows, macOS or Linux with Python

Use case | Who | What they get back
Summaries and extraction | App teams | Short summaries or fields pulled from text
RAG question answering | Search builders | Answers drawn from passages you supply
Tool calling | Agent developers | Function calls built from a tool list you define
Code completion | Editor plugin makers | Fill-in-the-middle suggestions

How to set it up: You need these first.

• Python with pip
• The torch, accelerate and transformers packages
• About 681 MB for the BF16 weights, or about 170 MB at 4-bit by estimate

1. Install the libraries with pip.
2. Load the tokenizer and model by name.
3. Build a chat, apply the chat template and generate.

pip install torch torchvision torchaudio
pip install accelerate
pip install transformers

pip install -U huggingface_hub
hf download ibm-granite/granite-4.0-h-350m

How to use it: Write a chat as a list of role and content messages, apply the chat template, tokenize and generate. Drop the device map if you run on CPU. For tool calling, you define your tools in the same schema that OpenAI uses for functions.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda"
model_path = "ibm-granite/granite-4.0-h-350M"
tokenizer = AutoTokenizer.from_pretrained(model_path)
# drop device_map if running on CPU
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device)
model.eval()
# change input text as desired
chat = [
    { "role": "user", "content": "Please list one IBM Research laboratory located in the United States. You should only output its name and location." },
]
chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
# tokenize the text
input_tokens = tokenizer(chat, return_tensors="pt").to(device)
# generate output tokens
output = model.generate(**input_tokens,
                        max_new_tokens=100)
# decode output tokens into text
output = tokenizer.batch_decode(output)
# print output
print(output[0])

Tool | What it does
apply_chat_template | Formats your messages with the model's role markers
model.generate | Produces the reply, capped here at 100 new tokens
tools list | Describes functions the model can call, in the OpenAI function schema

Example: Ask for one IBM Research laboratory in the United States, name and location only. The printed output repeats the prompt with the role markers and then the reply. The card's expected output names the Almaden Research Center in San Jose, California.

For newcomers and experts: If you are new, this is a small chat model you can run on a laptop. Start with the short Python script and change the question to your own. Keep tasks simple and short.

If you know the field, the card lists summarization, classification, extraction, question answering, RAG, code tasks, function calling, multilingual dialogue and fill-in-the-middle completions. It names 12 supported languages and says you can fine-tune it for others. Its tool calling uses the OpenAI function definition schema.

Granite-4.0-H-350M-Base
Supervised fine-tuning and RL
Model merging
Instruct model you load
Chat template builds your prompt

Good to know:

• Languages: 12 in all, namely Chinese, Japanese, Korean, Arabic, Czech, Dutch, Italian, Portuguese, French, Spanish, German and English
• Size: 340M parameters in Safetensors
• Developer: the Granite Team at IBM
• Community: other people have made 43 quantized and 16 fine-tuned versions

• License: Apache 2.0
• Cost to run: none named, the source lists no API key, account or paid service
• Data it reaches: the messages you give your own script, and the source names no web or account access
• Not for: the source forbids no uses, but it points the model at compact on-device and research work

On Hugging Face, 13 November 2026

• Task: text generation
• Library: transformers
• Parameters: 340M
• Weights: about 681 MB in BF16; about 170 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 4,734
• Likes: 114
• License: apache-2.0
• Based on: ibm-granite/granite-4.0-h-350m-base
• Quantized versions: 44
• Fine-tunes: 16
• Tags: transformers, safetensors, granitemoehybrid, text-generation, language, granite-4.0, conversational

Links

• Model: https://huggingface.co/ibm-granite/granite-4.0-h-350m
• Files: https://huggingface.co/ibm-granite/granite-4.0-h-350m/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download ibm-granite/granite-4.0-h-350m

Then follow the usage in its model card. Nothing in this kit runs by itself.