MiniCPM-V

What it is

MiniCPM-V and MiniCPM-o are open multimodal models from OpenBMB, developed by THUNLP and ModelBest. The README’s line is “See, hear, and interact on your local devices”. MiniCPM-V 4.7 reads images, several images at once, documents and video, in two sizes: 35B-A3B, the series’ first mixture-of-experts model, and a dense 1B. MiniCPM-o 4.5 is a 9B model that takes video and audio in while it writes text and speaks. This repository holds the README, the docs and folders such as eval_mm, finetune and web_demos. The weights are on Hugging Face and ModelScope, and the phone apps are in a second repository.

1. Pick a size: 1B, 35B-A3B or the 9B omni model
2. Get the weights from Hugging Face or ModelScope
3. Run them with a framework, or open a phone app
4. Send an image, a video or audio with your question
5. Read the text reply, or hear speech from MiniCPM-o

Who it suits

Developers and researchers who want a model that reads images, documents and video on their own hardware, and people who build phone or edge apps. It needs real hardware, and the size you pick decides how much. Skip it if you want a polished chat product: the phone apps are demos, and the live demos run on someone else’s servers.

Where to use it

MiniCPM-V is a family of models, not a coding-agent plug-in. The README documents no agent integration. The docs give these routes to run it.

How to set it up

Get it saves a zip of Mindket’s setup notes in your downloads folder, with a page for each route from OpenBMB’s README, docs and cookbook. The main route is Transformers, which the 4.7 page lists first.

1. Install the packages with the line below.
2. Load the 1B or the 35B-A3B model with the page’s code.
3. Build a messages list with an image and a question, run it through the processor’s chat template, then call generate. The page has the full example.

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

The page uses the 35B-A3B name. For the 1B size, swap in openbmb/MiniCPM-V-4.7-1B.

How to use it

For a quick test, Transformers includes a light OpenAI-compatible server. The page says it suits quick testing and moderate load. Start it, then send a request.

pip install "transformers[serving]>=5.18.0"
transformers serve openbmb/MiniCPM-V-4.7-35B-A3B --port 8000 --host 0.0.0.0

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

The page’s command listens on every network interface because of --host 0.0.0.0. Use 127.0.0.1 if only you need it. For instruct mode the pages recommend temperature 0.7, top_p 0.95, top_k 100, min_p 0.0, presence_penalty 0 and repetition_penalty 1.03.

Example

You have no big GPU and want to know what a photo shows. You build llama.cpp, download the 1B F16 file and its mmproj file from Hugging Face, and run the cookbook’s command from build/bin/.

./llama-mtmd-cli \
    -m  /path/to/MiniCPM-V-4.7-1B/MiniCPM-V-4_7-1B-F16.gguf \
    --mmproj /path/to/MiniCPM-V-4.7-1B/mmproj-model-f16.gguf \
    -c 8192 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 \
    --reasoning off \
    --image xx.jpg -p "What is in the image?"

The xx.jpg is the guide’s placeholder for your picture. The guide shows no output, so what you get depends on the photo. The GGUF files at the root of each repository switch thinking off in their template, so replies come back plain.

For newcomers and experts

If you are new. A multimodal model takes more than text. You give MiniCPM-V a picture or a video with a question, and it answers in text. MiniCPM-o also takes audio and can answer in speech. The 1B model is the small one. Try a live demo from the README first, to see what it does before you install anything. Then start with the 1B model and one route from the table.

For experts. The 4.7 page says a mixture-of-experts model’s weights often do not fit in device memory, and that loading experts on demand can stall decoding. Its answer is Lookahead Expert Scheduling, which predicts the experts the next layer needs and loads them early, with layer-wise quantization and DSpark speculative sampling. The page reports 13+ tokens a second with 3 GB of memory on an iPhone 17 Pro Max. The apps README adds that the 35B-A3B variant streams its experts from SSD and was validated on a 12 GB iPhone 17 Pro Max.

Two image settings matter in Transformers. The setting downsample_mode is 16x by default, and 4x keeps four times as many visual tokens for finer detail. It must go to both apply_chat_template and generate. The setting max_slice_nums defaults to 9, and the page recommends 36 for an image and 1 for video.

Thinking differs by route. The vLLM guide says its chat template turns thinking on by default, so pass enable_thinking false for plain replies. The hosted API has it off by default. The llama.cpp guide’s own commands use top-p 0.8 and repeat-penalty 1.05, while its sampling note recommends 0.95 and 1.03. DSpark draft heads speed up decoding, but llama-mtmd-cli does not support draft models, so use llama-cli or llama-server and pair each draft with the target of the same size.

Good to know

Most of it runs on your machine, and a few paths reach out.

• What installs: nothing from these notes. Each route downloads its own files: Python packages and weights, a llama.cpp build and GGUF files, a serving framework or a phone app
• Moving targets: on 3 November 2026 the Ollama guide said its bundled llama.cpp could not yet load 4.7 GGUF files, and the SGLang guide said to use a fork, tc-mb/sglang. Check both guides before you start
• Hosted routes: the hosted API and the live demos run on other servers, so your input leaves your machine. The API guide gives no price or data policy. The MiniCPM-o 4.5 page warns that its web demo can be slow or miss output when served from overseas
• Experimental: the MiniCPM-o 4.5 page calls it an experimental trial. It lists full-duplex streaming that still needs a stronger base, speech that can mispronounce characters in omni mode, and replies that mix English and Chinese
• The maker’s statement: the project page says the models’ output does not represent the developers’ views, and that they accept no liability for problems from using the models
• Phone apps: they fetch the model files after you install them. The 35B-A3B download is about 16.6 GB, so plan for about 17 GB of free storage
• Usage data: the README and the docs pages Mindket read say nothing about telemetry
• Removal: the README and the docs give no uninstall steps, and each route keeps its own downloads. Mindket’s notes say to remove them with that tool’s own means
• License: Apache-2.0. The LICENSE file reads “Copyright OpenBMB”. The model license section invites you to fill in an optional registration questionnaire