TIPSv2 B/14

TIPSv2 B/14

Turns images and text into matching embeddings, with spatial patch features, for zero-shot image classification. Google's Base variant of TIPSv2, for vision researchers and developers.

Made by google on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: TIPSv2 B/14 is the Base variant of Google's TIPSv2 family, contrastive vision-language models that turn images into features with spatial detail that line up with text embeddings. It has 86M vision parameters and 110M text parameters. The record tags it for zero-shot image classification.

Who it suits: Researchers and developers who need image features aligned with text for classification, search or spatial analysis.

Industry | Who uses it | How they use it | Example
Research | Vision researchers | Test text and image alignment and spatial features | Show the patch features as a color image
Software | ML engineers | Classify images against a list of text labels without training | Choose between bus, car, dog and cat
Search | Retrieval developers | Compare text embeddings with image embeddings | Match a text query to a set of pictures

Anyone who wants a finished app can skip it, since this is an encoder you wire up in code.

Where to use it: It runs in Python through Hugging Face Transformers, on a CPU or a CUDA GPU.

• Python with Transformers, loading the model through AutoModel
• A CUDA GPU, if you move the model over with model.cuda()
• The project's GitHub repository, which holds extra examples and plots, among them zero-shot segmentation
• Windows, macOS or Linux with Python, for downloading the files

Use case | Who | What they get back
Zero-shot classification | ML engineers | The best-matching label from your own list
Text and image embeddings | Developers | Vectors of 768 numbers you can compare
Spatial feature maps | Researchers | A 32 by 32 grid of patch features that can be drawn as colors

How to set it up: You need Python and a handful of packages.

• Python with pip
• The transformers, torch, torchvision, sentencepiece and scikit-learn packages
• Images turned into tensors in the 0 to 1 range
• Optionally, a CUDA GPU

1. Install the packages with pip.
2. Load the model with AutoModel, using trust_remote_code set to True.
3. If you like, download the files yourself with the hf command.

pip install transformers torch torchvision sentencepiece scikit-learn

pip install -U huggingface_hub
hf download google/tipsv2-b14

How to use it: Load the model, turn your image into a tensor, then encode the image and your text. The card says to skip ImageNet normalization and use only ToTensor, so values stay between 0 and 1.

from transformers import AutoModel

model = AutoModel.from_pretrained("google/tipsv2-b14", trust_remote_code=True)
model.eval()

Open your own picture as a PIL image called image, then encode it at 448 by 448. The card's sample returns a global embedding of shape (1, 1, 768) and per-patch features of shape (1, 1024, 768).

from torchvision import transforms
from PIL import Image

transform = transforms.Compose([
    transforms.Resize((448, 448)),
    transforms.ToTensor(),
])

pixel_values = transform(image).unsqueeze(0)
out = model.encode_image(pixel_values)

Text queries give one embedding each, so two queries return shape (2, 768).

text_emb = model.encode_text(["a photo of a bus", "a photo of a dog"])

For zero-shot classification, normalize both sides, compare them and take the closest label.

import torch.nn.functional as F

classes = ["bus", "car", "dog", "cat"]
cls = F.normalize(out.cls_token[:, 0, :], dim=-1)
text_emb = F.normalize(model.encode_text(classes), dim=-1)
similarity = cls @ text_emb.T
print(classes[similarity.argmax()])

On a GPU, move the model and the input across.

model = model.cuda()
out = model.encode_image(pixel_values.cuda())
text_emb = model.encode_text(["a city"])

Tool | What it does
encode_image | Returns a global image embedding in cls_token and per-patch features in patch_tokens
encode_text | Returns one 768-number embedding for each text query
PCA on patch_tokens | Turns the spatial features into a color picture, as the card's sample does

Example: You give it a photo and the labels bus, car, dog and cat. It normalizes both embeddings, compares them and prints the label with the highest similarity, which the card's own sample shows as bus.

For newcomers and experts: This model gives a picture and a sentence each a list of numbers, so the two can be compared. When the lists are close, the sentence fits the picture. The first small thing to try is loading the model and comparing one photo with a few labels.

The vision side is a ViT encoder with 12 layers and the text side is a Transformer with 12 layers, joined by an embedding size of 768. The patch size is 14 by 14 pixels. Text goes through a SentencePiece tokenizer, lowercased, with at most 64 tokens, and images can be any resolution.

Turn the image into a 0 to 1 tensor
Encode it with encode_image
Encode your labels with encode_text
Normalize both and compare them
Pick the closest label

Good to know: The family comes in four sizes, and this card describes the smallest one, the Base variant.

Variant | Vision params | Text params | Embed dim
B/14 | 86M | 110M | 768
L/14 | 303M | 184M | 1024
SO400m/14 | 412M | 448M | 1152
g/14 | 1.1B | 389M | 1536

• Remote code: the loading line sets trust_remote_code to True, which lets the repository's own code run.
• Inputs: images are tensors in the 0 to 1 range, and text is limited to 64 tokens.
• License: Apache 2.0.
• Cost to run: none named.
• Data it reaches: the images you load and the model files from the Hub. The card's own sample also fetches an example image over the web.

On Hugging Face, 15 November 2026

• Task: zero shot image classification
• Library: transformers
• Parameters: 196M
• Weights: about 784 MB in F32; about 98 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 8,629
• Likes: 137
• License: apache-2.0
• Quantized versions: 1
• Fine-tunes: 5
• Tags: transformers, safetensors, tipsv2, feature-extraction, vision, image-text, contrastive-learning, zero-shot, zero-shot-image-classification

Links

• Model: https://huggingface.co/google/tipsv2-b14
• Files: https://huggingface.co/google/tipsv2-b14/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download google/tipsv2-b14

Then follow the usage in its model card. Nothing in this kit runs by itself.