Florence-2-base

Florence-2-base

A small Microsoft vision model that handles captioning, object detection and segmentation by changing the text prompt. It suits developers and researchers who need several image tasks from one model.

Made by microsoft on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Florence-2-base is the 232 million parameter version of Microsoft's Florence-2 vision model, packaged for the transformers library. One model covers several image jobs, and you choose the job with a short task prompt, such as captioning, object detection or segmentation. It was trained on FLD-5B, a set with 5.4 billion annotations over 126 million images, and it works zero-shot or after fine-tuning.

Who it suits: It suits developers and researchers who want one compact vision model for several tasks instead of a separate model for each. It runs on a CPU or a CUDA GPU.

Title: Who might use it
Industry | Who uses it | How they use it | Example
Accessibility | App developers | Draft written descriptions of images | Ask for a detailed caption of a photo
Computer vision | Vision engineers | Find objects in a picture with a prompt | Run the object detection prompt on a street scene
Research | Machine learning researchers | Fine-tune a small base model on their own images | Start from the base size and adapt it to a new task

If you need the strongest accuracy, you can skip the base size. The card also lists a larger 0.77 billion version.

Where to use it: The card shows it in Python through the transformers library, on a GPU when one is present and the CPU otherwise.

• Python with the transformers library
• A CUDA GPU or a plain CPU
• Windows, macOS or Linux with Python

Title: Jobs it does
Use case | Who | What they get back
Image captioning | Developers | A short, detailed or more detailed caption
Object detection | Vision engineers | Detected objects with their boxes
Segmentation and more | Researchers | Results for other tasks chosen by the prompt

How to set it up: You need Python with pip, the transformers library, PyTorch and Pillow. Download the model files, then load them in your script.

• Python with pip
• The transformers library and PyTorch
• Pillow, to open your own image

1. Install the Hugging Face download tool.
2. Download the model files.
3. Load the model and processor in Python.

pip install -U huggingface_hub
hf download microsoft/Florence-2-base

How to use it: Load the model and processor, open an image with Pillow as the variable named image, pick a task prompt and generate. The card's own sample downloads a car photo from the web, which is left out here. The model works in float16 on a GPU.

from transformers import AutoProcessor, AutoModelForCausalLM
device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-base", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-base", trust_remote_code=True)
prompt = "<OD>"
inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)
generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=1024,
    do_sample=False,
    num_beams=3,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))
print(parsed_answer)

The card then defines a helper called run_example and swaps the prompt to change the task. These are the caption prompts it shows.

prompt = "<CAPTION>"
run_example(prompt)
prompt = "<DETAILED_CAPTION>"
run_example(prompt)
prompt = "<MORE_DETAILED_CAPTION>"
run_example(prompt)

Example: You open a photo of a street, set the object detection prompt and run the script. You get back parsed objects with their positions. Switching to the detailed caption prompt returns a written description of the same photo.

For newcomers and experts: If you are new, think of it as one model you steer with a short instruction. Start with the caption prompt on a photo you know, and read the text it returns.

If you know the field, it is a sequence-to-sequence model that turns every task into text, so one set of weights serves many tasks. The card lists four sizes, two pretrained and two fine-tuned. The sample sets trust_remote_code, so check that setting against your security rules.

An image and a task prompt go in
The processor prepares the inputs
The model generates text
The result is parsed into the task output

Good to know: The family comes in two sizes, each pretrained on FLD-5B or fine-tuned on downstream tasks.

Title: Model size in billions of parameters
Florence-2-base: 0.23
Florence-2-large: 0.77
Florence-2-base-ft: 0.23
Florence-2-large-ft: 0.77

• License: MIT
• Cost to run: none named
• Data it reaches: the model files from Hugging Face, and the images you load
• Not for: work that needs the larger 0.77 billion model's capacity

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 232M
• Weights: about 463 MB in F16; about 116 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 14,139,837
• Likes: 439
• License: mit
• Quantized versions: 6
• Fine-tunes: 21
• Tags: transformers, pytorch, safetensors, florence2, image-text-to-text, vision

Links

• Model: https://huggingface.co/microsoft/Florence-2-base
• Files: https://huggingface.co/microsoft/Florence-2-base/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download microsoft/Florence-2-base

Then follow the usage in its model card. Nothing in this kit runs by itself.