Florence-2-large

Florence-2-large

Captions images, finds objects and reads text in pictures when you pair an image with a short task prompt. Made for developers and researchers who run vision tasks in Python with transformers.

Made by microsoft on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: Florence-2-large is a vision model from Microsoft. You give it an image and a short text prompt, and it answers with text for tasks such as captioning, object detection and segmentation. This copy is a continued pretrained version with a 4k context length and 777M parameters. Its OCR output now separates lines with a newline.

Who it suits: It suits developers and researchers who want one model for several image tasks, chosen by changing the prompt instead of swapping models.

Industry | Who uses it | How they use it | Example
Retail | Catalog engineers | Caption product pictures and detect the objects in them | Drafting a short description for a product photo
Research | Computer vision researchers | Study a prompt-based vision model and compare its sizes | Comparing large with the base model on the same images
Document work | App developers | Run the OCR task on page images or screenshots | Reading the text from a screenshot, line by line
Accessibility | App teams | Describe images in words for people who cannot see them | Drafting a text description for a photo

Skip it if your work is text only, since this model always works on an image paired with a task prompt.

Where to use it:

• Python scripts that use the transformers library
• Windows, macOS and Linux
• A GPU, which the card's code picks when one is present, or a CPU

Use case | Who | What they get back
Caption an image | Developers | A written description of the picture
Detect objects | Computer vision teams | Detected objects as a parsed answer
Read text in a picture | App developers | Recognized text, with line breaks between lines

How to set it up:

• Python with the transformers, torch and Pillow packages
• Roughly 1.6 GB of disk space for the weights in F16
• A GPU is optional, since the card's code falls back to the CPU

1. Install the Python packages the card's code imports.
2. Download the model files, or let the code fetch them on first run.
3. Load the model and processor, with trust_remote_code set to True.

pip install -U huggingface_hub
hf download microsoft/Florence-2-large

How to use it: You choose a task prompt, pass it with an image to the processor, and call generate. Then post_process_generation turns the raw text into a parsed answer for that task. The card's own example uses the object detection prompt.

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch_dtype, trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

prompt = "<OD>"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, torch_dtype)

generated_ids = model.generate(
    input_ids=inputs["input_ids"],
    pixel_values=inputs["pixel_values"],
    max_new_tokens=4096,
    num_beams=3,
    do_sample=False
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]

parsed_answer = processor.post_process_generation(generated_text, task="<OD>", image_size=(image.width, image.height))

print(parsed_answer)

Tool | What it does
AutoProcessor | Prepares the text prompt and the image for the model
AutoModelForCausalLM | Loads the model, with trust_remote_code set to True
generate | Produces the answer tokens for the prompt and image
post_process_generation | Turns the generated text into a parsed answer for the task

Example: You hand the model a picture and the prompt <OD>. The script decodes the generated text and prints the parsed answer, which holds the objects the model found in the picture. The image variable is a PIL image you load yourself.

For newcomers and experts: Think of it as one model that you steer with a short instruction. A good first try is to run the card's object detection code on a single picture and read what it prints.

If you know the field, note that it uses a sequence-to-sequence design and was trained on the FLD-5B dataset, which the card puts at 5.4 billion annotations over 126 million images. The card reports a COCO object detection AP of 39.8 for this version. This version had only 0.1B samples of continued pretraining.

Pick a task prompt
Processor packs prompt and image
Model generates answer tokens
Post-processing parses the answer

Good to know:

• Training note: Microsoft says this continued pretraining used few samples, so the model might not be trained well.
• Size: 777M parameters, with weights of about 1.6 GB in F16.
• Siblings: the card lists base and large sizes, plus fine-tuned versions of each.
• License: MIT.
• Cost to run: none named.
• Data it reaches: it downloads model files from Hugging Face, and the card's sample also fetches a picture from the web.
• Not for: the card lists no forbidden uses, but warns that this version may be under-trained.

On Hugging Face, 13 November 2026

• Task: image text to text
• Library: transformers
• Parameters: 777M
• Weights: about 1.6 GB in F16; about 388 MB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 403,113
• Likes: 1,889
• License: mit
• Quantized versions: 3
• Fine-tunes: 43
• Tags: transformers, pytorch, safetensors, florence2, image-text-to-text, vision

Links

• Model: https://huggingface.co/microsoft/Florence-2-large
• Files: https://huggingface.co/microsoft/Florence-2-large/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download microsoft/Florence-2-large

Then follow the usage in its model card. Nothing in this kit runs by itself.