MoEViE-L16-384

MoEViE-L16-384

A Mixture-of-Experts vision encoder from Meta that matches images to text labels with no extra training, here the L/16 model at 384px. Licensed for non-commercial research use.

Made by facebook on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: MoEViE-L16-384 is one checkpoint of MoE-ViE, a family of vision encoders from Meta for image and video understanding. It is trained to link images with text, so it can sort a picture into labels you type without extra training. Each block pairs an always-on expert with a pool of routed ones, so only a small share of the weights runs for each token.

Who it suits: Researchers and engineers who study or build vision-language systems and can use a CUDA GPU.

Industry | Who uses it | How they use it | Example
Research | Computer vision researchers | Evaluate zero-shot image and video understanding | Score a set of images against typed labels
Media | Retrieval engineers | Match text queries with images or video clips | Search a photo set by text
Software | ML engineers | Use the encoder as a feature extractor in a larger system | Pull image and text features for a downstream model

Anyone building a commercial product or without a CUDA GPU can skip it, since the license is non-commercial and the kernels need CUDA.

Where to use it: It runs in Python on a machine with a CUDA GPU.

• Python with the open_clip package, set up from the project's repository
• A computer with a CUDA GPU, because the expert kernels compile with Triton at run time
• The repository's demo script and zero-shot evaluation suite

Use case | Who | What they get back
Zero-shot classification | Researchers | A probability for each label you provide
Text and image matching | Engineers | Text and image features you can compare for retrieval
Video clip understanding | Researchers | Results the card reports on Kinetics-400 and MSR-VTT

How to set it up: Setup means cloning the project's repository and installing its requirements.

• Python with pip
• A CUDA GPU
• The project's GitHub repository, cloned to your computer
• The packages listed in its requirements file

1. Clone the project's GitHub repository and open its folder.
2. Install the packages from requirements.txt.
3. Run the Python script in the next section, which downloads the model from the Hub the first time.
4. If you prefer, download the files yourself with the hf command.

cd moe_vie
pip install -r requirements.txt

pip install -U huggingface_hub
hf download facebook/MoEViE-L16-384

How to use it: Give the model an image and a list of text labels, and it prints a probability for each label. The card's script does this for one picture.

import torch
from PIL import Image
from open_clip import create_model_and_transforms, get_tokenizer, image_to_device

MEAN, STD = (0.5, 0.5, 0.5), (0.5, 0.5, 0.5)

model, _, preprocess = create_model_and_transforms(
    "MoEViE-L16-384",
    pretrained=True,   # downloads from the Hub
    force_preprocess_cfg=dict(
        patch_size=16, size_range=(384, 384), center_crop=True, window_size=1
    ),
    image_mean=MEAN, image_std=STD,
)
model = model.cuda().eval()
tokenizer = get_tokenizer("MoEViE-L16-384")

labels = ["a diagram", "a dog", "a cat"]
packed, _ = preprocess.collate_fn([(preprocess(Image.open("cat.png").convert("RGB")), 0)])
packed = image_to_device(packed, "cuda", torch.float32, mean=MEAN, std=STD)
text = tokenizer(labels).cuda()

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(packed, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    probs = (model.logit_scale.exp() * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", probs)

Tool | What it does
create_model_and_transforms | Loads the model and the matching image preprocessing
get_tokenizer | Prepares your text labels for the model
encode_image | Turns the image into a normalized feature vector
encode_text | Turns the labels into feature vectors you compare with the image

Example: You point the script at a photo called cat.png and give it three labels, a diagram, a dog and a cat. It prints a probability for each label. The card does not show the numbers, but the highest should land on the label that fits the picture.

For newcomers and experts: This model learned to connect pictures with words. You show it a picture and a few phrases, and it tells you which phrase fits best. The first small thing to try is running the card's script on one photo with three labels.

It is a contrastive vision-language encoder built from Mixture-of-Experts blocks. Each block has one shared expert that is always on plus a routed pool, with per-token top-k routing on a sigmoid gate. The L/16 model has width 1024, depth 24 and 4 of 32 experts active, counting the shared one, for 1.7B total and 0.3B activated parameters.

Start with an image and typed labels
Encode the image with the MoE encoder
Encode the labels as text features
Scale the image to text similarity
Apply softmax across the labels

Good to know: The numbers below come from the card's zero-shot results for the three model sizes.

Title: IN-1k zero-shot top-1 accuracy by size
B/16 at 224px: 79.3%
L/16 at 384px: 83.6%
H/14 at 448px: 85.1%

• Hardware: a CUDA GPU is required, because the expert kernels compile with Triton at run time.
• Size: this L/16 checkpoint has 1.7B total and 0.3B activated parameters.
• License: CC BY-NC 4.0, for non-commercial research use.
• Cost to run: none named, though you need a CUDA GPU.
• Data it reaches: model files fetched from the Hub on first use, plus the image and labels you supply.
• Not for: commercial use, since the license limits it to non-commercial research.

On Hugging Face, 13 November 2026

• Task: zero shot image classification
• Library: moe_vie
• Downloads in the last 30 days: 37
• Likes: 7
• License: cc-by-nc-4.0
• Tags: moe_vie, zero-shot-image-classification

Links

• Model: https://huggingface.co/facebook/MoEViE-L16-384
• Files: https://huggingface.co/facebook/MoEViE-L16-384/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download facebook/MoEViE-L16-384

Then follow the usage in its model card. Nothing in this kit runs by itself.