Llama-V-JEPA-LM-Llama3.1-8B

Llama-V-JEPA-LM-Llama3.1-8B

Answers questions about video clips by joining a frozen V-JEPA 2.1 video encoder to Llama-3.1-8B-Instruct, made for people working on video understanding.

Made by kkt94 on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: This is a video-language model, so video and text go in and text comes out. A frozen V-JEPA 2.1 video encoder feeds Meta's Llama-3.1-8B-Instruct through a small bridge. After the bridge, a clip of 32 frames becomes 1,024 visual tokens. Training drew on 4.53M public samples.

Who it suits: Researchers and engineers who work on video question answering will get the most from it, especially when motion and change over time matter. It also suits students who want to study how a video encoder joins a language model.

Industry | Who uses it | How they use it | Example
Research | Video-language researchers | Compare video understanding with other open models of a similar size | Testing how well it reads direction, speed and order
Robotics and simulation | Engineers | Ask how a scene changes from frame to frame | Asking which way an object moved
Media | Video analysts | Put questions to short clips and read the text answers | Asking what happens in a clip
Education | Students | Study how a world-model encoder joins a language model | Reading the architecture notes in the card

If you do not work with video, you can skip it. It also carries Meta's licence terms, which you should read first.

Where to use it:

• The transformers library, which Hugging Face lists for it
• Python on Windows, macOS or Linux, to download the files
• A machine with room for about 18.1 GB of BF16 weights, or about 4.5 GB at 4-bit as an estimate

Use case | Who | What they get back
Video question answering | Researchers | A text answer about a clip
Motion reading | Engineers | An answer about direction, speed, order or state
Model comparison | Students | A reference point against other open video models

How to set it up: You need Python with pip and enough memory for the weights. The source gives no run code, and its record says the run steps are on the model page.

• Python with pip, on Windows, macOS or Linux
• Disk space for about 18.1 GB in BF16
• The transformers library, listed for this model
• A read of the Llama 3.1 licence and acceptable use policy

1. Install the Hugging Face hub package.
2. Download the model files.
3. Run the model with the steps on the model page.

pip install -U huggingface_hub
hf download kkt94/Llama-V-JEPA-LM-Llama3.1-8B

How to use it: Give the model a clip of 32 frames, sampled evenly at 384 by 384, together with a text question. It writes a text answer. The source shows no call, so follow the model page for the exact code.

Example: You give it a short clip of a ball moving across a table and ask which way the ball went. The model reads the visual tokens for the clip and answers in text.

For newcomers and experts: A video-language model answers questions about video. A small first step is to read the card's highlights, which cover direction, speed, order and state, before you download the large weights.

The video encoder stays frozen at 1.01B parameters. The bridge first projects the encoder features onto 2,048 fixed PCA directions with one shared scale, then passes them through a trainable MLP of 13.2M parameters. The language model is fully fine-tuned at 8.03B, for 9.07B in total. Training uses counterfactual supervision, where pixel changes to public videos create variants whose answers are fixed by construction. The card reports its own scores, and the token count per clip stays at 1,024 whatever the resolution.

A clip of 32 frames at 384 by 384
The frozen V-JEPA 2.1 encoder reads it
The spectral bridge makes 1,024 visual tokens
Llama-3.1-8B-Instruct reads tokens and question
The model writes a text answer

Title: Average over 12 video scores, as the card reports
Llama-V-JEPA-LM-Llama3.1-8B: 70.51 points
PLM-8B: 64.79 points

Good to know:

• License: Llama 3.1 Community License, shown as llama3 on Hugging Face
• The V-JEPA 2 encoder is MIT, and the FGQA training data follows the Llama 3.2 Community License
• Cost to run: none named
• Data it reaches: the clips you feed it, and the source names no web or account access
• Not for: uses that Meta's Llama 3.1 Acceptable Use Policy forbids
• English is the only listed language

On Hugging Face, 13 November 2026

• Task: video text to text
• Library: transformers
• Parameters: 9.1B
• Weights: about 18.1 GB in BF16; about 4.5 GB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 120
• Likes: 8
• License: llama3.1
• Languages: en
• Based on: meta-llama/Llama-3.1-8B-Instruct
• Tags: transformers, safetensors, jepa_llm, feature-extraction, video, video-question-answering, video-language-model, multimodal, v-jepa, world-model, llama, llama-3

Links

• Model: https://huggingface.co/kkt94/Llama-V-JEPA-LM-Llama3.1-8B
• Files: https://huggingface.co/kkt94/Llama-V-JEPA-LM-Llama3.1-8B/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download kkt94/Llama-V-JEPA-LM-Llama3.1-8B

Then follow the usage in its model card. Nothing in this kit runs by itself.