MOSS-Video-Preview-Real-Time-SFT

MOSS-Video-Preview-Real-Time-SFT

Comments on video while it plays, taking frames one by one and answering with low delay. For developers and researchers building live video assistants in Python with transformers.

Made by OpenMOSS-Team on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: MOSS-Video-Preview-Real-Time-SFT is a video-and-text model of about 10.7B parameters from OpenMOSS-Team. It was derived from MOSS-Video-Preview-SFT with an extra round of real-time supervised fine-tuning. The aim is quick answers to a stream of frames, so it can speak about a scene while the footage is still arriving.

Who it suits: It suits developers and researchers who work on live video understanding and can run a GPU or NPU machine.

Industry | Who uses it | How they use it | Example
Interactive apps | Developers | Build assistants that comment on video as it plays | Describe a video feed frame by frame
Monitoring | Operations teams | Watch a continuous stream and get instant feedback on actions | Report a change in a scene as it happens
Research | Video-language researchers | Study streaming cross-attention and silence-and-speak switching | Interrupt the model mid-answer and watch it correct itself

Anyone who needs a small model for a phone or laptop can skip it, since the BF16 weights are about 21.3 GB.

Where to use it: You run it from Python with transformers on hardware the card names.

• Python with transformers, on a CUDA or NPU machine
• A script that feeds frames from a video file or a continuous stream
• Windows, macOS or Linux, with Python, for downloading the files

Use case | Who | What they get back
Live scene commentary | App developers | Text that follows the video as frames arrive
Continuous monitoring | Operations teams | Quick text feedback on what is happening
Streaming research | Researchers | A checkpoint for testing low-delay video understanding

How to set it up: You need Python, a capable GPU or NPU and a video file to try.

• Python with the transformers library
• OpenCV (cv2) and Pillow (PIL), which the card's script imports
• A CUDA or NPU machine with room for the weights, about 21.3 GB in BF16
• A video file of your own

1. Install the Hugging Face hub tools.
2. Download the model files.
3. Install transformers, OpenCV and Pillow.
4. Save the script from How to use it and point video_path at your video.

pip install -U huggingface_hub
hf download OpenMOSS-Team/moss-video-preview-realtime-sft

How to use it: The card recommends streaming inference through the real_time_generate() API. The script below feeds frames from a video, sends a prompt, and starts generation in its own thread. The card's script then reads tokens from token_queue, but that last loop is cut off in the text shown here.

import os, queue, threading, time, cv2
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

def feed(video, q, fps=1.0):
    cap=cv2.VideoCapture(video); step=max(1, round((cap.get(cv2.CAP_PROP_FPS) or 25)/fps)); i=0
    while cap.isOpened():
        ok, f = cap.read()
        if not ok: break
        if i % step == 0: q.put(Image.fromarray(cv2.cvtColor(f, cv2.COLOR_BGR2RGB))); time.sleep(1/fps)
        i += 1
    cap.release()

checkpoint = "OpenMOSS-Team/moss-video-preview-realtime-sft"
video_path = "data/example_video.mp4"
prompt = "Describe the video."

processor=AutoProcessor.from_pretrained(checkpoint, trust_remote_code=True)
model=AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True, device_map="auto")

image_queue, prompt_queue, token_queue = queue.Queue(), queue.Queue(), queue.Queue()
threading.Thread(target=feed, args=(video_path, image_queue), daemon=True).start()
time.sleep(1)
prompt_queue.put(prompt)
threading.Thread(
    target=lambda: model.real_time_generate(image_queue, prompt_queue, token_queue, processor),
    daemon=True,
).start()

Tool | What it does
real_time_generate | Generates text while frames keep arriving, using an image queue, a prompt queue, a token queue and the processor
feed | The card's helper that reads a video file and queues about one frame per second
AutoProcessor | Loads the processor for the checkpoint
AutoModelForCausalLM | Loads the model for the checkpoint

Example: You point video_path at a short clip and keep the default prompt, Describe the video. The helper queues about one frame a second. The model starts producing text as frames arrive, without waiting for the clip to end.

For newcomers and experts: The model works as a see-and-say system. It looks at video frames as they come in and talks about them without waiting for the whole clip. A good first try is to run the card's script on a short clip of your own.

For someone who knows the field, the card describes an image-video unified cross-attention design with native frame-by-frame injection and a cross-attention KV cache. Frames enter at a steady rate through a non-blocking pipeline that is decoupled from text generation, which keeps time to first token low. The model is also tuned to switch between staying silent and speaking, and it can be interrupted and revise its answer.

Video frames join an input queue
Cross-attention KV cache keeps context
Model decides to stay silent or speak
Tokens stream out as they are made

Good to know: The card calls this a preview checkpoint tuned for real-time use. It lists Flash Attention 2 support and compatibility with CUDA and NPU. Weights are about 21.3 GB in BF16 and about 5.3 GB at 4-bit by estimate. It handles English and Chinese.

• The script passes trust_remote_code=True, which runs code from the model repository, so read that code first
• License: Apache-2.0
• Cost to run: none named, but you need a GPU or NPU machine with enough memory
• Data it reaches: the video file you point it at, plus the model files it downloads from Hugging Face

On Hugging Face, 13 November 2026

• Task: video text to text
• Library: transformers
• Parameters: 10.7B
• Weights: about 21.3 GB in BF16; about 5.3 GB at 4-bit (estimate)
• Formats: Safetensors
• Downloads in the last 30 days: 74
• Likes: 30
• License: apache-2.0
• Languages: en, zh
• Based on: OpenMOSS-Team/moss-video-preview-sft
• Tags: transformers, safetensors, video_mllama, text-generation, multimodal, video, vision-language, mllama, streaming, realtime, low-latency, video-text-to-text

Links

• Model: https://huggingface.co/OpenMOSS-Team/moss-video-preview-realtime-sft
• Files: https://huggingface.co/OpenMOSS-Team/moss-video-preview-realtime-sft/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download OpenMOSS-Team/moss-video-preview-realtime-sft

Then follow the usage in its model card. Nothing in this kit runs by itself.