TRIBE v2

TRIBE v2

Estimates how the brain responds, as seen in fMRI, to video, audio and text, using a multimodal Transformer that maps onto the cortical surface. For neuroscience and AI researchers.

Made by facebook on Hugging Face. These notes are Mindket's, written from its model page and model card; the model's own files and license are on Hugging Face.

About

What it is: TRIBE v2 is a brain encoding model. You give it a video, an audio clip or a text, and it estimates the fMRI response a brain would show, laid out over the cortical surface. It joins three feature extractors, LLaMA 3.2 for text, V-JEPA2 for video and Wav2Vec-BERT for audio, inside one Transformer. Its title calls it a base model across sight, hearing and language, meant for neuroscience experiments run on a computer.

Who it suits: Researchers who study how the brain handles sight, sound and language, and who are comfortable with Python and neuroimaging data.

Industry | Who uses it | How they use it | Example
Neuroscience | Brain imaging researchers | Predict fMRI responses to videos, audio or text | Compare predicted activity for two video clips
Cognitive science | Language researchers | Study how spoken words map onto cortical responses | Predict responses to a recorded sentence
AI research | Multimodal model researchers | Study a Transformer that maps features onto the cortex | Train a new model on the card's study datasets with Slurm

You can skip it if you want a chat, writing or image model, or if you need commercial use, since the license is non-commercial.

Where to use it: It runs in Python 3.10 or newer, and the card links a Colab demo notebook with brain visualizations.

• A Python environment on Windows, macOS or Linux
• The card's Colab demo notebook
• A Slurm cluster, for the grid-search training runs

Use case | Who | What they get back
Brain response prediction | Neuroscientists | An array of predictions shaped by timesteps and vertices
Brain visualization | Researchers | Plots of predicted activity, with the plotting extra installed
Training from scratch | ML researchers | A model trained on the study datasets the card names

How to set it up: Install the code from the project's repository, then load the pretrained weights from Hugging Face. Inference needs only the basic install.

• Python 3.10 or newer
• The tribev2 code from the facebookresearch project
• A Hugging Face account that can open the restricted LLaMA 3.2-3B model, because the text side of the model depends on it

1. Get the tribev2 code and open its folder.
2. Install it with pip in editable mode.
3. Add the plotting or training extra only if you need it.
4. Load the pretrained model in your own script, as shown in the next section.

pip install -e .
pip install -e ".[plotting]"
pip install -e ".[training]"

pip install -U huggingface_hub
hf download facebook/tribev2

How to use it: Load the pretrained model, build an events table from a video, and ask for predictions. You can pass a text or audio path to the same call, and text is turned into speech and transcribed to get word timings.

from tribev2 import TribeModel

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")

df = model.get_events_dataframe(video_path="path/to/video.mp4")
preds, segments = model.predict(events=df)
print(preds.shape)  # (n_timesteps, n_vertices)

To train from scratch, set the data and output paths, sign in to Hugging Face, then run the test or the grid searches.

export DATAPATH="/path/to/studies"
export SAVEPATH="/path/to/output"
export SLURM_PARTITION="your_partition"
huggingface-cli login
python -m tribev2.grids.test_run
python -m tribev2.grids.run_cortical
python -m tribev2.grids.run_subcortical

Tool | What it does
TribeModel.from_pretrained | Loads the pretrained model from Hugging Face into a cache folder
get_events_dataframe | Builds the events table from a video, text or audio path
model.predict | Returns the brain-response predictions and their segments
pip install with the plotting extra | Adds brain visualization
pip install with the training extra | Adds training dependencies such as PyTorch Lightning and W&B

Example: Load the model, point get_events_dataframe at a video file, and call predict. Printing the shape of the result gives the number of timesteps by the number of vertices on the cortical mesh.

For newcomers and experts: fMRI measures brain activity, and this model guesses what that activity would look like while someone watches or hears something. A first small try is the card's Colab demo notebook, which walks through the steps with brain plots.

For experts, the predictions are for an average subject and sit on the fsaverage5 cortical mesh of about 20k vertices. The model fuses LLaMA 3.2, V-JEPA2 and Wav2Vec-BERT features in a unified Transformer, and the project code covers training with PyTorch Lightning and surface projection. The trade is group-level output, not predictions for one person, under a non-commercial license.

Video, audio or text goes in
Three feature extractors encode it
A unified Transformer fuses them
Predictions land on the cortical mesh

Good to know: Read these limits before you plan a study.

• Average subject: predictions describe an average subject, as the card explains.
• Mesh: the output lives on fsaverage5, about 20k vertices.
• Restricted model: the text side depends on LLaMA 3.2-3B, which needs approved access.
• Text input: text is converted to speech and transcribed to get word timings.
• License: CC-BY-NC-4.0, which is non-commercial.
• Cost to run: none named. Training names a Slurm partition and optional W&B, and no paid API is mentioned.
• Data it reaches: the video, audio or text files you give it, plus the weights downloaded into a cache folder.
• Not for: commercial use, or predictions for an individual subject.

On Hugging Face, 13 November 2026

• Downloads in the last 30 days: 25,771
• Likes: 760
• License: cc-by-nc-4.0
• Quantized versions: 3
• Fine-tunes: 3

Links

• Model: https://huggingface.co/facebook/tribev2
• Files: https://huggingface.co/facebook/tribev2/tree/main

Get the model

Download its files with Hugging Face's command line:

pip install -U huggingface_hub
hf download facebook/tribev2

Then follow the usage in its model card. Nothing in this kit runs by itself.