A foundation model of vision, audition, and language for in-silico neuroscience (TRIBE v2)

TL;DR

TRIBE v2 is a tri-modal (video, audio, language) foundation model that predicts whole-brain fMRI responses to arbitrary naturalistic and experimental stimuli. Trained on a unified dataset of >1,000 hours of fMRI from 720 subjects, it generalizes zero-shot to new subjects, stimuli, and tasks, delivering several-fold improvements over linear encoding models [§Abstract]. Beyond prediction accuracy, the paper claims TRIBE v2 enables in-silico neuroscience: rerunning classical visual and neuro-linguistic experiments on the model recovers established empirical findings [§Abstract]. Successor to the Algonauts 2025 winner TRIBE v1 ([arxiv:2507.22229]); checkpoints, code, and demo released.

Key claims

TRIBE v2 fuses pretrained text (Llama-class), audio (w2v / wav2vec-class), and video (V-JEPA-class) representations through a prediction transformer that maps multimodal features onto a cortical-surface fMRI target [§Approach / Fig. 1].
Zero-shot generalization to held-out subjects, stimuli, and tasks is the headline capability — i.e. the model is trained once on a 720-subject corpus and then predicts a new person’s fMRI to a new movie or podcast without per-subject fine-tuning [§Abstract].
Reported improvement over traditional linear encoding models is “several-fold” in accuracy across naturalistic conditions [§Abstract]; the Meta blog post quantifies a “nearly 2–3×” improvement vs. prior methods for both movies and audiobooks.
In-silico replication: when TRIBE v2 is run on stimuli from seminal visual and neuro-linguistic paradigms, the predicted brain responses recover decades of established empirical results, suggesting the model can be queried as a stand-in for additional fMRI experiments [§Abstract].
Latent features of TRIBE v2 are claimed to be interpretable and to expose a fine-grained topography of multisensory integration on the cortical surface [§Abstract].
Training corpus: >1,000 hours of fMRI, 720 subjects [§Abstract]; companion blog says 500+ hours and 700+ people, suggesting the announcement framing rounds down on a larger held-back set.

Method

TRIBE v2 follows the v1 recipe but scales it. Three frozen unimodal foundation backbones (a Llama-class language model, a w2v-class audio encoder, and a V-JEPA-class video encoder) extract time-aligned embeddings from the stimulus stream. A transformer then ingests these multimodal embeddings across time and predicts high-resolution fMRI BOLD responses on the cortical surface for the target subject and timepoint. The fMRI prediction head is structured so that a single shared encoder handles common cortical-response patterns across the 720-subject corpus, with subject information injected through embeddings rather than per-subject heads — which is what enables zero-shot generalization to unseen subjects. Training uses BOLD targets recorded while participants watch movies (Friends, Hidden Figures, etc., extending v1’s Algonauts corpus) and listen to podcasts.

The “in-silico experimentation” claim is operationalized by running TRIBE v2’s forward pass on stimuli from canonical fMRI experiments (visual localizers, neuro-linguistic paradigms) and checking whether the predicted activations recover the spatial topography that the original empirical literature reports.

Results

Headline numbers from the abstract and Meta announcement, pending PDF verification:

“Several-fold” accuracy improvement over linear encoding baselines on prediction of held-out fMRI responses [§Abstract].
~2–3× improvement over prior brain-encoding methods for both movies and audiobooks (per Meta announcement; the paper presumably contains the per-modality table).
Recovers known cortical-response topographies in in-silico re-runs of classical visual and language paradigms [§Abstract].
v1 (TRIBE, arxiv:2507.22229) won 1st place at Algonauts 2025 with a “significant margin” — v2 supersedes that model with more data and a generalization-across-subjects regime.

Why it’s interesting

Brain-encoding models are an orthogonal use of foundation-model representations from anything else in the wiki: instead of generating video/audio/text or understanding it, TRIBE v2 uses pretrained representations as a regressor onto neural activity, and the resulting model is then itself a scientific instrument. The closest existing wiki neighbour is V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning — V-JEPA 2.1 is the kind of video SSL backbone that TRIBE v2 likely relies on for the video stream, and Meta’s release positions video-SSL features as the substrate that makes high-resolution brain prediction tractable. The “frozen pretrained backbones fused by a small task-specific transformer” recipe also rhymes with the unified-multimodal stream of The Design Space of Tri-Modal Masked Diffusion Models (text + image + video diffusion), but here the output space is BOLD signal rather than pixels — a useful sanity check on whether tri-modal representations carry enough signal to predict a third, unrelated modality (neural activity). The zero-shot-to-new-subjects framing is also of note for Luma’s broader interest in foundation models that generalize across instances without per-target fine-tuning, similar in spirit to the cross-subject/cross-task generalization themes in Skillful joint probabilistic weather forecasting from marginals (one weather foundation model, many basin/site predictions zero-shot).