ReferEverything: Towards Segmenting Everything We Can Speak of in Videos

TL;DR

REM is a referring video object segmentation (Ref-VOS) framework built by repurposing a pretrained text-to-video diffusion model (ModelScopeT2V): instead of predicting noise, the model is fine-tuned to predict mask latents conditioned on a natural-language phrase, while keeping the rest of the diffusion architecture intact. The bet is that the diffusion model’s universal visual-language mapping — learned on Internet-scale video-text data — transfers more cleanly to referring segmentation than purpose-built RVOS networks trained only on small mask datasets. REM matches SOTA on in-domain benchmarks (Ref-DAVIS, Ref-YTB) but outperforms by up to 12 IoU points out-of-domain, and generalizes to non-object dynamic concepts (“waves crashing”, “smoke”, “raindrops”) that standard RVOS models cannot describe. The authors release a new eval-only benchmark, Referring Video Process Segmentation (Ref-VPS), to quantify this generalization.

Key claims

The core architectural change is shifting the diffusion objective from noise prediction to mask-latent prediction while preserving the rest of the generative model’s representation [Abstract; §3].
A two-stage fine-tuning recipe is used: first tune only the spatial weights on Ref-COCO image-text pairs for 1 epoch, then jointly tune all weights for 40 epochs on Ref-YTB video-text plus 12k Ref-COCO samples [Appendix / §3.2].
REM is on par with SOTA RVOS methods on in-domain Ref-DAVIS and Ref-YTB [Abstract; §4 / Table 1].
REM outperforms prior SOTA by up to 12 IoU points on out-of-domain benchmarks (BURST, VSPW, Ref-VPS) [Abstract; §4].
The model generalizes to non-object dynamic concepts (smoke, raindrops, waves crashing) — categories never present in the mask training set — because the underlying video diffusion model already has visual concepts for them [Abstract; §4 / Fig. on Ref-VPS].
A new eval-only benchmark, Ref-VPS (Referring Video Process Segmentation), is introduced specifically to measure this — dynamic phenomena at the intersection of video and language [Abstract; Appendix §A].
Compared methods are evaluated by re-running official checkpoints (MUTR, UNINEXT, VD-IT) on BURST/VSPW/Ref-VPS rather than relying on numbers from their original papers [Appendix Table C].
Robust to frequent occlusions, POV changes, and distortions, where prior SOTA (UNINEXT, VD-IT) drops tracking [§4 / project-page demos].

Method

REM starts from ModelScopeT2V, a U-Net-based text-to-video latent diffusion model. The authors keep the VAE, text encoder, and spatiotemporal U-Net intact, but reframe the denoising target: instead of denoising RGB latents from noise, the model predicts a mask latent — a latent encoding of the per-frame segmentation mask for the referred entity — from RGB video latents and the natural-language phrase. The cross-attention pathway that the original T2V used to condition image content on text is now repurposed to condition mask content on the referring expression.

Training proceeds in two stages to avoid catastrophic forgetting of the diffusion prior. Stage 1 fine-tunes only the spatial (per-frame) weights on Ref-COCO image-text-mask triples for one epoch, which is enough to teach the model the “predict mask latent” objective on a single frame. Stage 2 unfreezes all weights and trains jointly on Ref-YTB video-text-mask sequences plus a 12k slice of Ref-COCO for 40 epochs, giving the model temporal mask consistency without erasing its spatial grounding. At inference, REM denoises the mask latent conditioned on the input video latent and the referring phrase, then decodes to a per-frame mask.

The headline design choice is conservatism: the architecture, objective form, and supervision pattern stay as close as possible to the pretrained generative model’s, so the visual-language representations learned during T2V pretraining are not overwritten by the narrow Ref-VOS fine-tune. This is what the authors credit for the out-of-distribution generalization to non-object concepts.

Results

In-domain (Ref-DAVIS, Ref-YTB): on par with SOTA RVOS methods (UNINEXT, MUTR, VD-IT) on region similarity (J) and contour accuracy (F) [§4 / Table 1].
Out-of-domain (BURST, VSPW, Ref-VPS): up to +12 IoU points over the strongest prior baseline (VD-IT) [Abstract; §4].
Non-object concept segmentation: REM produces coherent masks for “waves crashing”, “smoke from a chimney”, and similar process queries; prior RVOS models either return empty or anchor onto an unrelated rigid object [§4; project page].
Robustness: qualitative comparisons on the project page show REM tracking through heavy occlusion, POV swings, and motion blur where UNINEXT/VD-IT drop the target entity.
Ablations: preserving the full diffusion prior (vs replacing decoder layers or training from scratch) is the dominant factor in the out-of-domain gain [§4 / ablation table].

Why it’s interesting

REM is the cleanest demonstration the wiki has of generative video models as universal perception backbones: rather than training a discriminative head on top of a frozen feature extractor, the entire denoising U-Net is repurposed by changing what it denoises. That’s the same intuition behind Image Generators are Generalist Vision Learners (Vision Banana) (image generators as generalist vision learners), Back to the Features: DINO as a Foundation for Video World Models (DINO features for video world models), and Video Analysis and Generation via a Semantic Progress Function — REM is the 2024 Ref-VOS instance of the broader pattern. It also lands directly in Layered Image/Video Decomposition territory: REM produces per-frame mask layers conditioned on text, which is the exact primitive OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models and VideoMaMa: Mask-Guided Video Matting via Generative Prior consume downstream. Compared to SAM 3: Segment Anything with Concepts — SAM 3’s Promptable Concept Segmentation — REM and SAM 3 attack the same problem from opposite directions: SAM 3 builds a dedicated detector+tracker with 4M concept labels, while REM piggybacks on Internet-scale T2V pretraining and never adds a recognition head. Worth comparing the two on Ref-VPS-style process queries, where SAM 3 has not been benchmarked.