Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

TL;DR

Sa2VA bolts SAM-2 onto a frozen LLaVA-like MLLM (InternVL2 family, 1B/4B/8B/26B) and trains end-to-end on a co-training mix of image QA, video QA, image referring segmentation, video referring segmentation, and grounded caption generation. The bridge is a single special [SEG] token: the LLM emits it as part of its text output, and SAM-2’s decoder consumes its hidden state as a spatial-temporal prompt to produce masks (or tracked masklets). The MLLM is LoRA-finetuned, SAM-2’s image/memory encoder is frozen, only the decoder is tuned — yielding one model that does referring image/video segmentation, image/video chat, GCG, and visual-prompt understanding in a single instruction-tuning pass. The paper also releases Ref-SAV, a 37K-video / 72K-expression auto-annotated Ref-VOS dataset and a 1.1K-video / 1.9K-expression human-validated benchmark. Reported SOTA on RefCOCO/+/g, MeViS, Ref-DAVIS17, ReVOS, with image/video chat scores close to the underlying InternVL2.

Key claims

A single [SEG] token shared between LLaVA and SAM-2’s decoder is sufficient to unify referring image/video segmentation, image/video VQA, grounded caption generation, and visual-prompt understanding under one instruction-tuning objective [§3.1, §3.2].
The MLLM and SAM-2 are deliberately decoupled: SAM-2 features are not fed back into the LLM and only the [SEG] hidden state flows from LLM into SAM-2, keeping the MLLM as a plug-in-replaceable component [§3.2 “Decoupled Design”].
Sa2VA-8B reaches 81.6 / 76.2 / 78.7 cIoU on RefCOCO / RefCOCO+ / RefCOCOg, beating GLaMM-7B by +2.1 / +3.6 / +4.5 [§4.1, Table 2].
Sa2VA-8B reaches 57.0 / 75.2 / 57.6 J&F on MeViS / Ref-DAVIS17 / ReVOS, beating prior SOTA VISA-13B by +12.5 / +4.8 / +6.7 J&F [§4.1, Table 2].
Co-training on segmentation + chat preserves chat quality where prior grounded-MLLMs collapse: Sa2VA-8B scores 1617/511 on MME, 81.6 on MMBench, 75.1 on SEED-Bench — within range of InternVL2-8B (the chat base), while LISA/GLaMM/PixelLM score effectively zero on the same metrics [§4.1, Table 2].
Mask tracking re-uses SAM-2’s memory module without modification: the LLM emits one [SEG] per object on the key frame; the resulting mask is propagated to remaining frames via SAM-2’s memory encoder, so no new tracker is trained [§3.2 “Utilizing SAM-2 knowledge for mask tracking”].
Ref-SAV is built by a three-stage auto-annotation pipeline over SA-V (object/part → scene → video-level descriptions using InternVL2-76B + Qwen2-72B for consistency), yielding 37,311 videos and 72,509 long-text object expressions [§3.3].
On the Ref-SAV benchmark, zero-shot Sa2VA-8B reaches 41.3 overall J&F vs ~10–12 for prior Ref-VOS SOTAs (UniRef++, UNINEXT, MeVIS, VISA) — fine-tuning on Ref-SAV training lifts it to 50.0 [§4.1, Table 5].
Sa2VA-26B improves further over 8B on every benchmark, indicating the recipe scales with the chosen MLLM backbone [§4.1, Table 2].
Training: 4e-5 LR, MLLM LoRA-tuned, perception encoder frozen, 8× A800 80GB for 48 hours, 8,192 max sequence length [§4 “Implementation Details”].
Total training mix is ~1.1M image-text/video-text pairs: 665K LLaVA-1.5 + 100K ChatUniVi + 56K refer-expression + 214K GCG + 5.8K Ref-YTVOS/MeViS/ReVOS + 37K Ref-SAV [§4, Table 3].

Method

Sa2VA’s architecture is two stacks running in parallel with a single token-level bridge. Inputs go through the LLaVA-like MLLM (the paper uses InternVL2 variants at 1B / 4B / 8B / 26B): visual encoder → projection layer → LLM, with input text and visual tokens jointly attended. The LLM’s vocabulary is extended with a [SEG] token. For chat-only inputs the model emits regular text; for referring segmentation, the LLM emits text containing one [SEG] per requested object, and the hidden state of each [SEG] is read out and forwarded to SAM-2’s mask decoder as a spatial-temporal prompt. SAM-2’s image and memory encoders are kept frozen; only the mask decoder is tuned. Crucially, no features flow from SAM-2 back into the LLM — Sa2VA bets that the MLLM can produce all the spatial-temporal information SAM-2 needs through the single [SEG] hidden state. For video referring segmentation, the LLM emits one [SEG] per object using the key frame; the mask decoder produces the key-frame mask; SAM-2’s memory module propagates the mask to remaining frames as in regular SAM-2 inference. Training is one-shot supervised instruction tuning (no separate pre-training stage) with three losses: text regression on the LLM head, pixel-wise CE on the mask, and dice loss on the mask, weighted equally.

Results

Image referring segmentation (cIoU): Sa2VA-8B / 26B = 81.6 / 82.5 on RefCOCO, 76.2 / 78.8 on RefCOCO+, 78.7 / 79.7 on RefCOCOg [Table 2]. Prior SOTA grounded MLLMs (GLaMM-7B / GSVA-13B): 79.5 / 79.2 on RefCOCO at best.
Video referring segmentation (J&F): Sa2VA-8B = 57.0 / 75.2 / 57.6 on MeViS / Ref-DAVIS17 / ReVOS, vs VISA-13B 44.5 / 70.4 / 50.9 [Table 2].
Chat preservation: Sa2VA-8B MME = 1617/511, MMBench = 81.6, SEED-Bench = 75.1, Video-MME = 52.1, MMBench-Video = 1.34, GCG = 31.0 — within ~1–2 points of the underlying InternVL2-8B chat metrics, where competing grounded MLLMs sit at zero or near zero on chat benchmarks [Table 2].
Ref-SAV benchmark (J&F overall): Sa2VA-8B zero-shot = 41.3; fine-tuned on Ref-SAV train = 50.0; prior best (VISA zero-shot) = 11.8 [Table 5]. The long-expression subset is the hardest (Sa2VA-8B 49.3 zero-shot / 58.7 fine-tuned).
Specific-tuned models (Table 6): matching co-training, single-task fine-tuning gives ~1.5 cIoU lift on the three RefCOCO* datasets, confirming co-training is not far from the per-task ceiling.
Scale ablation (Tab. 2 trend): 1B → 4B → 8B → 26B monotonically improves every benchmark; e.g. RefCOCO 77.4 → 78.9 → 81.6 → 82.5.

Why it’s interesting

Sa2VA is a clean exemplar of the decoupled school of Unified Multimodal Models — the architecture that the UniG2U-Bench taxonomy (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?) calls “module-separated, stitched at inference,” where a frozen-ish perception model and a frozen-ish MLLM communicate through a thin token-level interface rather than via end-to-end gradient flow. The bridge token here is [SEG], mirroring the [CLS]/register-token pattern but used as a cross-model conditioning signal rather than an in-stream aggregator. For Luma’s segmentation + captioning pipelines, Sa2VA is plausibly the upstream that SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning (Meta’s production SAM 3.1) does not replace: SAM 3.1 is faster multi-object tracking with text-prompt detection, but Sa2VA additionally produces chat + grounded captions in the same forward pass, which is what makes it a candidate model for joint segmentation + captioning. It also contrasts with SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation — both retrofit SAM via LoRA, but SPDA-SAM keeps everything inside the perception world (depth-aware instance segmentation) while Sa2VA reaches across to language. The Ref-SAV pipeline (auto-annotate → human-validate, with InternVL2-76B + Qwen2-72B doing the labeling) is also of interest as a Synthetic Training Data recipe for video — a 37K-video dataset built with a stack of off-the-shelf VLMs.