Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

TL;DR

Video-STaR is the first video self-training recipe for LVLMs. Given any labeled video dataset — action labels, bounding boxes, action-quality scores — the LVLM cycles between answer generation, label-rationalization, and finetuning on filtered outputs that contain the gold label. Using Video-LLaVA as the base, it boosts TempCompass by 10% and raises Kinetics700-QA accuracy by ~20% over the base model. Same family of ideas as Zelikman et al.’s STaR, lifted to video LVLMs with a Parser-Verifier label-check as the gate. Published at ICLR 2025.

Key claims

Naively converting labeled video datasets into question–answer pairs (the “Video-LLaVA+” baseline) does not improve LVLM performance and can degrade it; the self-training loop is what makes the supervision usable [§4.3, Tab. 5].
Label Rationalization (give the model the gold label and ask it to justify) is essential on hard datasets where pure Answer Generation yields ~0 samples. On FineDiving, removing rationalization collapses accuracy [§4.4, Tab. 6].
The Parser-Verifier gate is the load-bearing component: answers are kept only if their extracted label (via Regex + NER) matches the gold within a 5% margin [§2.3].
On TempCompass, Video-STaR improves the Video-LLaVA base by +4.6 points average (45.7 → 50.3) and outperforms the data-matched Video-LLaVA+ baseline (47.2) — i.e. the gain is not just from training on more data [§4.2, Tab. 3].
On Kinetics700-QA, Video-STaR lifts accuracy by ~20% over the Video-LLaVA base, while the Video-LLaVA+ data-matched baseline does not improve and sometimes worsens [§4.3, Tab. 5].
On FineDiving action-quality assessment, score-prediction accuracy rises markedly under Video-STaR, and the model additionally produces a free-text rationale per dive [§4.3].
Distillation from a stronger model (Gemini Pro Vision labelling on a subset, the “Vid-LLaVA-Gemini” baseline) underperforms Video-STaR’s self-training on TempCompass (38.2 vs 50.3) — self-generation with a verifier beats teacher-distillation here [§4.2, Tab. 3].
VSTaR-1M, the resulting instruction-tuning dataset, is constructed from three source datasets (Kinetics700, STAR-benchmark, FineDiving) with a deliberately diverse label-type mix (action class, bounding box, temporal localization, action-quality score) [§3.1].

Method

A pre-trained instruction-tuned LVLM (Video-LLaVA, Vicuna-7B v1.5 backbone) is improved over generation–training cycles. In each cycle, an LLM is first prompted with a task description and the video labels to write candidate questions. Then for every (video, question) pair the LVLM is queried twice: (1) Answer Generation — direct answer + rationale; (2) Label Rationalization — for questions the model failed in step (1), the gold label is shown and the model is asked to justify it. Both outputs are filtered by a Parser-Verifier: the Parser extracts predicted labels from the generated text using a mix of Regex (for bounding boxes, time ranges) and NER (for entity-like labels); the Verifier compares them to the gold with task-appropriate metrics (IoU for boxes/temporal localization, BERT embedding similarity for sentence ordering, etc.), keeping only outputs within a 5% margin. The pre-trained model (not the previous cycle’s checkpoint) is then finetuned on the filtered dataset plus VideoInstruct-100K and LLaVA-150K, producing the next cycle’s model. Three cycles were run; only Answer-Generation outputs are used in producing the final reported model.

Results

TempCompass (fine-grained temporal QA): Video-STaR 50.3 average vs Video-LLaVA 45.7 vs Video-LLaVA+ 47.2 vs Vid-LLaVA-Gemini 38.2 vs Gemini-1.5-Pro 66.0 [Tab. 3].
Kinetics700-QA: ~20% absolute accuracy gain over Video-LLaVA; Video-LLaVA+ does not improve [§4.3, Tab. 5].
FineDiving-QA (action quality): substantial score-prediction accuracy improvement vs base; rationales accompany the predictions [§4.3].
Zero-shot VQA (MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA): consistent wins over Video-LLaVA but smaller margins; the paper argues these benchmarks are near their noise ceiling because Gemini Pro Vision scores comparably to open-source models on the same subset [§4.2, Tab. 4].
Ablations: removing Answer Generation hurts uniformly across all datasets; removing Label Rationalization hurts mainly on FineDiving (collapse) and slightly on MSVD-QA [§4.4, Tab. 6–7].

Why it’s interesting

This is the direct ancestor of the recent wave of synthetic-data-with-a-gate recipes the wiki already tracks. The same Zelikman-2022 STaR loop and the same “verifier as the load-bearing piece” thesis underwrites Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (which generalizes the rationalization step from label-presence to “Qwen2.5-VL-7B can solve from the trace” hindsight curation), Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (multi-agent tool-use synthesis with gated rewards), and Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (which showed that at compute parity, sampling more from a weaker model and filtering beats sampling fewer from a stronger one). Video-STaR is the cleanest video instance: the verifier is concrete (label-string match within a 5% margin), the baseline ablations are honest (Video-LLaVA+ shows data-matched naive supervision doesn’t work), and the Gemini-distillation comparison shows self-generated-and-gated can beat teacher-pseudo-labelling at fixed data scale.