Video-CoE: Reinforcing Video Event Prediction via Chain of Events

TL;DR

Video-CoE post-trains a Qwen2.5-VL-3B/7B base into a SOTA video event prediction (VEP) model by enforcing a Chain of Events (CoE) reasoning paradigm: the model first segments the input video into an explicit temporal event chain (each event tagged with start/end timestamps and a fine-grained description), then reasons over both the raw video and the constructed chain to predict the next event. Training is two-stage — CoE-SFT on a small Qwen2.5-VL-72B-synthesized dataset that teaches the logical-reasoning pattern, then CoE-GRPO with a multi-component reward (format adherence, length constraint, video-clip similarity to event descriptions, and answer correctness) that unlocks fine-grained temporal modeling. On FutureBench at 7B the resulting model hits 75.00 avg accuracy vs. 67.28 for vanilla GRPO and 65.45 for Qwen2.5-VL-72B; it also reports a substantial jump in attention to visual tokens, which the paper identifies as the root cause of vanilla MLLMs’ VEP weakness.

Key claims

VEP is significantly harder for current MLLMs than other video tasks; the leading commercial models (GPT-4o, GPT-5) only reach ~57-59% on FutureBench, and even Qwen3-VL-30B-A3B (the strongest base) tops out at 66.86% [§3, Table 1].
The two named failure causes are (a) lack of logical reasoning ability for future events — models rely on textual answer-option cues rather than grounding in video content — and (b) insufficient utilization of visual information, confirmed by an attention-distribution analysis over input option tokens showing visual-token attention much lower than textual [§1, Fig. 1(a-b)].
The Chain of Events paradigm reformulates VEP as P(y | V, q) → P(C | V, q) · P(y | C, V, q) — first construct event chain C from video V and question q, then reason over both to predict event y [§4.1, Eq. 2].
CoE-SFT data is built by Qwen2.5-VL-72B from {video, question, correct future event} with explicit instruction to output the logical reasoning process without analyzing other options, followed by manual quality check at >85% pass rate; event chain C is intentionally not constructed during SFT because the teacher’s chain quality didn’t meet bar [§4.2].
CoE-GRPO introduces explicit <event>start_time end_time description</event> tags marked in the model output and a dense CoE reward R_CoE = 1[tags well-formed] − w · |#events − L_ideal|/L_ideal + s summing format, length, and a cross-modal similarity reward s computed by cropping the video to each predicted event interval and averaging VideoCLIP-XL similarity between clip and description [§4.3, Eq. 3-5].
CoE-GRPO-7B reaches 75.00 avg on FutureBench (vs. 67.28 vanilla GRPO, 64.39 vanilla SFT, 52.94 base) — the headline 1-Hop number is 80.9 [Table 1].
On AVEP, CoE-GRPO-7B reports Verb F1 65.16 / Action F1 8.29 vs. 50.01 / 2.62 for Qwen2.5-VL-7B and 62.45 / 6.48 for vanilla GRPO — a clean win over both base and naive RL [Table 2].
Judge-model win-rate (Qwen2.5-VL-72B as judge over open-set predictions) ranks CoE-SFT 38.13%, CoE-GRPO 32.42%, vanilla Instruct 16.21%, vanilla SFT 7.88%, vanilla GRPO 5.37% — the gap between CoE-* and the vanilla recipes is larger here than on closed-option benchmarks, evidence that the gain transfers to open-set prediction [Table 4].
Naive visual-attention boosters (prompt-guided “describe the video in detail”, constant-bias on visual token attention) hurt on FutureBench (45.74, 52.57 vs 52.94 base), whereas CoE reaches 75.00 — the paper’s argument that the problem isn’t fixable at inference time [§5.3, Table 5].
SFT-only is necessary but not sufficient: CoE-SFT-7B reaches 65.72 avg, GRPO on top adds +9.28 to 75.00; the SFT stage establishes the logical-reasoning template, GRPO on top installs the event-chain construction and temporal grounding [Table 1].
Group size ablation shows monotonic improvement with rollout count up to G=16 (77.20) at the price of training cost; the authors recommend G=4 as the practical point [Table 5].
Event chain length is non-monotonic: too short or too long hurts; the paper reports an optimum near L_ideal = 5 events for the FutureBench videos in their setup [Table 5].
The cross-modal similarity reward s is required: ablating it drops avg from 74.61 to 72.00 (and 1-Hop from 77.5 to 73.4), confirming it’s not just a regularizer — the model would otherwise reward-hack by emitting plausible-looking but unanchored timestamps [Table 5].

Method

The paper formalizes a CoT reasoning process for VEP as P(y | V, q) = P(R | V, q) · P(y | R, V, q), where R is the reasoning trace. Vanilla MLLMs collapse R into a description-then-option-analysis pattern that lets the textual answer choices do the work; CoE replaces R with a structured pair (C, T) where C = [(t_s, t_e, d)_i] is a temporally-ordered event chain tagged with start/end times and a description, and T is the subsequent logical reasoning from C to the predicted future event.

Stage 1 (CoE-SFT) trains the model on a small dataset where Qwen2.5-VL-72B has produced the logical-reasoning text T connecting visible video evidence to the correct future event (the event chain C is left out because teacher-produced chains were too noisy to help). The point of SFT is not to cold-start C construction but to make the model emit T without dropping back to option-by-option analysis — and it works: the SFT model already beats vanilla SFT on the same data.

Stage 2 (CoE-GRPO) introduces the <event> tags and trains them via GRPO with reward R = R_acc + α · R_CoE where R_CoE includes a format-validity indicator, a soft length term −w · |n − L_ideal|/L_ideal + b, and an alignment-style similarity term s = (1/n) Σ_i sim(clip(V, t_s_i, t_e_i), d_i) computed by cropping the input video to each predicted event window and scoring it against the predicted description with VideoCLIP-XL. The similarity reward is the load-bearing anti-hacking term — without it, the model can output plausibly-formatted but unanchored timestamps.

Training is on up to 16 NVIDIA H20 GPUs at ≤32 frames and ≤448² resolution; group size 4, KL coefficient 1e-2, clipping 0.2, lr 1e-6, 1000 steps.

Results

Headline numbers on FutureBench (avg) for both base sizes:

Model	Base	+Vanilla SFT [43]	+CoE-SFT (ours)	+Vanilla GRPO [43]	+CoE-GRPO (ours)
Qwen2.5-VL-3B	46.30	57.86	63.60	65.45	68.28
Qwen2.5-VL-7B	52.94	64.39	65.72	67.28	75.00

The 7B-CoE-GRPO number (75.00) exceeds Qwen2.5-VL-72B base (58.33), Qwen3-VL-30B-A3B (66.86), GPT-4o (59.04), and GPT-5 (57.92) — a 7B post-trained model beats both 30B-class open and frontier commercial baselines on this benchmark.

On AVEP (open-form action prediction over Ego4D-style verb/noun/action), CoE-GRPO-7B reports verb F1 12.24 / 18.75 (test/val), noun F1 65.16 / 64.03, action F1 8.29 / 9.88 — vs. Qwen2.5-VL-7B-GRPO baselines of 9.64 / 10.42, 62.45 / 63.49, 6.48 / 7.26. The action-F1 win is the most meaningful since it requires both verb and noun correct.

The open-set judge-model evaluation (Qwen2.5-VL-72B picks the best response across 5 candidates) gives CoE-SFT 38.13% win, CoE-GRPO 32.42%, vanilla baselines below 17%. The authors note CoE-SFT > CoE-GRPO under this judge is plausibly because the judge is biased toward SFT-style prose (the judge’s own training distribution) rather than CoE-style structured reasoning.

Ablations show: (a) prompt-guided and constant-bias visual attention boosters both degrade base accuracy; (b) GRPO group size G ∈ {2, 4, 16} climbs monotonically from 60.61 → 74.61 → 77.20; (c) event chain length L_ideal ∈ {3, 5, 7} peaks at 5; (d) similarity-model choice (VideoCLIP-XL / ViCLIP / CLIP-large) is close (74.61 / 73.01 / 74.24) — VideoCLIP-XL marginally best; (e) removing the similarity reward drops avg from 74.61 to 72.00.

Why it’s interesting

Three threads on the wiki converge here. First, this is another clean datapoint for Reasoning RL: the now-canonical SFT → GRPO two-stage pattern, with the same finding that SFT alone is insufficient (CoE-SFT 65.72 vs CoE-GRPO 75.00 on 7B), reinforcing the Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning story that the RL stage is what generalizes. The novel reward design here is the cross-modal similarity term — VideoCLIP-XL between cropped clip and predicted description — which functions as an anti-hacking grounding signal in the same role that “answer-recoverable from trace” plays in Video-Thinker’s hindsight curation, but installed at the reward step rather than the data step.

Second, this is the third filed instance of Thinking with Modalities in the autoregressive-tagged-CoT family: Video-Thinker uses <grounding>/<caption>; Video-CoE uses <event>start end desc</event>. The two are very close architecturally — both train a Qwen2.5-VL base to emit structured temporal tags inside CoT and back the format with GRPO — but Video-CoE adds (a) a similarity-anchored reward that ties description quality to the visual content of the tagged span, and (b) a length-controlled event chain rather than free-form tag emission. The pair is now a useful 2-point sample of what reward design for tagged-video-CoT looks like in practice, and the similarity-reward trick is a transferable primitive for any wiki paper teaching tagged spatio-temporal CoT.

Third, the CoE-SFT data pipeline is squarely in Synthetic Training Data territory — Qwen2.5-VL-72B as teacher, manual gate (>85% pass) as verification, no human-written reasoning traces. The teacher-quality observation is notable: the authors tried to include teacher-generated event chains but found their quality too low, and instead let GRPO learn the chain construction from scratch. This is a useful “synthetic-data ceiling” datapoint — the SFT signal had to be downscoped to logical reasoning text only, with the structural-grounding component left to RL.