Action100M: A Large-scale Video Action Dataset

TL;DR

Action100M is a Meta FAIR / HKUST open-vocabulary video-action dataset built from 1.2M HowTo100M instructional videos (≈14.6 years), with 147M temporally localized segments annotated by a fully automated VLM+LLM pipeline (V-JEPA 2 for segmentation, Llama-3.2-Vision and Perception-LM-3B for frame and segment captioning, GPT-OSS-120B with Self-Refine for structured aggregation). Each segment carries five fields — brief/detailed action, actor, brief/detailed caption — totaling 21.3B English words. Training VL-JEPA on Action100M produces a 1.6B ViT-L (256px, 32 frames) that beats CLIP / SigLIP2 / PE-Core on zero-shot action recognition and text-to-video retrieval averages while seeing far fewer samples (3.3B vs 13B–86B). Released as the data foundation behind VL-JEPA, framed by the authors as a primitive for VLMs, world models, and robotics policies.

Key claims

The dataset contains 147,092,653 annotated segments over 1.2M HowTo100M videos (~14.6 years of duration), totaling 21.27B words across four annotation fields [§4, Tab. 1].
Annotation pipeline is fully automated and hierarchical: V-JEPA 2 ViT-g-384 embeddings → Ward-linkage agglomerative clustering with local-temporal connectivity → mid-frame captions (Llama-3.2-Vision-11B) + segment captions (Perception-LM-3B) organized as a Tree-of-Captions → GPT-OSS-120B Self-Refine (3 rounds) extracts brief/detailed action, actor, brief/detailed caption [§3, Fig. 2].
Compute budget for annotation is ~1.3M V100 GPU-hours (segmentation + captioning) + ~0.3M H100/H200 GPU-hours (LLM aggregation) [§1].
Trained VL-JEPA (ViT-L, 256px, 32 frames, 1.6B params, 3.3B samples seen) reports 52.5 average top-1 on 8 action-recognition benchmarks vs 44.7 for PE-Core ViT-G (2.3B params, 86B samples) and 39.8 for SigLIP2 ViT-g (1.9B params, 40B samples) [Tab. 3].
VL-JEPA also wins on 8-benchmark text-to-video retrieval R@1 (40.0 avg) vs PE-Core ViT-G (38.9), with particularly large gains on motion-focused benchmarks (Something-something-v2, EPIC-KITCHENS-100, EgoExo4D) [Tab. 3].
Stage 2 (video pretraining on Action100M) lifts action-recognition average from 21.5 (Stage 1, image-only) to 48.0 — quantifying how much of the model’s action capability comes from video data specifically, not from image-text pretraining [Tab. 3].
64% of segments are 0–3s, 23.8% are 3–10s, 10.2% are 10s–1min — the hierarchy is heavily biased toward fine-grained atomic motions [§4].
Action distribution is heavily long-tailed (e.g. “speak to camera” dominates); the paper addresses this with semantic resampling via EmbeddingGemma-300M embeddings + deduplication + clustering (DINOv2-style) [§5.4].
In a controlled 20k-step head-to-head, Action100M brief-action labels outperform direct PLM-3B pseudo-labeling on most benchmarks — the hierarchical-captioning + LLM-aggregation steps add measurable signal over a single-VLM baseline [§5.3, Fig. 9].
Dataset released at github.com/facebookresearch/Action100M; total storage ≈205 GB [§1, §4].

Method

The pipeline is built around the idea that strong reasoning models can be used cheaply if the video is first converted to text. Stage 1 transforms a video into a hierarchy of temporally coherent segments: V-JEPA 2 ViT-g-384 encodes 64-frame overlapping windows (stride 8, 1-in-4 frame subsample), spatial-average pooling yields a per-frame embedding, and hierarchical agglomerative clustering with local connectivity and Ward linkage builds the segment tree (nodes < 0.5s discarded). Stage 2 captions each node: leaf nodes get a Llama-3.2-Vision-11B description of the midpoint frame, internal nodes get a Perception-LM-3B description of 32 sampled frames; both are framed as a Tree-of-Captions. Stage 3 prompts GPT-OSS-120B with the node’s caption, children’s captions in depth-first order, and global context (root captions, video title/description/ASR) to extract five structured fields via 3 rounds of Self-Refine; nodes shorter than 4s are dropped at this stage. The whole pipeline is text-based after Stage 2, which is what makes the reasoning model affordable.

For the experiments, VL-JEPA (Chen et al., 2025c, the model series this dataset was built for) is trained in three stages: image-only pretraining on DataComp-1B / YFCC-100M (Stage 1, 100k iters at batch 24,576), then video pretraining on Action100M with 8 frames at batch 12,288 (Stage 2, 60k iters), then 32 frames with the V-JEPA 2 encoder unfrozen at batch ~3K and lr 1e-5 (Stage 3, 10k iters). InfoNCE loss aligns visual and text embeddings for CLIP-style zero-shot evaluation.

Results

VL-JEPA-ViT-L (Stage 3, 1.6B params, 256px, 32 frames, 3.3B samples seen) reaches 52.5 average top-1 on 8 zero-shot action benchmarks vs 44.7 for PE-Core ViT-G (2.3B, 86B samples) and 38.6 for SigLIP2 ViT-L (882M, 40B samples) — substantially better on a per-sample-seen basis [Tab. 3]. Motion-focused gains are the largest: SSv2 19.3 vs PE-Core’s 9.0, EPIC-KITCHENS-100 21.8 vs 6.4, EgoExo4D 33.2 vs 13.6 [Tab. 3]. Text-to-video retrieval R@1 averages 40.0 across 8 benchmarks (MSR-VTT, ActivityNet, DiDeMo, MSVD, YouCook2, PVD-Bench, Dream-1k, VDC-1k), with the model winning on 6 of 8 versus PE-Core ViT-G [Tab. 3]. Scaling curves (Fig. 1, Fig. 8) show consistent, log-linear improvement with samples seen, especially on motion benchmarks; Stage 2 (video pretraining) is responsible for most of the gap over image-only Stage 1 (21.5 → 48.0 avg). In ablations, Action100M’s brief action descriptions outperform direct PLM-3B pseudo-labeling and Ego4D atomic descriptions across most non-egocentric benchmarks [§5.3, Fig. 9].

Why it’s interesting

The data-construction recipe is the contribution worth reading carefully: a Tree-of-Captions built by medium-sized VLMs, then a reasoning LLM (GPT-OSS-120B) aggregating the captions with Self-Refine, never running the reasoning model on pixels. This is the cost-effective version of the dense-captioning pipelines now standard for video pretraining (cf. PE Video, PLM-Video-Auto, Koala-36M) and the controlled comparison vs direct PLM-3B labeling (§5.3) is the cleanest evidence yet that hierarchical-text + LLM-aggregation beats single-VLM captioning at scale. For the team, the most relevant downstream use is as a pretraining corpus for video-understanding backbones and a candidate ground-truth source for action-conditioned world-model / VLA training — VL-JEPA’s gains on motion-focused benchmarks (SSv2, EPIC-KITCHENS-100, EgoExo4D) suggest dense fine-grained action labels add signal that ASR-derived or single-caption datasets like InternVid or HD-VILA-100M don’t carry. Caveat: HowTo100M is procedurally narrow (cooking, DIY, crafts) and English-only, so coverage of non-instructional motion (sports, social, ego non-procedural) is still gappy.