DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

TL;DR

DenseWorld-1M is a 1M-image dataset of dense grounded captions — pixel-level masks paired with detailed object-level descriptions and spatial/relational scene descriptions — built by a three-stage automated labeling pipeline at ByteDance Seed. The pipeline runs open-world perception (entity masks + labels), per-object detailed captioning, and a final merge into spatial+relational dense captions; two purpose-built VLMs (DRC for detailed region captions, SCM for spatial caption merging) are trained to accelerate the labeling. The release is positioned as the first massive dataset to combine fine-grained masks, detailed per-entity descriptions, and inter-entity relations on high-resolution real-world images, addressing gaps in prior grounded-caption corpora. Reported as effective across vision-language understanding, visual grounding, and region captioning on 10+ downstream datasets.

Key claims

Existing grounded caption datasets are deficient on one of three axes — missing pixel-level locations, missing detailed per-entity descriptions, or missing inter-entity relations on high-resolution images — and DenseWorld-1M is the first to combine all three at scale [Abstract, §1].
The three-stage labeling pipeline factorizes annotation as (1) open-world perception → entity masks + labels, (2) mask-guided detailed object-caption generation, (3) merging object captions and masks into spatial-relational dense captions [Abstract, §3].
Two task-specific VLMs are trained to accelerate the pipeline: the Detailed Region Caption model (DRC) replaces stage-2 generation and the Spatial Caption Merging model (SCM) replaces stage-3 merging [§3, §4].
The dataset comprises 1M images with 23.1M object captions and 23.6M object mask annotations [Themoonlight summary; corroborated by paper §3 statistics].
Models trained on DenseWorld-1M produce visibly more detailed, dense, and grounded captions than baselines on qualitative comparisons [§5, Fig. teaser].
DRC and SCM each outperform direct prompting of larger general-purpose MLLMs on their respective sub-tasks, justifying the per-stage fine-tuned-VLM design [§5, Table comparisons].
Extensive experiments across more than 10 downstream datasets covering vision-language understanding, visual grounding, and region caption generation demonstrate DenseWorld-1M’s utility as a pretraining/fine-tuning corpus [Abstract, §5].

Method

The DenseWorld-1M pipeline is a bottom-up factorization of the dense-grounded-caption problem. Stage 1 — open-world perception — runs off-the-shelf segmentation + classification (SAM-family + open-vocabulary tagger) on each image to extract per-entity binary masks and category labels. Stage 2 — detailed object caption generation — takes each (image, mask, label) triple and produces an object-level detailed caption. Initially this is done with a large general-purpose MLLM; the authors then distill a smaller, task-specific Detailed Region Caption model (DRC) on this output to scale generation cheaply. Stage 3 — dense caption merging — composes the per-object captions, their masks, and inferred spatial relations into a single scene-level dense caption. Again, a general-purpose MLLM is used to seed the merge and a smaller Spatial Caption Merging model (SCM) is trained on the resulting traces to scale. The pipeline runs over 1M real-world high-resolution images sourced for the dataset, yielding ~23M masks and ~23M per-object captions plus dense scene-level captions. Output format is mask + label + detailed-caption per entity, plus a final paragraph-style dense caption with explicit spatial/relational phrasing tied to the masks.

Results

Headline scale: ~1M images, ~23.1M object captions, ~23.6M masks [§3 stats].
Comparison vs prior grounded-caption datasets (GranD, AS-1B, GLaMM-style annotations) on coverage of (masks, detailed text, relations, resolution) — DenseWorld-1M is the first to score on all four axes [Table 1].
DRC outperforms the larger general-purpose MLLM used to bootstrap it on per-region caption quality (BLEU/CIDEr/METEOR-style metrics) at substantially lower inference cost [§5].
SCM similarly outperforms its bootstrap teacher on the merge task, measured against human-rated dense-caption coherence and grounding metrics [§5].
Downstream: fine-tuning grounded MLLMs on DenseWorld-1M improves performance on visual grounding, region caption generation, and vision-language understanding benchmarks; “over 10 different datasets” is the headline coverage figure [Abstract].

Why it’s interesting

DenseWorld-1M comes from a near-identical author team (Xiangtai Li, Tao Zhang, Yanwei Li, Lu Qi, Jiashi Feng, ByteDance Seed) as Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos — and the two papers slot together as upstream and downstream. Sa2VA needs grounded-caption supervision (GCG, Ref-VOS) to train its [SEG]-bridged MLLM-plus-SAM-2 architecture; DenseWorld-1M is the dataset-side answer to “where does that supervision come from at 1M-image scale.” Both papers also share the bet that perception models (SAM-family) + MLLMs can be composed to scale annotation, and that task-specific distilled VLMs (DRC/SCM here; the [SEG] head in Sa2VA) beat just prompting a frontier model at every step.

The paper is also a clean instance of the Synthetic Training Data pattern this wiki has been tracking: a multi-stage automated pipeline whose verification gate is structural (mask-guided stage handoff) rather than a separate reward model — closest in shape to Action100M: A Large-scale Video Action Dataset (V-JEPA 2 + Llama-3.2-Vision + Perception-LM + Self-Refine over HowTo100M), but for images and with relational/spatial captions as the final product rather than action labels. Worth comparing to ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o as another GPT-style annotation-distillation effort — DenseWorld-1M is the explicitly-grounded counterpart.