Skip to content
Bud's Wiki

ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o

TL;DR

Section titled “TL;DR”

ShareGPT-4o is a project page from the OpenGVLab / InternVL team announcing a planned open-source release of GPT-4o-generated multimodal annotations: 200K images, 10K videos, and 10K audio files, all paired with detailed captions. The pitch is that frontier-model captions surpass existing image-text / video-text / audio-text corpora in diversity and informational richness (world knowledge, object properties, spatial relations, aesthetics), and the dataset is positioned as a modality-alignment resource for open LMMs. The page itself is a stub — only the abstract and BibTeX are exposed; the dataset itself is the artifact and lives in adjacent OpenGVLab repos. It is the multimodal extension of the earlier ShareGPT4V (1.2M GPT4-Vision captions) recipe.

Key claims

Section titled “Key claims”
  • Dataset scope: 200K images + 10K videos + 10K audio files, each annotated by GPT-4o with detailed captions [project page abstract].
  • Captions cover world knowledge, object properties, spatial relationships, and aesthetic evaluations — explicitly framed as richer than prior image-text corpora [project page abstract].
  • Construction recipe is “customize prompts and utilize GPT-4o’s powerful multimodal capabilities to annotate” curated source media [project page abstract].
  • Positioned as the audio+video extension of ShareGPT4V — the predecessor reported MME/MMBench gains of 222.8/22.0/22.3 and 2.7/1.3/1.5 on LLaVA-7B / LLaVA-1.5-13B / Qwen-VL-Chat-7B by swapping equivalent SFT data for ShareGPT4V captions [arXiv 2311.12793 abstract].
  • Release announcement timing: 2024/05/30 via the OpenGVLab/InternVL repo [InternVL README changelog].

Method

Section titled “Method”

The project page is intentionally minimal — it describes only the high-level construction pipeline: collect source images / videos / audio, design per-modality prompts, call GPT-4o, retain the resulting captions as the modality-alignment supervision. No filtering, scoring, or verification gate is documented on the page itself. The underlying recipe is the same one used in ShareGPT4V (Chen et al., arXiv 2311.12793): start with a small curated seed of frontier-model captions and rely on the captioner’s coverage rather than human post-editing. The ShareGPT-4o-Image follow-up paper (arXiv 2506.18095) extends the same recipe to image generation outputs (91K samples) and reports that Janus-Pro fine-tuned on it (Janus-4o) gains text-and-image-to-image generation capability from scratch in 6 hours on 8×A800.

Results

Section titled “Results”

The project page itself reports no numbers — it is a release announcement. Headline numbers come from the wider lineage:

  • ShareGPT4V (predecessor): +222.8 / +22.0 / +22.3 MME and +2.7 / +1.3 / +1.5 MMBench when substituting equivalent caption volume into LLaVA-7B / LLaVA-1.5-13B / Qwen-VL-Chat-7B SFT data [arXiv 2311.12793 abstract].
  • ShareGPT-4o-Image (successor, June 2025): 91K samples (45K T2I + 46K TI2I) suffice to transfer GPT-4o-level image generation to Janus-Pro in 6h on 8×A800 [arXiv 2506.18095 §1].
  • Dataset sizes vs. prior generation: 200K / 10K / 10K images / videos / audio, vs ShareGPT4V’s 1.2M images and ShareGPT4Video’s 40K (GPT4V) + 4.8M (ShareCaptioner-Video) captions [InternLM-XComposer ShareGPT4V README].

Why it’s interesting

Section titled “Why it’s interesting”

ShareGPT-4o is the audio-and-video extension of the GPT-4o-as-captioner playbook the wiki already tracks for text-and-image. It complements Synthetic Training Data by adding a multimodal instance to a concept page whose current evidence is mostly text + image + speech: the same “use a frontier multimodal model to generate captions, swap into SFT” recipe that drove the ShareGPT4V results extends to video and audio with smaller volumes (10K each) but the same alignment-supervision theory. It also contrasts with Action100M: A Large-scale Video Action Dataset which uses a hierarchical open-weight pipeline (V-JEPA 2 + Llama-3.2-Vision + Perception-LM + GPT-OSS-120B Self-Refine) to caption 1.2M videos — ShareGPT-4o instead bets on a single closed frontier model, with the cost trade-off that the dataset cannot be regenerated or audited from the captioner’s weights.

See also

Section titled “See also”