MobileCLIP2: Improving Multi-Modal Reinforced Training

TL;DR

MobileCLIP2 (TMLR Featured, August 2025) is the second iteration of Apple’s low-latency CLIP family. The contribution is not architectural — it’s the training-data recipe. The authors rebuild MobileCLIP’s multi-modal reinforced training pipeline with (a) better CLIP teacher ensembles trained on the DFN-2B dataset, (b) improved captioner-teacher CoCa models trained on DFN and fine-tuned on a diverse set of high-quality image-caption datasets, and (c) temperature tuning in contrastive distillation. The resulting reinforced dataset DFNDR-2B is released publicly. MobileCLIP2-S4 matches SigLIP-SO400M/14 zero-shot ImageNet-1k accuracy at 2× smaller size and beats DFN ViT-L/14 at 2.5× lower latency on iPhone 12 Pro Max.

Key claims

The MobileCLIP recipe — store image-text pairs alongside frozen teacher CLIP embeddings and frozen captioner-generated synthetic captions, then train students on this reinforced dataset — is improved by upgrading the teachers, not the students [Abstract, §1].
Better CLIP teacher ensembles: training the ensemble members on the curated DFN dataset (rather than LAION/CommonPool) materially improves the downstream student’s zero-shot accuracy [Abstract].
Better captioner teachers: training CoCa-style captioner models on DFN and fine-tuning them on a diverse selection of high-quality image-caption corpora (MSCOCO-38K, GBC1M, DOCCI, DCI variants, ShareGPT4V-style recaptions) produces caption-diversity gains that translate to student accuracy [Abstract, §3].
Temperature tuning matters in contrastive knowledge distillation — the paper ablates this as a non-trivial knob in reinforced training [Abstract].
Combining synthetic captions from multiple fine-tuned caption generators is additively beneficial — each captioner’s distribution covers a different slice of caption-diversity space [Abstract].
MobileCLIP2-B reports a 2.2% absolute ImageNet-1k zero-shot accuracy improvement over MobileCLIP-B at the same architecture, isolating the contribution to training-data quality [Abstract].
MobileCLIP2-S4 matches SigLIP-SO400M/14 zero-shot accuracy on ImageNet-1k at 2× fewer parameters [Abstract].
MobileCLIP2-S4 improves on DFN ViT-L/14 at 2.5× lower latency measured on iPhone 12 Pro Max [Abstract].
The DFNDR-2B reinforced dataset and the data-generation code (ml-mobileclip-dr) are released, with the pipeline supporting distributed processing of petabytes of data and billions of samples across 10,000+ GPUs [Abstract, §5].

Method

The training pipeline has three stages. (1) Train a CLIP teacher ensemble on DFN-2B, where each ensemble member is a strong ViT-L/14 or similar CLIP variant. (2) Train CoCa-family captioner models on DFN-2B as base captioners, then fine-tune each on one of the high-quality image-caption corpora (MSCOCO, GBC1M, DOCCI, DCI in short/complete/extended variants, ShareGPT4V-style human-or-GPT4-recaptioned data). Both context-77 and context-256 captioner variants are released. (3) For every image in the DFN-2B base corpus, run the ensemble of CLIP teachers to compute and store image+text embeddings, and run the ensemble of fine-tuned captioner models to generate and store multiple synthetic captions per image. The resulting reinforced dataset DFNDR-2B contains real image-text pairs + multiple frozen teacher embeddings + multiple synthetic captions per sample. Student CLIPs (the MobileCLIP2 family, ranging from S0 to L) are trained by sampling from this reinforced dataset and combining the standard contrastive image-text loss against real text with contrastive distillation against teacher embeddings (with tuned temperature) and against synthetic captions. The student architectures themselves are the same low-latency hybrid CNN-transformer backbones from MobileCLIP (and FastViTHD-like designs for the larger S4 variant).

Results

Zero-shot ImageNet-1k: +2.2% absolute on MobileCLIP2-B vs MobileCLIP-B at fixed architecture [Abstract].
MobileCLIP2-S4 matches SigLIP-SO400M/14 ImageNet-1k zero-shot accuracy at 2× fewer parameters [Abstract].
MobileCLIP2-S4 beats DFN ViT-L/14 ImageNet-1k accuracy at 2.5× lower iPhone 12 Pro Max latency [Abstract].
39 models on Hugging Face cite this paper (downstream backbones); DFNDR-2B dataset (86.4k samples public view) and ml-mobileclip-dr code released.

Why it’s interesting

MobileCLIP2 is a clean “the teachers and the captions are the model” demonstration that sits squarely in the Synthetic Training Data thesis the wiki has been building — but on the teacher-side of distillation rather than on the data-generation side. The paper isolates the contribution to training data by holding the student architecture fixed and reporting +2.2% on MobileCLIP-B; that ablation is the cleanest one on the page for “swap real teachers for better ones, gain accuracy.” Complements Hermes 3 Dataset — 1M post-training samples (NousResearch) and Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning which apply the multi-caption-generator-ensemble logic in non-CLIP settings. Worth pairing with the companion FastVLM: Efficient Vision Encoding for Vision Language Models (same Apple team, same FastViTHD-class backbones) — MobileCLIP2 is the CLIP-pretraining recipe that produces the encoder weights FastVLM then uses as its vision backbone. The wiki’s only directly comparable CLIP-recipe entry is CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema (CinemaCLIP, hybrid CLIP for film); MobileCLIP2 is the on-device counterpart.