Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models

TL;DR

Uml (Unpaired Multimodal Learner) is a training recipe in which a single shared network alternately processes inputs from different modalities — say images and text — with no paired correspondence between samples, no contrastive objective, no inferred alignment. Weight sharing is the sole coupling. The paper provides a linear-Gaussian theorem stack showing this strictly increases Fisher information on the shared latent state (sometimes by more per sample than collecting additional data in the target modality), then validates the effect across 9 vision benchmarks, the MultiBench multimodal suite, and an audio extension via ImageNet-ESC. The headline practical findings: unpaired text consistently lifts vision models (especially few-shot and fine-grained), BERT weights transfer as a useful ViT initialization, the image–text exchange rate sits around 1 image ≈ 228 words for aligned CLIP encoders (and ≈ 1000 words for unaligned DINOv2 + OpenLLaMA), and multimodal neurons emerge in the shared layers without paired supervision.

Key claims

Under a linear data-generating model where each modality is a noisy linear projection of a shared latent $\theta$ , adding unpaired auxiliary-modality samples strictly reduces estimator variance on the shared component — i.e. Fisher information increases pointwise [§3.1, Theorem 1].
Even when the auxiliary modality only informs some directions in the latent space, the joint estimator strictly outperforms the unimodal one along those directions [§3.1, Theorem 2].
A fixed budget of additional samples is sometimes better spent in the auxiliary modality than the target modality — i.e. the per-sample value can be larger than one [§3.1, Theorem 3].
Self-supervised Uml on MultiBench (MUStARD, MIMIC, MOSEI, MOSI, UR-FUNNY) and Oxford Pets / UCF101 / DTD beats unimodal baselines on every benchmark, with the largest gain on MUStARD (sarcasm: 59.66 → 63.28) [§4.1, Table 1].
Supervised Uml with frozen DINOv2 (ViT-S/14) + frozen OpenLLaMA-3B, with the classifier head initialized from per-class average text embeddings, lifts 9-benchmark few-shot linear-probing accuracy by ~3–6 absolute points at 1, 2, 4-shot; the average 1-shot accuracy goes from 45.52 to 51.36 [§4.2, Table 2].
Robustness to distribution shift improves under Uml: 16-shot ImageNet → ImageNet-V2 / Sketch / A / R all gain over unimodal [§4.2, Fig. 5].
Cross-modal weight transfer works without co-training: initializing a ViT’s transformer blocks from a pretrained BERT (with newly-learned patch + positional embeddings) beats from-scratch ViT on ImageNet for both frozen and unfrozen backbones [§4.3, Fig. 7].
The image–text exchange rate is quantifiable: on Oxford-Pets, 1 image ≈ 228 words for CLIP-aligned encoders and ≈ 1034 words for non-aligned DINOv2 + OpenLLaMA; gains saturate after a small number of text samples [§4.4, Fig. 8, Fig. 9].
Multimodal neurons — units in the shared layers that activate coherently for the same concept across images and unpaired text — emerge under Uml, replicating the Goh et al. 2021 CLIP finding without paired supervision [§4.5].
Audio classification on ImageNet-ESC-27 / 19 improves when image and/or text are added as unpaired auxiliary modalities, and benefits compound when all three are combined; CLIP-aligned image/text encoders produce the strongest gains [§4.2 audio paragraph, Fig. 6, §E.10, §E.13.3].

Method

Setup: two unpaired datasets, $\{x_i\}_{i=1}^{n_x}$ and $\{y_j\}_{j=1}^{n_y}$ , drawn independently from the marginals $p(x)$ and $p(y)$ . There is no assumed correspondence between any specific $x_i$ and $y_j$ . Each modality is encoded by its own (possibly frozen, possibly pretrained) encoder $\phi_x$ , $\phi_y$ into embeddings of the same dimension, which are then both fed through a shared network $f$ . In the self-supervised regime, each modality has its own decoder ( $g_x$ , $g_y$ ) and the loss is the sum of per-modality reconstruction / next-token MSE or CE losses through the shared trunk. In the supervised regime, a single classifier head sits on top of $f$ and receives gradients from both modalities; the head is initialized from average per-class text embeddings to give a usable image–class prior. At inference, the auxiliary pathway is discarded and only $f \circ \phi_x$ is used as the representation for the target modality; a fresh linear probe is then trained on downstream tasks.

The theoretical core treats each modality as $y_i = A_i \theta + \epsilon_i$ with $\theta = (\theta_c, \theta_x, \theta_y)$ split into shared and modality-specific components. The shared-parameter Fisher information matrix is the sum of per-modality blocks, so adding any modality with non-degenerate curvature along a direction strictly tightens the estimator variance bound in that direction — even with no pairing. The extension to three modalities is additive (sum of all auxiliary blocks).

Results

MultiBench self-supervised: average gain of 1.4–3.6 points across the 5 sarcasm / health / sentiment tasks vs unimodal pretraining [Table 1].
Standard vision few-shot (DINOv2-S/14 + OpenLLaMA-3B): full-finetune Stanford Cars 79.45 → 86.39, FGVC Aircraft 66.99 → 73.44, Caltech101 95.45 → 97.60; 1-shot Oxford Pets 63.51 → 73.59 [Table 2].
ImageNet → robustness probe (16-shot DINOv2 linear): Uml consistently outperforms unimodal on V2 / Sketch / A / R [Fig. 5].
BERT-init ViT on ImageNet: clear gains over from-scratch ViT, both frozen and unfrozen [Fig. 7].
ImageNet-ESC-27 audio accuracy improves under +V, +T, and +V+T; the all-three setting is strongest [Fig. 6, §E.10].
Random / semantically-unrelated auxiliary text fails to produce gains — confirming the effect is from semantic correlation, not regularization [§E.9].
Caption complexity matters: richer / more diverse captions lift few-shot results more than short labels [§E.7].

Why it’s interesting

This is the training-time analogue of the inference-time elicitation in Words That Make Language Models Perceive: both papers operationalize the Platonic Representation Hypothesis as a practical lever rather than a passive observation. For a generative-models team the most direct read is that unpaired auxiliary modalities can act as a strong regularizer on a target-modality model — DINO + a small slice of conceptually-related text already meaningfully sharpens few-shot vision classification and improves OOD robustness, without ever needing aligned (image, caption) pairs. The exchange-rate result (1 image ≈ 228 words under aligned encoders, ≈ 1034 under unaligned ones) is the closest thing in the wiki to a quantitative version of AGI Is Not Multimodal‘s argument that the gap between unimodal and multimodal models is smaller than the field usually assumes — and it does so without taking either side. It also nicely complements Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment, which provided dictionary-level evidence of cross-model representational convergence in vision encoders; the multimodal neurons that emerge in Uml’s shared trunk without paired supervision are the same phenomenon viewed from the activation side.