Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

TL;DR

Schnaus et al. (Isola + Xi Wang + TUM) treat the Platonic Representation Hypothesis as an actionable claim: if independently-trained unimodal encoders converge toward a shared geometry, then aligning them should not require paired examples. The paper proposes a Wasserstein-Procrustes method with a coarse geometric initialization that estimates a single orthogonal map between two disjoint embedding sets with zero pairs, and shows that off-the-shelf geometric alignment metrics (not seen during fitting) predict when this works. In the very-few-pair regime (1–100 pairs) the method substantially beats existing pair-based baselines while staying competitive once pair budgets grow, and the resulting orthogonal map is good enough to drive text-to-image generation without paired supervision.

Key claims

Coarse cross-modal alignment of independently-trained vision and language embeddings is achievable with no paired examples: a single orthogonal map estimated by Wasserstein-Procrustes with a coarse geometric initialization matches embedding sets across datasets, modalities, and model families [§Abstract].
Standard geometric alignment metrics computed on the two unpaired embedding sets predict when zero-pair alignment will succeed — i.e. the method’s applicability can be checked without running the alignment itself [§Abstract].
The method also benefits from pairs when they exist: in the very-few-pair regime it substantially outperforms prior pair-based alignment methods, and remains competitive as pair budgets grow [§Abstract].
The learned orthogonal map is strong enough to enable text-to-image generation without any paired text–image examples at alignment time [§Abstract].

Method

Two independently-trained unimodal encoders (vision, text, etc.) produce disjoint sets of embeddings — no (image, caption) pairs. The paper fits a single orthogonal map $R$ between the two sets using a Wasserstein-Procrustes objective (jointly estimating a transport plan and the orthogonal rotation that minimizes transported pairwise distance) initialized from a coarse geometric alignment of the two point clouds. Because the only free parameter is a single orthogonal matrix — and the geometric prior resolves the symmetry ambiguity Procrustes alone leaves open — the method is capable of unsupervised alignment where pair-based methods (CCA, linear probes, contrastive mapping) fail. Pairs, when present, are folded in as additional supervision for the same $R$ .

Results

Across multiple datasets, modalities, and encoder families, zero-pair alignment via Wasserstein-Procrustes + geometric init consistently recovers a cross-modal correspondence [§Abstract].
Pre-fit geometric alignment scores on the two unpaired embedding sets correlate with downstream alignment quality, giving a cheap predictor of applicability [§Abstract].
In the 1–few-pair regime, the method “substantially outperforms” existing methods; it remains competitive at larger pair budgets [§Abstract].
Downstream: text-to-image generation without any paired alignment supervision, driven by the zero-pair orthogonal map [§Abstract].

Why it’s interesting

This is the operationalization of the Platonic-Representation-Hypothesis thread the wiki has been tracking. Words That Make Language Models Perceive showed the shared geometry is elicitable at inference time from a single text-only LLM, and Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models (also Isola-adjacent) showed it is trainable into a shared backbone with unpaired modality streams. This paper closes the triangle: given two already-trained unimodal models, the shared geometry is recoverable post-hoc as a single orthogonal map — no retraining, no shared backbone, no paired data. The CLIP recipe (hundreds of millions of paired image–text) looks increasingly like one point on a spectrum where the limit is zero pairs. For Unified Multimodal Models this suggests the orthodox two-tower-with-contrastive-pairs training recipe is overspending on paired data, and that pair-light assembly from independently-trained unimodal encoders is a real option.