Skip to content
Bud's Wiki

Emergent Representation Geometry

TL;DR

Section titled “TL;DR”

Neural networks trained on local objectives (next-token prediction, masked reconstruction, contrastive prediction) spontaneously synthesize globally-organized embedding geometry that was not explicitly supervised — multi-hop graph distances in sequence models, modality-aligned kernels between text-only LLMs and frozen vision/audio encoders, and shared dictionary atoms across independently-trained vision backbones. The filed work argues this phenomenon is concrete enough to be a lever: it can be elicited at inference time via prompts (Words That Make Language Models Perceive) or amplified at training time by sharing weights across unpaired modalities (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models), and it has a partial mechanistic story rooted in the low-rank spectral bias of cross-entropy loss (Deep sequence models tend to memorize geometrically; it is unclear why). This concept is the wiki’s home for the Platonic Representation Hypothesis (Huh et al. 2024) and its descendants.

Key claims

Section titled “Key claims”
  • Decoder-only Transformers and Mamba memorizing the edges of a 10⁴-node, 10-hop path-star graph synthesize an embedding geometry that aligns leaves and first-hops of the same path across the entire graph, despite no example ever co-occurring both endpoints (Deep sequence models tend to memorize geometrically; it is unclear why §2.3, Fig. 6).
  • This is not an architectural necessity: freezing token embeddings under the same architecture and optimizer recovers a purely associative solution — the geometric bias is a learning bias, not a capacity story (Deep sequence models tend to memorize geometrically; it is unclear why Observation 3.1, §C.3).
  • The bias has a partial mechanistic explanation in the low-rank spectral bias of cross-entropy loss: top eigenvectors of the graph adjacency matrix dominate the learned embedding, and powers of AA — which encode multi-hop distance — are revealed without extra pressure (no bottleneck, no early stopping) (Deep sequence models tend to memorize geometrically; it is unclear why §5).
  • The same convergence shows up across models: a single shared TopK SAE dictionary trained on DinoV2, SigLIP, and supervised ViT activations on ImageNet contains atoms that fire universally across all three, and concept universality correlates positively with atom importance — i.e. the most-important concepts are the most-shared (Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment §3.1, §4.3, Fig. 6).
  • Cross-model convergence is also queryable at inference time from a unimodal model: a single-word sensory cue (“see”/“hear”) in the prompt and a 128-token continuation shifts a frozen Qwen3-32B’s mutual-kNN kernel toward DINOv2 (for SEE) or BEATs (for HEAR), with the mirror direction reducing alignment (Words That Make Language Models Perceive §3.2, Fig. 3).
  • The effect is a property of the autoregressive generation, not of the prompt as a static input: inserting the same sensory cue into a single-pass embedding readout hurts alignment instead of improving it (Words That Make Language Models Perceive §3.2, Fig. 3).
  • The effect scales with model size and with generation length up to a semantic-drift ceiling; alignment along the visual-vs-auditory projection axis grows monotonically from Qwen3-0.6B (no separation) to Qwen3-32B (cleanly bimodal) (Words That Make Language Models Perceive §3.4 Fig. 6, §3.5 Fig. 8).
  • The same shared geometry can be trained into a unimodal target: a single network with shared weights, fed unpaired image and text features, learns vision representations that beat DINOv2-S/14 unimodal baselines by 3–6 points on few-shot linear-probing across 9 benchmarks — gains accumulate when audio is added as a third modality (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §4.2, Table 2, Fig. 6).
  • The information-theoretic justification holds under linear assumptions: each modality is a noisy projection of the shared latent, so Fisher information on the shared component adds across modalities pointwise, with strict variance reduction in any direction the auxiliary modality covers (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §3.1, Theorem 1, Theorem 2).
  • Per-sample value of an auxiliary modality can exceed that of an in-modality sample: if the auxiliary covers a direction the primary doesn’t, additional same-modality data buys zero Fisher information there, while a single auxiliary sample buys strictly positive information (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §3.1, Theorem 3).
  • The cross-modal exchange rate is quantifiable: on Oxford-Pets, 1 image is worth ≈ 228 words under aligned CLIP encoders and ≈ 1034 words under unaligned DINOv2 + OpenLLaMA, with diminishing returns past a small text budget (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §4.4, Fig. 8, Fig. 9).
  • Multimodal neurons — units that activate coherently for the same concept across modalities — emerge in the shared layers of an unpaired multimodal network, replicating Goh et al. 2021’s CLIP-neuron finding without paired supervision (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §4.5).
  • BERT weights transfer as a useful initialization for ViT trunks on ImageNet (with newly-learned patch + positional embeddings), supporting the claim that the shared structure exists in weights, not just in activations (Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models §4.3, Fig. 7).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-08] Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds: First matched-prior tool for the “concepts are multidimensional manifolds” claim this concept has been collecting evidence for. Block-Sparse Featurizers formalize the additive-mixture-of-manifolds data-generating process into a MAP-derived block-sparsity penalty, then decompose DINOv3 / InceptionV1 / SDXL activations into subspaces with per-concept stable rank 2–4 (independent of allotted block capacity up to 128). The InceptionV1 result is the sharpest: all previously-identified curve-detector neurons read off a single continuous curve manifold with Fourier-harmonic internal structure — direct evidence that SAE-style 1D atoms fragment structures that are natively multidimensional. Complements Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment (cross-model 1D atoms) on the orthogonal axis of relaxing the SAE prior.
  • [2026-05-24] Words That Make Language Models Perceive: Inference-time elicitation of cross-modal alignment in text-only LLMs. Mutual-kNN kernel of Qwen3 generations matches DINOv2 / BEATs more closely under matched sensory cues, scales monotonically with model size and generation length, requires scene-appropriate (not random) sensory language, and pays off on caption-only MME VQA (+2.4 points). Reframes the Platonic Representation Hypothesis as actively elicitable rather than passively observed.
  • [2026-05-24] Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models: Training-time amplification of the same shared geometry. A single shared network fed unpaired image + text (+ audio) data — no contrastive loss, no inferred alignment — beats unimodal baselines on 9 vision benchmarks and on audio classification. Quantifies the image–text exchange rate and demonstrates pretrained-BERT-as-ViT-init transfer. Companion paper to Words That Make Language Models Perceive from the same lab.
  • [2026-05-22] Deep sequence models tend to memorize geometrically; it is unclear why: The original wiki-anchor datapoint — Transformers / Mamba spontaneously encode multi-hop graph distances in their token embeddings while solving an in-weights path-finding task. Establishes that the geometric bias is a learning bias (not architectural) and traces it to the low-rank spectral bias of CE loss via a Node2Vec reduction.

Open questions

Section titled “Open questions”

Papers

Section titled “Papers”

2025-12

Section titled “2025-12”

2025-10

Section titled “2025-10”

2025-02

Section titled “2025-02”