Words That Make Language Models Perceive

TL;DR

A text-only LLM can be coaxed into representing its inputs more like a vision or audio encoder simply by prepending a single sensory cue (“imagine seeing this”, “imagine hearing this”) and reading out hidden states from its autoregressive continuation rather than from a single forward pass. The shift is measured by mutual-kNN kernel alignment between the LLM’s caption embeddings and frozen DINOv2 / BEATs encoders on paired image–text and audio–text datasets. The effect grows with model scale (Qwen3 0.6B → 32B), grows with generation length up to a semantic-drift point, requires scene-appropriate sensory language (random visual words actively hurt alignment), and yields measurable downstream gains on caption-only VQA (MME: 64.78 → 67.14 with a SEE cue). The paper reframes the Platonic Representation Hypothesis as something that can be elicited at inference time, not just observed passively with scale.

Key claims

Generative representations — hidden states averaged over autoregressive continuations rather than from a single forward pass — already align more closely with frozen sensory encoders than single-pass embeddings, even with a neutral prompt [§3.1, Fig. 2].
A single-word sensory cue (“see” vs “hear”) in the generation prompt increases mutual-kNN alignment with the matching modality encoder (DINOv2 for SEE, BEATs for HEAR) and decreases alignment with the mismatched one [§3.2, Fig. 3].
Sensory cues do not steer single-pass embeddings the same way — inserting them actually hurts alignment in that setting; the alignment gain is a property of the generation trajectory, not of the prompt as a static input [§3.2, Fig. 3].
Alignment grows monotonically with generation length up to 128–256 tokens, then can decline due to semantic drift [§3.4, Fig. 6, Appendix A].
Larger Qwen3 models exhibit both higher cued alignment and stronger SEE-vs-HEAR separability along the visual–auditory projection axis [§3.5, Fig. 8].
Ablating sensory words from generations (while preserving semantic content) cuts alignment significantly; but random sensory words (10 visual descriptors sampled from Visual Genome) also hurt — only scene-appropriate sensory detail lifts alignment [§3.3, Fig. 7].
A SEE-framed prompt improves “VQA without V” on MME by 64.78 → 67.14 with p < 0.05; gains concentrate on visually-grounded categories (existence, count, scene, posters) and not on pure reasoning categories like numerical_calculation [§3.6, Table 1].
Audio alignment is consistently lower than vision alignment; the authors attribute this to BEATs encoding low-level acoustic structure that maps less cleanly to lexical descriptions than DINOv2’s object/scene features [§6, Appendix C].

Method

The setup builds on Huh et al. 2024’s mutual-kNN alignment framework. For each caption $c$ , the authors compute two candidate representations of an LLM: $z(c)$ is the last-layer hidden state of the final token of a single forward pass on the caption alone; $\bar z(c)$ averages hidden states across all tokens and all layers of a 128–256 token autoregressive continuation from a prompt template + caption. The prompt template is one of {no cue, SEE cue, HEAR cue}. All models are frozen — no training. The alignment metric is mutual-kNN: given embeddings of all 1024 captions in the paired dataset under both the LLM and the sensory encoder, the score is the average fraction of shared top-k neighbours.

Sensory encoders are DINOv2-Base (image side) and BEATs-Iter3 (audio side). Language models are Qwen3 at six scales from 0.6B to 32B. Datasets are WiT and DCI (image–caption) and AudioCaps 2.0 + Clotho v2 (audio–caption). For the sensory-word ablation, the authors run two interventions on the generated continuations: (i) preserve semantics but replace modality-specific language with neutral phrasing, and (ii) append or substitute 10 random visual attributes parsed from Visual Genome. For the VQA-without-V experiment, all MME images are first captioned by Qwen2.5-VL-3B-Instruct, then Qwen3-14B answers yes/no questions from caption + question alone under either a neutral instruction or a “visualize then answer” framing.

Results

Mutual-kNN alignment of Qwen3-32B vs DINOv2 on WiT: SEE cue > no cue > HEAR cue; the mirror pattern holds on AudioCaps vs BEATs [Fig. 3].
Alignment increases nearly monotonically with model size for the matching cue, while no-cue alignment lags; Qwen3-0.6B’s no-cue and SEE outputs sit on top of each other on AudioCaps, but by 32B the no-cue projection drifts back toward HEAR — small models default to a visual framing absent any cue [§3.5, Fig. 8(b)].
Sensory-word ablation drops both vision and audio alignment significantly (no exact deltas reported in the abstract; effect is reported as “significant” in Fig. 7(a)).
Captions + 10 random visual words decrease vision alignment relative to the original caption; captions replaced entirely by random visual words decrease it further — sensory-word presence alone is not sufficient [Fig. 7(b)].
MME caption-only VQA on Qwen3-14B: 64.78 (no cue) → 67.14 (SEE), p < 0.05, n = 2334; OCR excluded; biggest gains on posters (+8.84), commonsense (+1.43), count (+5.00), existence (+5.00), scene (+1.50); text_translation drops 97.5 → 92.5 [Table 1].

Why it’s interesting

This is one of the cleanest empirical handles on the Platonic Representation Hypothesis — and it shifts the framing from “passively wait for scale to align modalities” to “actively prompt to align them, at inference time, for free.” That matters for the wiki because several recent filed papers either implicitly assume this convergence (CLIP-style joint pretraining) or attempt to construct it via paired data: this work suggests a weaker training requirement may be sufficient if the prompt does the lifting. It complements Deep sequence models tend to memorize geometrically; it is unclear why which found that sequence models spontaneously synthesize geometric structure over non-co-occurring entities, and supplies the cross-model analogue: the LLM’s text-space geometry already partially mirrors a vision or audio encoder’s geometry, and generation makes that geometry queryable. It also sharpens AGI Is Not Multimodal‘s “AGI is not multimodal” thesis — the present paper is consistent with that thesis (text-only models can be cued into modality-appropriate representations) but extends the claim by showing that which modality lives implicitly in the model can be controlled at inference. The sibling paper Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models from the same lab pushes in the other direction: training-time exploitation of the same shared structure via unpaired weight sharing.