AGI Is Not Multimodal

TL;DR

Position essay (The Gradient, June 2025) from a Brown CS PhD candidate arguing that the dominant “scale + glue modalities together” path to AGI is structurally insufficient. Two claims do the work: (1) LLMs’ apparent semantic competence is most likely a syntactic phenomenon — induced rules over symbol sequences rather than a learned world model — so language proficiency is a misleading proxy for general intelligence; (2) today’s multimodal stacks (modality-specific encoders/decoders joined at a shared latent) make hidden structural assumptions about how observation and action spaces should be partitioned, which the Bitter Lesson predicts will hinder rather than help. The constructive proposal is to treat embodiment and environment interaction as primary and let modality-specific processing emerge, rather than being pre-specified.

Key claims

Disembodied definitions of AGI exclude problem spaces (“repair a car, untie a knot, prepare food”) that cannot be reduced to symbol manipulation; a complete AGI definition must include problems instantiated in physical reality [Preface].
The Othello-GPT result does not generalize to natural language: Othello moves provably determine board state, but there is no analogous reason a linguistic description determines the full state of the physical world [§2 of the world-grounding argument].
Generative models can score well on sequence prediction while failing to learn the world that produced the data, citing OthelloGPT’s “bag of heuristics” (e.g. “if B4 does not appear before A4, B4 is empty”) that hold for the training distribution but not all legal Othello games [§3 of the world-grounding argument].
The author’s central conjecture: LLM behavior is driven by brute-force memorization of abstract rules over symbols (a model of syntax), not by a learned world model — illustrated by the syntax/semantics/pragmatics decomposition and the observation that semantic well-formedness (“the fridge is in the apple”) can be re-encoded as new syntactic categories given a massive enough corpus [§§6–9 of the world-grounding argument].
The Bitter Lesson is widely misread: Sutton argues against hand-coded solutions but not against all structural assumptions — CNNs (translation invariance), Transformers (long-distance symbol relations), and 3DGS (object solidity) are explicit human structural intuitions that drove major advances [Bitter Lesson Revisited §3].
Scale maximalism worked for LLMs/LVMs because natural deposits of text and image data exist; an analogous embodiment-data corpus does not, which is why “stitch general models of narrow modalities together” is being offered as a substitute path [Bitter Lesson Revisited §4].
Joining modalities via shared latent space relies on modality-specific decoders to flesh out meaning, so “meaning” is decentralized across decoders trained under modality-specific objectives — not a recipe for coherent concepts [Bitter Lesson Revisited §6].
Today’s modality partition (text vs image as separate observation streams; text production vs motion planning as separate actions) is itself a structural assumption that may hinder discovery of more fundamental shared cognition — argued via the example that VLMs cannot count letters in a word “because they cannot see what they are writing” [Bitter Lesson Revisited §7].
Learn-from-scale trains models to copy the output of human concept-formation but not the process; the flexibility to form new concepts from few examples is a foundational AGI attribute that current in-context capabilities approximate but degrade with task complexity and distribution shift [Bitter Lesson Revisited §8].
Constructive recommendation: either be intentional about how modalities are united (drawing on classical linguistics / neuro-symbolic concept learning), or reformulate learning as embodied/interactive so disparate modalities fuse naturally — processing images, text, and video through the same perception system and producing text, manipulation, and navigation through the same action system [Conclusion].

Method

This is a position essay, not an experimental paper. The argument is structured as two literature-grounded claims (LLMs-are-syntax; multimodal-stitching-is-Bitter-Lesson-violating) plus a constructive proposal. Evidence is drawn from: the Othello-GPT line ([Li et al. ICLR 2023] and the LessWrong follow-up showing bag-of-heuristics learning), Vafa et al. NeurIPS 2024 on generative models that predict sequences without learning generating worlds, Andreas’s “Language Models, World Models, and Human Model-Building” MIT blog post, Sutton’s The Bitter Lesson, and the author’s own CogSci 2025 work on visual theory-of-mind enabling emergent writing systems. The syntax/semantics/pragmatics decomposition is the conceptual scaffold; the F1-racecar-built-from-ants metaphor is the scale-maximalism caricature.

Results

No quantitative results — this is opinion-piece scholarship. The strongest single empirical anchor cited is the “bag of heuristics” finding that OthelloGPT learns sequence-prediction rules that hold for the training set but not all legal games, used to argue that sequence-prediction quality is decoupled from world-modeling quality.

Why it’s interesting

This essay is a clean external articulation of a tension the wiki has been triangulating from several directions but hasn’t yet had a name for. It directly engages Unified Multimodal Models — the wiki’s filed UMM cluster (BAGEL, NEO-unify, OmniWeaving, UniMRG, UniG2U-Bench, AHA) is precisely the “shared latent + modality-specific decoders” recipe Spiegel argues is structurally limited; UniG2U-Bench’s finding that unified training generally hurts understanding (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?) and AHA’s audio-side hallucination rates of 80–99% (Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA)) are the kind of empirical evidence Spiegel’s structural argument predicts. The essay also complements The flavor of the bitter lesson for computer vision on the World Foundation Models page — both authors invoke the Bitter Lesson to argue that hand-engineered intermediates (3D representations for Sitzmann; modality partitions for Spiegel) should dissolve into end-to-end embodied/interactive substrates; they are independent arrivals at the same broad position from CV and from cognitive-AI respectively. Worth noting that Spiegel’s position is not the standard “scaling has stopped working” critique (cf. On the Slow Death of Scaling) — he grants that scale has worked for LLMs/LVMs and locates the failure in the modality-summation extension specifically.