Can Gemini Embeddings Be a Multimodal Encoder for LLMs? (Krafton AI experiment)

TL;DR

A research thread from Junhyuck Kim and Kangwook Lee (Krafton AI / advised by Kangwook Lee) asking whether Google’s gemini-embedding-2-preview API can serve as a universal multimodal encoder for a frozen LLM. The recipe is minimal: one API call per image or audio clip → 3072-d vector → 17M-param MLP adaptor → k “virtual tokens” prepended to a frozen Qwen3-4B-Instruct prompt; only the MLP is trained. It works well on categorical / closed-set tasks (image classification ≤97%, command STT ≤94%) and shows nontrivial zero-shot cross-modal transfer (an image-trained CIFAR-10 adaptor classifies environmental sounds at 59.8% — 3.6× random), but fails completely on open-vocabulary tasks like LibriSpeech transcription (0% exact match across every chunking and token-count variant). Companion artifacts: a blog post on kangwooklee.com and code/weights at github.com/krafton-ai/Can-Gemini-Embeddings-Be-a-Multimodal-Encoder-for-LLMs-.

Key claims

A single API embedding → 17M-param MLP adaptor → frozen Qwen3-4B pipeline reaches 97% CIFAR-10 object classification, 99% RAVDESS gender classification, 94% Google Speech Commands word-level STT, and 89% Fluent Speech Commands sentence-level STT (closed-set, exact-match) [tweet body, blog].
Harder fine-grained tasks degrade sharply on the same recipe: SVHN digits 62%, RAVDESS emotion 53%, IIIT-5K scene-text OCR 33% [tweet body].
The Gemini Embedding space has genuine cross-modal alignment: a CIFAR-10 image-trained adaptor classifies ESC-50 environmental sounds at 59.8% overall (dog barking → “dog” 95%, bird → “bird” 81%, cat → “cat” 70%, frog → “bird” 2%) with no audio seen during training, 3.6× above random [tweet body].
An audio-trained gender classifier partially transfers to face images at 62% on CelebA, having only heard voices during training [tweet body].
Training a single shared adaptor on all 8 tasks (4 audio + 4 image) with one virtual token collapses to “dog” for everything; 8 tokens recovers word STT to 90% and object classification to 69%; 32 tokens (261M params, LLM still frozen) reaches 85% object and 62% clothing — but sentence STT and OCR stay broken [tweet body].
On open-vocabulary speech (1000 unique LibriSpeech sentences) the recipe yields 0% exact-match across every variant tried — single embedding with 1 or 32 tokens, 2-second chunks with 1 token each, 1-second chunks with 1 token each. The model emits fluent English with zero word overlap with the actual speech [tweet body].
The author’s interpretation: what looks like transcription on closed sets is cluster recognition — the adaptor learns which of 169 known commands it’s seeing, not the actual phonetic content [tweet body].
Code, pre-trained adaptor weights, and pre-computed embeddings released at github.com/krafton-ai/Can-Gemini-Embeddings-Be-a-Multimodal-Encoder-for-LLMs- [blog].

Method

Every input (image or audio) passes through three stages, only the middle one trained:

Frozen Gemini Embedding API (gemini-embedding-2-preview) — one call → 3072-d vector.
Learned MLP adaptor (17M params) — projects the 3072-d vector into $k$ “virtual tokens” matching the LLM’s embedding width.
Frozen Qwen3-4B-Instruct — the $k$ virtual tokens are prepended to a text prompt, and the LLM emits a free-form answer via greedy decoding.

Supervision is task-driven: (image, label) pairs for classification, (audio, transcript) for STT. Cross-entropy is computed against the ground-truth label tokens. Per-task setups train a separate MLP for each task; the multi-task variant trains one shared MLP across all 8 tasks with task-specific text prompts and a swept number of virtual tokens (1, 8, 32). Open-vocabulary STT is also tested with audio chunking (1- and 2-second windows, one virtual token per chunk).

Results

Per-task adaptors, exact-match on held-out test sets, greedy decoding: object (CIFAR-10) 97%, gender (RAVDESS) 99%, word STT (Speech Commands) 94%, sentence STT (Fluent Speech Commands) 89%, clothing (Fashion-MNIST) 83%, digits (SVHN) 62%, emotion (RAVDESS) 53%, scene-text OCR (IIIT-5K) 33%.

Cross-modal zero-shot transfer (image-trained CIFAR-10 adaptor → ESC-50 audio): 59.8% overall, 3.6× random baseline. Audio-trained gender classifier → CelebA face images: 62%.

Shared multi-task adaptor: 1 token → collapses (defaults to “dog”); 8 tokens → word STT 90%, object 69%; 32 tokens → object 85%, clothing 62%, but sentence STT and OCR remain broken.

Open-vocabulary LibriSpeech STT (1000 unique sentences): 0% exact match across every configuration tried — single embedding (1 or 32 tokens), 2-second chunks (1 token each), 1-second chunks (1 token each). The interpretation: closed-set “STT” success is cluster recognition over 169 known commands, not phonetic transcription.

Why it’s interesting

A clean negative-and-positive result on the question “how much of multimodal understanding is just a good shared embedding space + a frozen LLM?” The positive half — 97% CIFAR-10 with 17M trained params, sub-minute training, and emergent cross-modal transfer — is a strong baseline for “what is the floor cost of bolting modality $X$ onto an open LLM,” and complements the existing UMM work cataloged in Unified Multimodal Models: it sits outside the AR / AR+Diffusion / MoT taxonomy because it doesn’t train the LLM at all, only a connector on top of a frozen API. The negative half is more interesting: 0% on LibriSpeech across every chunking variant says that the Gemini Embedding space, as currently exposed, encodes semantic categories but not phonetic content — which is a concrete falsification of the strongest “one encoder for everything” narrative around Gemini Embedding 2: SOTA multimodal embedding model (Google product announcement) (the Gemini Embedding 2 launch tweet). The 169-command “cluster recognition” framing is also worth flagging: closed-set STT benchmark numbers from any frozen-encoder + adaptor stack should now be read skeptically against open-vocabulary baselines, since you can hit 89% sentence-level accuracy without doing transcription at all. Contrasts with the encoder-free direction of NEO-unify: Building Native Multimodal Unified Models End to End (NEO-unify, no VE no VAE, pixel I/O end-to-end) — opposite ends of the “how much needs to be trained” spectrum on the same problem.