AToken: A Unified Tokenizer for Vision

TL;DR

AToken (Apple) is a single visual tokenizer that handles images, videos, and 3D assets in a shared 4D latent space, and is trained to serve both high-fidelity reconstruction and semantic understanding from one set of weights. The architecture is a pure transformer with 4D rotary position embeddings, supporting inputs of arbitrary resolution and temporal duration and emitting either continuous or discrete tokens. Training is adversarial-free — perceptual + Gram-matrix losses replace the GAN discriminator typically used in VQ-GAN-lineage tokenizers — and follows a progressive curriculum (image → +video → +3D). Headline numbers: 0.21 rFID with 82.2% ImageNet zero-shot for images, 3.01 rFVD with 40.2% MSRVTT retrieval for video, 28.28 PSNR with 90.9% classification for 3D, all from one tokenizer.

Key claims

AToken is the first unified visual tokenizer that simultaneously targets high-fidelity reconstruction and semantic understanding across images, videos, and 3D assets in a single framework [Abstract].
The latent representation is a shared 4D space (sparse), giving one geometry for 2D grids, temporal sequences, and 3D assets rather than per-modality latents [Abstract].
A pure transformer with 4D rotary position embeddings handles inputs of arbitrary resolution and duration without modality-specific heads [Abstract].
The training objective is adversarial-free, combining perceptual and Gram-matrix losses; this is sufficient to reach state-of-the-art reconstruction quality without a GAN discriminator [Abstract].
A progressive training curriculum (single images first, then videos, then 3D) is used, and the tokenizer supports both continuous and discrete latent tokens [Abstract].
Multimodal training enhances single-modality performance — image reconstruction improves as video and 3D capabilities are added during the curriculum [Apple ML Research page; corroborated by independent summary].
Image: rFID 0.21, ImageNet linear/zero-shot 82.2% [Abstract / Apple ML page].
Video: rFVD 3.01, MSRVTT text-video retrieval 40.2% [Abstract / Apple ML page]. (The tweet quotes 32.6% — appears to be a different MSRVTT split / metric than the 40.2% quoted in the abstract.)
3D: PSNR 28.28, 3D classification 90.9% [Abstract / Apple ML page].
The tokens drive multiple downstream applications without rebuilding the tokenizer: image generation with both continuous and discrete tokens, text-to-video generation, image-to-3D synthesis, and multimodal LLMs [Abstract].

Method

AToken is a pure-transformer encoder/decoder over a sparse 4D latent space. Visual inputs (an image, a video clip, or a 3D asset) are patchified and indexed with 4D rotary position embeddings — the embedding carries the modality’s natural axes (e.g. spatial x/y for an image, x/y/t for a video, x/y/z for 3D), so the same attention stack reads all three. The latent supports both continuous heads (for diffusion-style downstream usage) and discrete heads (for autoregressive MLLM usage). Training combines reconstruction (pixel-level) with perceptual losses on a feature backbone and Gram-matrix style losses to capture texture statistics; there is no adversarial discriminator. The curriculum starts with single images, then adds video, then 3D — and the paper reports that earlier-modality reconstruction improves as the model picks up later modalities, evidence that the shared 4D latent is gaining from multimodal training rather than being diluted by it.

Results

Image: 0.21 rFID and 82.2% ImageNet accuracy at the tokenizer level [Abstract].
Video: 3.01 rFVD reconstruction and 40.2% MSRVTT retrieval [Abstract].
3D: 28.28 PSNR reconstruction and 90.9% 3D classification accuracy [Abstract].
Downstream applications validated across image gen (continuous and discrete), T2V, image-to-3D, and multimodal LLM understanding, with competitive performance across all benchmarks [Abstract].
Stated finding: image reconstruction improves as video + 3D capabilities are added to training — the multimodal training is net-positive on single-modality metrics [Apple ML page].

Why it’s interesting

AToken stakes out a more aggressive version of the unified-tokenizer thesis than anything currently filed on the wiki. UniTok: A Unified Tokenizer for Visual Generation and Understanding argues the bottleneck for a 2D image tokenizer that serves both reconstruction and understanding is discrete-codebook capacity, and addresses it with multi-codebook quantization on a single modality. AToken pushes the same recipe across three modalities at once and gets the additional empirical claim that multimodal training helps single-modality reconstruction — a stronger statement than UniTok’s “joint reconstruction+contrastive is essentially conflict-free at fixed capacity.” The 4D RoPE + shared sparse latent is also the cleanest filed answer to the “what should a video / 3D tokenizer share with an image tokenizer” question that What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion and OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence address from the latent-geometry / sparsity angles. For the Unified Multimodal Models concept page, AToken is the first filed tokenizer-side data point that includes 3D; the discrete branch (UniTok / Liquid / VILA-U) has stayed 2D, and the encoder-free branch (NEO-unify: Building Native Multimodal Unified Models End to End) skips a tokenizer entirely. Also notable that this is from Apple — pleasant break from the steady stream of ByteDance / Alibaba unified-tokenizer releases.