NAC: Neural Action Codec for Vision-Language-Action Models

TL;DR

NAC ports the neural audio codec recipe (SEANet convolutional encoder + multi-scale RVQ + Vocos/ISTFT decoder + adversarial training) to robot action tokenization for VLAs. Flatten an action chunk into a 1D multi-channel pseudo-waveform, drop the mel-spectrogram loss (which collapses downstream policy performance to zero on kinematic signals), keep the DAC-style adversarial discriminator (removing it also collapses performance), and use offset codebooks for a structured autoregressive policy interface. Compresses to 12 tokens per chunk (matching OAT/VQ-VLA, 19× fewer than binning, 3× fewer than FAST) and reports the highest downstream success on LIBERO-10 (49.7%), RoboMimic (33.9%), and 8 real-world tasks (50%) among 6 tokenizers.

Key claims

Audio-domain mel-spectrogram loss must be dropped for kinematic signals. Training NAC with a mel-spectrogram reconstruction loss drives downstream policy success to ~0% on LIBERO-10, while MSE, L1, DCT, or plain (non-mel) spectrogram losses all yield 44–49% [Table 1a]. Mel-frequency perceptual weighting matches human hearing, not robot kinematics.
The adversarial discriminator is load-bearing, not decorative. Removing the DAC discriminator collapses downstream success to 0%; DAC beats MPD and MRD variants on the same policy metric (49.5% vs 46.3%, 45.7%) [Table 1b]. First filed application of adversarial training to action tokenization per the authors.
The ISTFT head materially outperforms a linear decoder. Replacing the ISTFT reconstruction head with a linear layer drops policy success from 48.3% → 42.1% and reconstruction MSE from 0.0002 → 0.0006 [Table 1c].
NAC beats FAST, OAT, VQ-VLA, Diffusion Policy, and binning on downstream success at matched compression. LIBERO-10: 49.7% vs OAT 44.2, FAST 38.0, Diffusion Policy 25.5, VQ-VLA 10.9, Bin 4.0. RoboMimic: 33.9% vs OAT 31.9, Diffusion Policy 27.3, FAST 28.4. Real-world 8-task: 50% vs OAT 40, FAST 40, VQ-VLA 31, Diffusion Policy 22.5, Bin 6.25 [Table 2].
Multi-scale RVQ + offset codebooks let the policy predict all-scale-0 tokens first, then all-scale-1, etc., as a flat causal sequence — vocabulary size K·V + 1 with a BOS token, standard cross-entropy loss, modulo-arithmetic decoding [§3.3, Eq. 6].
Compression cost per chunk: 12 tokens at 1024 codebook entries — same as OAT/VQ-VLA, 3× fewer than FAST, 19× fewer than binning. End-to-end encode+decode latency 3.5 ms — slower than FAST (0.29 ms) but faster than VQ-VLA (11.0 ms) [Table 3].

Method

NAC treats a continuous action chunk A ∈ R^(B×H×D) as a 1D multichannel signal by flattening along time+feature into A' ∈ R^(1×(H·D)). A SEANet-style convolutional encoder with strided downsampling ratios [8,5,4,2] produces a latent at hop length 320, which is quantized by 4-stage multi-scale RVQ with per-stage temporal average-pooling — earlier stages capture coarse global structure, later stages capture high-frequency residuals. Codebooks are trained jointly (k-means init, EMA updates) with a heavily upweighted commitment loss (×1000) to prevent codebook collapse. The decoder is a Vocos backbone (Conv1D embedding → ResNet+attention → ConvNeXt blocks) with an ISTFT head that predicts STFT magnitude+phase and inverts back to the 1D signal — synchronized to the encoder hop length. Adversarial training uses a DAC discriminator (chosen over MPD and MRD for robotic signals per §4.2). The generator loss is L_g = MSE_recon + L_adv — critically, no mel-spectrogram term.

The behavioral cloning policy NACPolicy treats NAC’s 4 codebooks as offset vocabularies of size 4·1024 + 1 = 4097. During training the ground-truth code sequence is offset and concatenated in scale-major order (all scale-0 tokens, then all scale-1, …) and cross-entropy is minimized causally. At inference, tokens are partitioned back by scale via modulo arithmetic and passed to the frozen NAC detokenizer; the first H/2 predicted actions execute in receding-horizon fashion.

Results

Ablations (Table 1) isolate three load-bearing design choices: dropping mel loss (mel → ~0%, MSE → 49.2%), keeping the discriminator (DAC → 49.5%, no discriminator → 0%), and using ISTFT over linear head (48.3% vs 42.1%). Downstream policy comparison (Table 2) shows NAC first on all three regimes: LIBERO-10 49.7 ± 1.0% (vs OAT 44.2, FAST 38.0), RoboMimic 33.9 ± 1.9% (vs OAT 31.9), real-world 50% across 8 physical tasks with the largest gains on precise-correction tasks (grape grasping, block stacking). Compression stats (Table 3): 12 tokens per action chunk at 1024-entry codebooks (matches OAT/VQ-VLA count, 3× fewer than FAST’s 36, 19× fewer than Bin’s 224), 3.5 ms end-to-end tokenization+detokenization latency (~3× slower than FAST’s 0.29 ms but ~3× faster than VQ-VLA’s 11 ms). Parameter count 63M — comparable to OAT (65M) and VQ-VLA (66M).

Why it’s interesting

NAC adds a learned-tokenizer row to the action-representation lever on VLA Models that is orthogonal to the RL / clean-teleop / unified-VLM / WFM+action-expert / native-video-pretraining recipes already tracked — the substrate under all of them is “predict discrete action tokens with a transformer,” and NAC directly attacks the compression↔latency↔downstream-performance Pareto frontier of that substrate. Contrasts sharpest with Is Diversity All You Need for Scalable Robotic Manipulation?, which normalizes velocity out of demonstrations at the data level (velocity is noise), whereas NAC’s multi-scale RVQ preserves velocity in the tokenizer’s high-frequency residuals and lets the policy pick it up (velocity is signal). Sibling in framing to Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models: both change the structural interface between the action space and the transformer, but Action-QFormer changes the query interface between VLM and action head, while NAC changes the tokenization of actions themselves.