ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models

TL;DR

ActionPiece (DeepCybo) is an action tokenizer for autoregressive VLAs whose central claim is that pointwise MSE reconstruction is the wrong objective — what matters is whether relative physical distances between actions across demonstrations survive quantization, because tokens that collapse similar actions distort the corrective adjustments a policy needs to make in different contexts. The paper introduces Physical Rank Consistency (PRC) as a tokenizer metric that measures how well local physical-distance rankings survive encode+decode, and trains ActionPiece with two ranking-preservation losses: one supervising near-far ordering in the encoder feature space and its quantized counterpart, and one applying the same ordering to codeword assignment distributions. Under a fixed Qwen3-VL-4B autoregressive policy setup it reports 94.8% on LIBERO, 68.8% on the unseen-task LIBERO-Plus split, 71.9% on SimplerEnv, and 51.5% average across VLA-Arena L0–L2.

Key claims

Pointwise reconstruction error under-specifies action tokenization for policy learning. Small MSE errors can leave similar actions clustering around a representative motion after decode, so context-dependent adjustments are “diminished, distorted, or even reversed” [Abstract]. This is what the PRC metric is designed to detect.
Physical Rank Consistency (PRC) evaluates decoded actions, giving a common reference across token vocabularies and decoder architectures. By scoring how local physical-distance rankings survive tokenize→detokenize, PRC complements pointwise reconstruction with a measure of relational fidelity that is independent of codebook size or decoder family [Abstract].
Two joint objectives are load-bearing: physical rank preservation on encoder+quantized features, and quantization regularization on codeword assignment distributions. Both objectives augment reconstruction rather than replace it; component ablations show they jointly improve both PRC and downstream policy success [Abstract].
On a fixed Qwen3-VL-4B autoregressive policy training setup, ActionPiece reports 94.8% on LIBERO and 68.8% on unseen-task LIBERO-Plus [Abstract].
Additional evaluations: 71.9% on SimplerEnv and 51.5% average across VLA-Arena L0–L2 [Abstract].

Method

ActionPiece extends a standard learned action tokenizer (encoder + VQ codebook + decoder → discrete tokens fed to an autoregressive VLA) by adding two ranking-based auxiliary losses on top of the usual reconstruction objective. The physical rank preservation loss operates on distances between action samples in the encoder feature space and in the quantized-feature space, and pushes the ordering of near/far neighbors in feature space to match the ordering of near/far neighbors in physical action space; the quantization regularization loss extends the same ordering constraint to codeword assignment distributions, so actions that are near/far in physical space induce assignment distributions that are also near/far as measured over the codebook. At deployment the tokenizer runs unchanged: discrete tokens go into a standard autoregressive VLA (Qwen3-VL-4B in the reported experiments) and executions come back through a frozen decoder. The abstract does not disclose codebook size, per-chunk token count, encoder architecture, or latency.

Results

Under the shared Qwen3-VL-4B policy training setup [Abstract]:

LIBERO overall: 94.8%.
LIBERO-Plus (unseen tasks): 68.8%.
SimplerEnv: 71.9%.
VLA-Arena L0–L2 average: 51.5%.

Component ablations “show that the two objectives jointly improve PRC and policy success” [Abstract] — ranking preservation and quantization regularization are both load-bearing, though the abstract does not disclose per-component deltas or head-to-head numbers vs FAST / OAT / VQ-VLA / NAC / binning. The paper only reports numbers on ActionPiece + Qwen3-VL-4B; no matched-tokenizer sweep is described in the abstract.

Why it’s interesting

Direct sibling to NAC: Neural Action Codec for Vision-Language-Action Models on the learned-action-tokenizer row of the VLA Models recipe-lever board — both papers change the tokenizer used by autoregressive VLAs, but they attack orthogonal failure modes. NAC (Jawaid & Xiang) imports the neural-audio-codec stack (SEANet + multi-scale RVQ + Vocos/ISTFT + DAC discriminator) and its ablations argue the adversarial signal is what makes the codec preserve high-frequency structure; ActionPiece argues that even with an adversarial or reconstruction-trained codec, the wrong invariant is being preserved — pointwise fidelity rather than cross-demonstration ordering — and adds an explicit rank-consistency supervision on top. Sibling in structural-interface framing to Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models but at the tokenization layer rather than the query interface between VLM and action head. Sits productively next to Is Diversity All You Need for Scalable Robotic Manipulation?‘s “canonicalize velocity out of demonstrations” — both papers treat cross-demonstration structure as the object of design, but Shi et al. normalize it at the data level while ActionPiece preserves and supervises it at the tokenizer level. The 68.8% LIBERO-Plus figure is the concept-relevant comparable: it puts ActionPiece + Qwen3-VL-4B in the same regime as Anchor-Align: Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment Anchor-Align, Act with Intent: Distilling Behavior Intent for Vision-Language-Action Models INDI, and LIT — two-stage VLA/WAM training constrains visual conditioning via latent interface to prevent visual shortcuts LIT — four different training-time answers to the “why does BC generalize poorly under distribution shift” question, each targeting a different layer of the stack (observation representation, latent intent, visual interface, action tokenizer).