Andrew Carr — SuperMimic open reimplementations diverge during Bones Seed; four engineers compare notes

TL;DR

Andrew Carr (Cartwheel co-founder) asks publicly whether anyone has reimplemented SuperMimic (SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model) in a way that does not diverge roughly 20% into “Bones Seed” training — reports that his team’s and other open implementations all collapse before fully recreating the paper’s results. The thread draws three concrete technical replies: Lizhi Yang (morphmimic repo) finds the critic overestimates without SHAC-style targets and that a LayerNorm critic only pushes divergence out to ~80k iterations; Kyle Morgenstein (@KyleMorgenstein) reports regularization + differentiable rewards for BPTT help, especially for hardware transfer; Sammo (@sammo3x) reports a port that trains stably through all of BONES-SEED after 11 failed runs — 0 → 68k iterations stable, plus an 18k fine-tune to 86k, no collapse, but short of the paper’s 452-failed-clip number and has not run the full 320k schedule.

Key claims

Carr: multiple independent open reimplementations of SuperMimic diverge about 20% through the “Bones Seed” training phase and cannot fully recreate the paper’s reported results [tweet OP].
Lizhi Yang (lzyang2000/morphmimic): without an SHAC-style target, the critic overestimates and the actor chases that error through ∂V/∂s — a LayerNorm critic alone only extends divergence to ~80k iters, and clamping TD targets at r_max/(1−γ) stopped divergence on resume (long runs untested) [reply, github.com/lzyang2000/morphmimic].
Kyle Morgenstein: adding more regularization helps significantly, especially for hardware transfer, and notes that for BPTT (backprop-through-time) the rewards have to be differentiable [reply].
Sammo (@sammo3x): a port trains stably across all of BONES-SEED after 11 failed runs — 0 → 68k iterations with no collapse, plus an 18k fine-tune to 86k; has not run the full 320k schedule, and is short of the paper’s 452 failed clips [reply].

Method

Not applicable — this is an open help request from Carr, with technical replies from three engineers running their own reimplementations of the SuperMimic recipe (differentiable world model + BPTT policy learning for humanoid whole-body motion tracking). The common failure signature is divergence of the actor-critic pair at ~20% of training on the “Bones Seed” curriculum. The failure modes named across replies are:

Critic overestimation under BPTT, remedied by SHAC-style targets or by clamping TD targets at r_max/(1−γ) (Lizhi Yang).
LayerNorm-only on the critic merely postpones divergence to ~80k iters rather than preventing it (Lizhi Yang).
Under-regularization, particularly as a hardware-transfer failure (Kyle Morgenstein).
Non-differentiable rewards incompatible with BPTT (Kyle Morgenstein).
Dataset-size truncation — Sammo’s working port is “short of the paper” at 452 failed clips, suggesting the reported SuperMimic numbers may depend on the full-schedule / full-data regime that the open reimplementations have not yet matched.

Results

Lizhi Yang’s morphmimic fix (clamped TD targets): divergence stopped on resume, long runs not yet validated.
Sammo’s port: 0 → 68k iterations stable through BONES-SEED, +18k fine-tune to 86k, no collapse — but has not run the full 320k-iteration schedule from the paper and is still short of the paper’s final clip-failure count.
Carr’s team and “other open implementations”: all diverge ~20% through Bones Seed; no fully recreated result in the open yet.

Why it’s interesting

SuperMimic is the sharpest recently-filed compute-reduction datapoint for humanoid whole-body control (SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model), reporting >1000× GPU-hour reduction (3 h on 1 GPU vs 7 days on 128 GPUs) and flat world-model prediction error from 1 h to 264 h of motion data. If no open reimplementation can clear BONES-SEED without collapse, the headline scaling claim is unverified outside the authors’ pipeline — a reproducibility hole the team should factor into any plan to reproduce the recipe for internal use. The specific failure signature named in the thread (BPTT critic overestimation, non-SHAC targets, missing clamp) is the same stability-locus the Training stability at scale concept page tracks for LLM pretraining — but applied to the model-based analytic-gradient inner loop that SuperMimic uses in place of PPO. This is one of the first filed examples of the “architecture + optimizer + target-clamp” stability discipline being needed for a differentiable-world-model RL loop at humanoid scale, not just for LLM pretraining stacks.

Sammo’s 11-failed-runs-to-stable datapoint is the specific signal to carry forward: BONES-SEED is apparently reachable in the open with sufficient hyperparameter + target-clamp care, but the full 320k-iteration schedule and the paper’s clip-failure count remain unmatched. Complements GEN-0: Embodied Foundation Models That Scale with Physical Interaction GEN-0’s “ossification at O(1B) parameters” ossification observation and Mitigate Silent Expert Death In Ultra-Sparse MoE‘s “silent expert death” — all three are stability failure modes that are invisible to training-loss curves in the small regime and only surface at full-schedule scale.