OpenWAM — Extensible framework for video-action world models in robot learning

TL;DR

OpenWAM is a pre-release (v0.1.0, AGPL-3.0) Stanford SVL framework that factors WAM (world-action model) / VLA training + evaluation into five typed contracts — ExperimentConfig, VariantPipeline, VisualTower, PolicyVariant, ActionDecoder — so a single trainer, evaluator, and simulator-rollout runtime span three architectures (parallel_stream, dual_expert, causal_video_prediction) and six video/action programs plus GJD (Generalist Joint Denoising). The bet is that if the visual stack and program-selector interface are held fixed, controlled ablations across topologies and conditioning schedules become tractable in the same codebase — with the same audit trail (config hash, checkpoint identity, exact loader-cursor resume) — rather than requiring a per-paper fork.

Key claims

Three maintained topologies share one runtime: parallel_stream (single transformer over video+action tokens), dual_expert (separate video and action experts), and causal_video_prediction (video-only) — all support the six standard programs plus GJD except the video-only variant [README §Maintained Methods].
Six standard program selectors — video_then_action, action_then_video, joint, decoupled_same_step, video_noisy_to_action, action_noisy_to_video — plus GJD which samples joint / FDM / IDM submodes within one model; standalone forward_dynamics / inverse_dynamics preserve the strict GJD conditional contract (one clean t0 latent in a singleton chunk, one-frame conditional history, no task text, matched-prediction loss only) [README §Maintained Methods].
Reference 30-layer configs are FSDP workloads characterized on 4× 48 GB GPUs; program ablation is a single --set policy_variant.program=... flag change against the same trainer [README §Training].
--resume-from restores full training state (model, optimizer, scheduler, strategy/scaler, step state, sampler epoch/batch cursor) but the run is not bitwise identical because process and stochastic dataset/worker RNG streams aren’t checkpointed; resumable checkpoints are written only at optimizer boundaries because partial gradients aren’t serialized [README §Training].
Extension SDK exposes role-scoped open_wam.sdk modules for out-of-tree datasets, policies, decoders, attention profiles, and simulators, loaded through repeatable --extension module[:hook] arguments with a packaged scaffold that verifies registration, gradients, inference state, and packaging before custom code is introduced [README §Use OpenWAM With Your System].
Adapters ship for LIBERO, RoboTwin, CALVIN, heterogeneous LeRobot data, and synthetic fixtures; benchmark extras are dependency overlays rather than upstream source trees [README §Quick Start, §Evaluation And Rollout].
Reproducibility is treated as a first-class artifact: evaluation / sanity / simulator commands emit a versioned result envelope with source state, argv, config hashes, checkpoint identity, dataset metadata, package versions, and device details; refactors near model execution must pass immutable characterization tests without regenerating expected values [README §Reproducibility].

Method

The composition boundary is fixed: ExperimentConfig → VariantPipeline → VisualTower → PolicyVariant → ActionDecoder. VariantPipeline owns shared training + inference orchestration. VisualTower owns frontend encoding, visual backbone execution, decode stages, and runtime hooks — held constant so experiments vary one owned contract at a time. PolicyVariant owns parameter topology, architecture-specific packing, conditioning adapters, and recurrent state. ActionDecoder owns final supervised outputs, masks, losses, metrics, and committed actions. GJD is implemented as a single model that samples joint, FDM, and IDM submodes at training time; the strict conditional contract of the standalone FDM/IDM programs is preserved so that ablations against them stay meaningful. The parallel_stream topology is called out in the README as running the six standard programs, GJD, and standalone conditional FDM/IDM “through the exact LingBot-compatible runtime” [README §Maintained Methods] — signaling this framework is intended, at least in part, as an open reproduction / reimplementation substrate for the LingBot-VA line (Causal World Modeling for Robot Control (LingBot-VA)).

Results

No benchmark tables or per-task numbers are published in the README — “detailed evaluation results, trained model checkpoints, datasets, and the OpenWAM research paper are being prepared for public release” per the Upcoming Research Release section. What’s shipped today is the CPU public contract (config load + dataset construction + train forward pass + batch inference + recurrent rollout-style inference on synthetic fixtures with no private data, checkpoint, GPU, or external simulator required) plus the role-scoped SDK and the artifact-manifest infrastructure that would carry future release numbers. So the artifact is currently the framework, not a model / policy checkpoint / benchmark result.

Why it’s interesting

Sits on the same training-to-deployment authoring axis that FluxVLA Engine — Standardized VLA Engineering Foundation FluxVLA Engine occupies for VLAs, but scoped narrowly to WAMs / video-action models and to controlled program ablations across a fixed visual stack — which is closer in intent to the ablation discipline of Fast-WAM: Do World Action Models Need Test-time Future Imagination? Fast-WAM’s imagine-vs-no-imagine study or the GJD framing of LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control LingBot-VA 2.0, but as a shared substrate rather than a single paper’s methodology. Distinct from the deployment-plane infrastructure filings on the wiki (Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots Embodied.cpp runtime; DimOS — The Agentive Operating System for Physical Space (v0.0.13) DimOS message-passing plane) — OpenWAM is a training + evaluation framework, not a serving runtime. The “LingBot-compatible runtime” note under parallel_stream is the biggest concrete tell: this reads as an open academic-lab substrate for the WAM design space the closed industrial WAM line (LingBot-VA / Dyna-2 / µ₀) has been staking out, with the reproducibility manifest infrastructure (Scale Robot Policy Evaluation with Ray (Distributed Sim-Eval on Anyscale) sibling in the serving-side scaling axis) attached from day one.