AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization

TL;DR

AnchorWorld (Tsinghua + Kuaishou Kling team) is an egocentric interactive world model on Wan2.2 TI2V-5B that takes embodied human motion as the primary control signal and customizes the simulated world through a sparse set of anchor views — each carrying an RGB image (appearance), a 3D pose (spatial grounding in a unified world coordinate), and a text evolution prompt (state-change description). To compensate for the first-person view’s truncated body, training jointly uses third-person view (TPV) videos with a projection-based action representation so the model sees full-body motion under both viewpoints. Anchor-view conditioning is injected without architectural changes: in-context image-token concatenation along frame dim, pose-embedding addition pre-self-attention, and masked cross-attention for evolution prompts. Project page reports SOTA against PlayerOne and Context-as-Memory baselines on action control, static scene consistency, dynamic scene evolution, and UE/real-world OOD generalization.

Key claims

Egocentric first-person-view training data is severely under-supervised for full-body motion because limbs and torso are out-of-frame; jointly training on third-person view videos with a projection-based action representation that combines full-body 3D motion with camera trajectories injects the missing supervision [§Method, Hybrid-View Human Action Control].
The same 3D human-motion → 2D-projection pipeline supports both TPV pre-training and FPV adaptation by simply aligning the projection camera with the human head perspective — a single representation for both view regimes [§Method].
Three orthogonal localized priors per anchor view — RGB image, 3D pose in a unified world coordinate, and an evolution prompt — are sufficient to customize a self-evolving world without retraining the backbone [§Method, Evolvable Anchor-View Customization].
Anchor-view conditioning can be added to a frozen pretrained video DiT via three minimal hooks: (1) frame-dim concatenation of anchor image tokens with video latents (in-context conditioning), (2) pose embeddings added pre-self-attention, (3) masked cross-attention restricting each evolution prompt to its own anchor’s image and video tokens [§Method].
On egocentric action control, AnchorWorld beats a re-implemented PlayerOne on Wan2.2 TI2V-5B with hand poses excluded for fairness [§Comparisons, Baseline Settings].
On static scene consistency, AnchorWorld beats PlayerOne+anchor-view injection (PlayerOne-Scene) and CaM trained on both the project’s egocentric data (CaM-Ego) and CaM’s official UE dataset (CaM-UE) [§Comparisons].
On dynamic scene evolution, baselines must be adapted by appending the evolution prompt to their global text prompt because no prior work shares the setting; AnchorWorld’s masked cross-attention design outperforms this adaptation [§Comparisons].
The model generalizes out-of-distribution to UE and real-world scenes, indicating the anchor-view abstraction is not over-fit to the training dataset [§Comparisons].
During inference, anchor views can use a clean image without the first-person human inserted, even though training uses gray-body / red-wireframe visualization aids in the rendered TPV stream [§Visualization Note].
Hand poses are deliberately removed from the 3D human-action input because egocentric hand-pose estimation in the chosen dataset suffers from frequent out-of-view hands, occlusions, and multi-person interference [§Visualization Note].

Method

The base generator is Wan2.2 TI2V-5B. The control input is a 3D human motion sequence (root trajectory + body joints, with hands excluded). Two stages: (1) TPV pre-training uses third-person videos paired with full-body 3D motion, projected to 2D under the TPV camera; (2) FPV adaptation aligns the projection camera with the human head, so the same projection-based action representation now produces FPV-aligned 2D motion observations. This gives spatial pose awareness without first-person video supervision having to convey the full body.

World customization uses anchor views. Each anchor view is a tuple (RGB image, 3D pose in unified world coordinates, evolution prompt). The RGB carries appearance; the 3D pose locates the view in a unified world coordinate system shared with the human; the evolution prompt describes how the local scene evolves over time. The three signals enter the pretrained video DiT through three injection paths:

In-context conditioning — image tokens of the anchor views are concatenated with the video-latent tokens along the frame dimension before attention layers.
Pose embeddings — each anchor view’s 3D pose is encoded to a pose embedding and added to the corresponding video tokens before self-attention (no learned adapter on this path).
Cross-attention masking — evolution prompts go into the standard cross-attention layers, with a mask that forces each prompt to interact only with its own anchor-view image tokens and the video tokens (not with other anchors’ prompts/images).

The architecture itself is not modified — only the input streams and an attention mask are added. Training uses a progressive schedule (project page shows “Progressive Training” section).

Results

Qualitative comparisons reported across four task settings: egocentric action control, static scene consistency, dynamic scene evolution, and OOD generalization to UE / real-world scenes [§Comparisons].
Baselines: re-implemented PlayerOne on Wan2.2 TI2V-5B excluding hand poses (PlayerOne paper at arXiv:2506.09995, code unreleased at re-implementation time); PlayerOne-Scene = PlayerOne + AnchorWorld’s anchor-view injection; CaM-Ego = Context-as-Memory (Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval) trained on AnchorWorld’s egocentric data; CaM-UE = official CaM checkpoint on its UE dataset [§Baseline Settings].
For dynamic scene evolution, prior work doesn’t share the setting; baselines are adapted by appending evolution prompts to their global text prompt.
Concrete numerical metrics are not surfaced on the public project page (only qualitative side-by-side videos and ablation videos); the project page additionally hosts a demo gallery covering embodied egocentric control, anchor-view customization, dynamic text control, OOD scene generalization, out-of-view evolution, spatial pose awareness, and TPV human action control.

Why it’s interesting

AnchorWorld is the cleanest filed instance of the anchor-view conditioning pattern for interactive world models — sparse, localized “what’s at this 3D pose, looking this way, evolving like this” priors instead of either a dense scene memory or a global text prompt. The closest analog in the wiki is Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval, which is actually used here as a baseline (CaM): both target scene-consistent long-form FPV generation, but CaM stores raw historical frames and retrieves via FOV-overlap, whereas AnchorWorld stores interpretable anchor tuples that double as text-controllable evolution handles. Methodologically it is also a sibling to EgoX: Egocentric Video Generation from a Single Exocentric Video — both pair a frozen Wan-family video DiT with shaped multi-channel conditioning to bridge the third-person ↔ first-person view gap, EgoX as exo-to-ego translation and AnchorWorld as exo-supervised ego simulation. The hybrid TPV/FPV training (project full 3D motion under either camera so the model “sees” the full body during egocentric rollouts) is a structural fix for the same egocentric-supervision-gap that EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World addresses on the data side and EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks addresses on the evaluation side. The minimal-architecture-change recipe (in-context image concat + pose-embedding addition + masked cross-attention) keeps it firmly in the camera-controlled-video-diffusion cluster’s dominant “frozen Wan + small structural hook” lineage, but with the control axis extended from camera trajectories to anchor-tuple scene evolution. Worth watching whether anchor tuples generalize to non-egocentric world models — they are conceptually closer to a sparse “scene-graph as control surface” than to a video adapter.