ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion

TL;DR

ActionMesh is a feed-forward video-to-4D model from Meta Reality Labs that predicts a topology-consistent, rig-free animated 3D mesh from a single monocular video in ~3 minutes — an order of magnitude faster than per-scene optimization baselines (LIM, DreamMesh4D, V2M4). The recipe is “temporal 3D diffusion”: minimally inflate the self-attention in a pretrained image-to-3D latent diffusion backbone (TripoSG / 3DShape2VecSet) across a temporal axis so it emits a sequence of synchronized shape latents, then post-process with a temporal 3D autoencoder that converts the sequence of independent meshes into per-vertex deformations of a single reference mesh. Masked generative training lets the model accept a known 3D mesh as input, which unlocks {3D+text}-to-animation, {image+text}-to-4D, text-to-4D, motion transfer/retargeting, and long-video extrapolation from the same checkpoint. On a 32-scene Objaverse benchmark it reports 21% / 46% / 45% improvements over the best prior baseline on per-frame chamfer, global-aligned chamfer, and a motion-fidelity chamfer.

Key claims

Per-frame application of an image-to-3D generator to a video produces severe cross-frame inconsistencies (orientation flips, surface flickering); the temporal-diffusion stage exists specifically to eliminate this [§3.2, Fig. 3].
“Inflated attention” — reshaping (B, F·N, D) and running standard self-attention across all F frames before reshaping back — is sufficient to add a temporal axis to a pretrained 3D latent diffusion model while reusing every pretrained weight; relative-frame rotary embeddings inside the inflated attention further smooth motion [§3.2].
Masked-generation training (some 3D latents kept noise-free with flow-matching step set to 0, no diffusion loss on them) lets the same checkpoint condition on a known reference mesh at inference, which is what enables {3D+text}-to-animation, retargeting, and chunked extrapolation [§3.2, §3.4].
The temporal 3D autoencoder ingests the sequence of independent shape latents, runs inflated self-attention across them, and outputs a per-vertex 3D displacement field from frame i to frame j of a chosen reference mesh, yielding a single-topology animation downstream [§3.3].
On a 32-scene Objaverse benchmark, ActionMesh reports CD_frame 0.050, CD_4D 0.069, CD_motion 0.137 vs the best prior result of 0.063 / 0.127 / 0.247, and runs in 3 min vs 15–45 min [Table 1].
Stage II preserves Stage I’s 3D reconstruction quality (CD_frame unchanged) while converting a generic 4D mesh into a topology-consistent animated mesh; removing both stages (per-frame TripoSG) degrades CD_4D from 0.069 to 0.187 [Table 2].
The backbone is swappable: replacing TripoSG with Craftsman still yields competitive results (CD_frame 0.072, CD_4D 0.117) [Table 2].
Trained on synthetic data only, the model handles real-world DAVIS videos (jumping horse, walking bear, multi-part aquarium scene) after foreground segmentation [§4.2, Fig. 6].

Method

ActionMesh sits on top of the 3DShape2VecSet / TripoSG family: a 3D VAE encodes a point cloud sampled on a mesh surface into a small set of latent vectors (a “vecset”); a rectified-flow DiT generates such latents conditioned on an image (DINOv2 features). ActionMesh adds two minimal modifications.

Stage I — Temporal 3D Diffusion. Given F video frames, the model emits F synchronized 3D latents in parallel. The only architectural change is that each self-attention block is “inflated”: tokens are reshaped from (B, F, N, D) to (B, F·N, D), attention runs across all frames jointly, then reshaped back. Relative frame indices are injected as rotary positional embeddings inside the inflated attention to encourage smooth motion. To support a “start from a known 3D mesh” mode (and thus retargeting, extrapolation, etc.), training is run in a masked-generative regime: a random subset of latents in the sequence is kept noise-free with the flow-matching step pinned to 0 and excluded from the diffusion loss, so the denoiser learns to condition all other frames on the clean ones via attention.

Stage II — Temporal 3D Autoencoder. Stage I outputs a 4D mesh whose topology drifts frame-to-frame, which breaks texturing and rigging. Stage II reuses the original 3D encoder unchanged to compute per-frame latents, then trains a modified decoder that takes the entire latent sequence (again with inflated self-attention + relative-frame RoPE) and, given a query point with normal and a (source frame, target frame) pair, predicts the 3D displacement of that point from source to target. At inference the queries are the vertices of a chosen reference mesh, so the output is an animated 3D mesh with constant topology.

Applications. Because Stage I is masked-generative, the same checkpoint solves {3D+text}-to-animation (render the mesh, run a video model on the rendering, then feed both to ActionMesh), {image+text}-to-4D (image-to-3D first, then the above), text-to-4D (text-to-image, then above), motion transfer (run the {3D+video} path with semantically-mismatched A → B object pairs), and long-video extrapolation (chunk the video, carry the last predicted mesh forward as the source for the next chunk).

Results

Quantitative (Objaverse-32): CD_frame 0.050 (vs 0.063 for V2M4, best prior), CD_4D 0.069 (vs 0.127 for LIM), CD_motion 0.137 (vs 0.247 for DM4D); 3 min runtime vs 15 min (LIM) to 35 min (DM4D, V2M4) [Table 1].
Qualitative (Consistent4D): sharper details and stronger motion fidelity than LIM, DreamMesh4D, V2M4, and concurrent ShapeGen4D; the others show shape softening or temporal drift [§4.1, Fig. 5].
Ablations: Stage II alone adds 0 cost on CD_frame relative to Stage I; removing Stage I (per-frame TripoSG) drives CD_4D from 0.069 to 0.187, so cross-frame attention is the load-bearing component. Swapping TripoSG for Craftsman: CD_frame 0.072, CD_4D 0.117 — still beats prior SOTA [Table 2].
Real-world generalization: trained on Objaverse synthetic only; produces plausible 4D for DAVIS jumping horse, walking bear, aquarium scene [Fig. 6].
Motion transfer: retargets bird-flight video onto a 3D dragon mesh without explicit retargeting training, provided semantic correspondences between source and target hold [Fig. 4].

Why it’s interesting

The dominant 4D-content stack on the wiki is generative-video + explicit Gaussian/point-cloud state (see 4D Scene Generation and NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos / VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control / SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time) — pose-free 4DGS reconstructors plus a frozen Wan2.1 video backbone. ActionMesh attacks a strictly different output format: a single-topology animated mesh that drops straight into Blender/Maya texture and rigging pipelines, which is what production graphics actually consumes (the The flavor of the bitter lesson for computer vision critique of explicit 3D representations explicitly carves out meshes at the human–machine interface as legitimate). Compared to Vista4D: Video Reshooting with 4D Point Clouds, which keeps a 4D point cloud and re-renders, ActionMesh commits to mesh topology and pays for it with a Stage-II autoencoder. The ‘inflate-attention + masked generation’ recipe is also a cheap retrofit pattern for any pretrained image-to-3D vecset model (TripoSG, Craftsman, Hunyuan3D), which makes it a useful template independent of the specific benchmark numbers.