TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models

TL;DR

TalkingMachines adapts the pretrained WAN 2.1 14B I2V model into an audio-driven, real-time, infinite-length FaceTime-style avatar generator. The recipe is asymmetric distillation of a bidirectional teacher into a 2-NFE sparse causal autoregressive student via a modified CausVid pipeline, plus an 1.2B audio-projection module fed through new cross-attention layers with face-masked audio queries and a Speaking/Silence-mode embedding. Engineering-wise the contribution is a Score-VAE disaggregation server with CUDA-stream-overlapped NCCL collectives that keeps Time Taken Between Chunks below real-time on H100 hardware. Headline regime: 512×512, 2 diffusion steps, chunks of 3 latent frames, sparse causal attention restricted to the current chunk + previous chunk + starting reference chunk.

Key claims

A pretrained bidirectional I2V DiT (WAN 2.1 14B) can be converted into an audio-conditioned, real-time, infinite-length streaming avatar generator via three stages: warm-up at the target 512×512, audio pretraining with frozen non-audio params, and asymmetric sparse-causal DMD distillation [§5.2].
New cross-attention layers with a face-region attention mask plus a 1.2B audio projection module — temporally aligned via a local 5-latent-frame window per latent frame — beat global cross-attention for lip-sync because speech-driven motion is predominantly temporally local [§4.1].
A Speaking/Silence-mode design (zero-embedding swap for silent faces, SyncNet-conf-based per-face mode assignment at training) lets a single model handle multi-character listening/speaking turn-taking without separate models per role [§4.1].
CausVid’s per-chunk-independent timestep sampling (diffusion-forcing-style) is suboptimal for few-step generators because students exhibit piecewise flow trajectories — sharing one global timestep across all chunks within a video aligns the student with its pretraining setup [§4.2].
A sparse causal attention pattern — each chunk attends only to itself (bidirectional), the previous chunk, and the starting chunk containing the ground-truth reference image — gives temporal continuity, prevents error accumulation by always re-anchoring to the clean reference, and shrinks the KV cache for faster inference [§4.2, Eq. 7].
DMD distillation from 24 NFE (12 × 2 with CFG) to 2 NFE on 7 chunks of 3 latent frames is stable when (a) student-generated synthetic samples are progressively mixed in after initial convergence on real clips, and (b) a regression loss is added on top of DMD without any GAN term [§4.2].
Score-VAE disaggregation across separate GPUs is now necessary, not optional, for real-time DiT video — single-GPU VAE decoding + d2h transfer of the output tensor is ~30% of end-to-end chunk time, and sequence-parallel diffusion amplifies the share until VAE dominates [§4.3, Fig. 3].
A 2-GPU server with one diffusion worker + one dedicated VAE master beats a 2-GPU self-contained server on Time Taken Between Chunks; scaling to 2 diffusion workers + 1 decoder shows the cleanest scalability and consistently meets real-time [§4.3, Fig. 3].
CUDA streams overlap blocking NCCL collectives with VAE decoding on the master rank; KV-cache + timestep/guidance/context-embedding caching across timesteps eliminates redundant block recomputations [§4.3].
HSDP (PyTorch FSDP with parameter/gradient/optimizer-state sharding inside localized GPU groups) outperforms ZeRO Stage 3 at the Stage 3 DMD training, where three full diffusion-model copies (two trainable, ~450 GB total) must co-reside; H200 + InfiniBand gives ~2× per-step speedup over H100 + GPUDirect-TCPX, largely from intra-node sharding [§5.3].
720×720 training exceeds single-GPU activation memory even with activation checkpointing, requiring 2D parallelism: ZeRO/HSDP for parameters/gradients/optimizer state + DeepSpeed Sequence Parallelism for activations [§5.3].
Ablation on 2×2 (chunk size {3,7} × NFE {2,4}) at 512×512 shows lip-sync (Sync-C, Sync-D) is roughly invariant across configurations; FVD degrades mildly when chunk size and NFE are both reduced; the 3×2 setting uses 1 score GPU + 1 VAE GPU vs 4+1 for 7×4, a 4× hardware reduction at modest quality cost [§5.4, Table 1].
The end-to-end FaceTime-style application integrates an audio LLM → TalkingMachines (disaggregated DiT + VAE on H100s) → LiveKit WebRTC and runs on desktop and mobile browsers [§6].

Method

The base model is WAN 2.1 14B I2V, conditioned on text, an initial reference image, and audio embeddings. Audio is injected via new cross-attention layers (audio K/V from a 1.2B projection module, video queries on a face-region attention mask) and a per-latent-frame ±5-frame local-window alignment, inspired by Hallo3 and MagicInfinite. A Speaking/Silence-mode embedding (selected by SyncNet conf at training time) handles multi-character turn-taking by swapping the audio embedding for a learned silence token during listening phases.

Training proceeds in three stages: (1) 9k steps on 128 H100s adapting WAN 2.1 480p to 512×512 human-centric data; (2) 30k steps on 384 H100s training only the new audio layers with all other params frozen; (3) 20k steps on 128 H100s of asymmetric sparse-causal DMD distillation, all layers trainable. Stage 3 holds three model copies in memory (two trainable), totaling ~450 GB, motivating Hybrid Sharded Data Parallel over ZeRO Stage 3.

The distillation itself extends CausVid. A 21-latent-frame video is split into 7 chunks of 3 latent frames; bidirectional attention is preserved within each chunk, while a sparse causal mask restricts inter-chunk attention to (current, previous, starting) — the starting chunk always carrying the clean reference image so the student cannot drift far from a known-good state. Unlike CausVid, all chunks within a video share one sampled timestep (matching the pretraining regime; piecewise flow trajectories make mismatched-timestep cross-chunk attention suboptimal). Training mixes real clips with student-generated synthetic samples in a progressive schedule (real-only until partial convergence, then mix in synthetics), and adds a regression loss on top of the standard DMD loss without a GAN.

At inference time the system runs on a disaggregated Score-VAE server: one (or more) diffusion worker(s) on dedicated GPUs, and one master GPU running the VAE decode + d2h transfer + WebRTC streaming. NCCL collectives moving inputs/outputs between worker and master are overlapped with VAE decoding via additional CUDA streams. KV pairs for the current and previous chunks, plus timestep, guidance, and context embeddings, are cached across diffusion steps.

Results

512×512 target resolution chosen for both real-time deployment and faster training iterations; the 480p WAN 2.1 backbone is adapted downward.
2×2 ablation (Table 1, internal realistic talking-person dataset): chunk-size 7 / NFE 4 — FVD 208.2, Sync-C 6.28, Sync-D 8.94, 4 score GPUs + 1 VAE GPU; 7/2 — FVD 235.1, 6.13/8.77, 2+1; 3/4 — FVD 215.4, 5.89/8.55, 2+1; 3/2 — FVD 238.1, 5.83/8.73, 1+1.
Lip-sync (Sync-C, Sync-D) holds roughly constant across all four configurations; FVD shows mild degradation when both axes are reduced.
3/2 setting (1 score GPU + 1 VAE GPU) is the most compute-efficient real-time-feasible operating point; 7/4 is the highest-perceptual-fidelity setting at 4× the hardware.
Score-VAE disaggregation: in Case 4 (2 diffusion workers + 1 decoder), TTBC stays below the per-chunk video duration consistently; self-contained 1-GPU and 2-GPU baselines do not.
H200 + InfiniBand gives ~2× per-step Stage 3 training speedup over H100 + GPUDirect-TCPX, primarily by fitting all relevant tensors in a single node.
Deployment: real-time interactive avatar over LiveKit WebRTC on standard desktop and mobile browsers, with arbitrary input styles (photorealistic, anime, 3D avatars) animated via the reference-image conditioning path.

Why it’s interesting

This is the cleanest production-side instantiation of the CausVid → Self-Forcing → Causal Forcing lineage on the wiki: it commits to a specific 2-NFE / 3-frame-chunk / sparse-causal regime, picks a tight application (FaceTime-style audio-driven avatar), and reports the engineering details that make it run — disaggregation, CUDA-stream-overlapped NCCL, HSDP-over-ZeRO3 — that the more theoretical papers leave out. The shared-global-timestep deviation from CausVid is independently motivated and worth carrying into Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation‘s pipeline, which agrees that diffusion-forcing-style noisy-prefix AR training is suboptimal but for a different theoretical reason. The reference-chunk-as-permanent-anchor pattern is also a much simpler form of the structured memory bank in FlowAct-R1: Towards Interactive Humanoid Video Generation — TalkingMachines uses just (current, previous, reference) instead of FlowAct-R1’s 4-slot [reference | long-term ≤3 | short-term | denoising stream], suggesting the minimal anti-drift cache for a face-only target is much smaller than the humanoid case. Finally, this is the first wiki entry where Score-VAE disaggregation is presented as a required (not optional) optimization for real-time DiT video — a sharper statement than FastVideo: Create a 5s 1080p Video in 4.5s on a Single GPU‘s systems story, since FastVideo’s 5s/4.5s budget hides the VAE share that becomes binding at interactive latency.