FlowAct-R1: Towards Interactive Humanoid Video Generation

TL;DR

FlowAct-R1 is a real-time, streaming, infinite-duration interactive humanoid video generator built by adapting a Seedance MMDiT into an autoregressive video model via a chunkwise diffusion-forcing curriculum, a structured memory bank (reference + long-term queue + short-term + denoising stream), and a multi-stage distillation pipeline that reduces sampling to 3 NFEs. It is targeted at lifelike conversational avatars driven jointly by audio and text, with full-body control rather than the portrait-region scope of prior real-time work (INFP, ARIG, TalkingMachines). Reported real-time performance is 25 fps at 480p with TTFF ≈ 1.5 s on A100. Sits squarely in the same forgetting-vs-drifting design space as Self-Forcing, Self-Forcing++, LongLive, Rolling Forcing and FramePack — but specifically engineered around interactive humanoid use cases (live streaming, virtual companionship, video conferencing).

Key claims

Stream buffer is a fixed-size 4-component sequence — single reference latent, long-term memory queue (max 3 fully denoised latents), most-recent short-term memory latent, and a denoising stream of 3 chunks × 3 latents — sized to a 0.5 s wall-clock chunk; this yields continuous 25 fps 480p output [§2.1].
A “fake-causal” attention mask lets the denoising stream attend to reference + memory + itself, but blocks reference and memory from attending into the denoising stream — keeping fully denoised tokens uncorrupted across deepening DiT layers and reducing compute [§2.2].
To bridge the train-inference gap, the model probabilistically swaps GT memory latents for “generated-GT” latents (re-noised+denoised by an intermediate model) during training, an explicit Self-Forcing++–style trick to expose training to inference-time cumulative memory errors [§2.2].
Three-stage training curriculum: (1) Autoregressive Adaptation of a full-sequence MMDiT, (2) Joint Audio-Motion Training with Whisper-derived 25/s acoustic tokens via an IP-Adapter cross-attention branch, (3) Distillation to 3 NFEs [§2.2, §2.3].
Multi-stage distillation: first eliminate CFG by distilling guidance-scale-conditioned outputs into one model, then naive step distillation partitioning original NFEs into 3 segments, then DMD-style few-step score distillation modified to chunk videos and simulate streaming rollout [§2.4].
System-level optimizations: FP8 quantization on selected attention/linear layers, frame-level hybrid parallelism replacing token-level sequence parallelism to cut all-to-all overhead, fused kernels per DiT block, and an asynchronous DiT-denoising / VAE-decoding pipeline [§2.4].
Memory Refinement at inference: short-term memory is identified as the largest source of cumulative error; the system periodically re-noises and re-denoises short-term memory frames using reference + long-term memory as stable guidance [§2.5].
An MLLM is invoked at short intervals on (latest audio segment, reference image) to predict plausible upcoming actions, fed as priors to the MMDiT — explicitly to suppress the motion-repetition failure mode observed in KlingAvatar 2.0 and LiveAvatar [§2.6, §3].
Headline metrics: 25 fps 480p video on NVIDIA A100, TTFF ~1.5 s, 3 NFEs per chunk (chunk-size=3, micro-step=1, no CFG) [§1, §2.4].
User-study (GSB, 20 participants) vs. KlingAvatar 2.0, LiveAvatar, Omnihuman-1.5: FlowAct-R1 is reported as the only system to simultaneously achieve streaming + real-time + full-body control + generalization + vividness [§1 Table 1, §3].

Method

The backbone is Seedance, a Multimodal Diffusion Transformer with native cross-modal alignment, chosen for “reduced parameters, shot-based temporal slicing, and window-based spatial attention” that together make real-time interaction tractable. Video latents come from a VAE, text from Seedance’s native text path, audio from Whisper at 25 features/s aggregated into condition vectors with temporal overlap and fused via IP-Adapter-style cross-attention.

The core inference loop runs over a fixed-length stream buffer: [reference latent | long-term memory queue (≤3) | short-term memory latent | denoising stream (3 chunks × 3 latents)]. Each chunk is denoised in 3 NFEs under a fake-causal mask, output, then promoted into the memory hierarchy, in a tight 0.5 s → 0.5 s wall-clock loop. Training inserts errors into the memory components (“generated-GT” sampling) to simulate inference-time drift, in the same family of fixes as Self-Forcing++ and Context Forcing’s Error-Recycling Fine-Tuning. The distillation stack runs CFG-elimination → naive step distillation → chunk-aware DMD, with system-level FP8 + fused kernels + async VAE on top.

Results

The paper does not report public-benchmark numbers (VBench, MovieGenBench, etc.). The empirical claims are:

A 5×5 capability table (Stream / Real-Time / Full-body Control / Generalization / Vividness) where FlowAct-R1 is the only method with all five — Neural Voice Puppetry, INFP, LiveAvatar are missing one of {full-body, vividness}; Omnihuman-1.5 and KlingAvatar 2.0 miss streaming/real-time [§1, Table 1].
A user-study GSB evaluation over motion naturalness, lip-sync accuracy, frame structure stability, and motion richness with 20 raters, favoring FlowAct-R1 [§3]. Specific GSB margins are not reproduced in the abstract/intro; they live in Fig. 3 which is not part of the fetched markdown.
The system-level result: stable 25 fps at 480p with TTFF ~1.5 s on A100 [§2.4].

The evaluation is qualitative + user-study only; there is no quantitative comparison on a shared benchmark against the cited streaming baselines (Self-Forcing, Self-Forcing++, LongLive, Rolling Forcing, MagicInfinite, MotionStream).

Why it’s interesting

For Luma, this is the application-specialized cousin of the streaming AR-video lineage already filed under Autoregressive Video Generation: same forgetting-vs-drifting dilemma, same Self-Forcing-family training trick, same chunkwise rollout, but engineered end-to-end for an interactive product (audio-driven full-body avatars at 25 fps). Two ideas worth pulling out: (1) the “fake-causal” attention mask that protects memory tokens from corruption by the still-being-denoised stream — a cleaner statement of an asymmetry that other AR-video papers handle ad hoc; and (2) Memory Refinement, an inference-time periodic re-noise-and-denoise pass on short-term memory specifically, which is a different drift fix from Context Forcing’s training-time error-recycling and from WMReward’s inference-time guidance — it sits in between. The MLLM-as-action-planner module is also notable: it externalizes the “what should happen next” question out of the video DiT and into a separate, slower-loop planner, which is a recipe that may transfer to general long-form video generation beyond avatars.