X-Streamer: Unified Human World Modeling with Audiovisual Interaction

TL;DR

X-Streamer (ByteDance Intelligent Creation) is an end-to-end 18B framework that turns a single portrait into a real-time, multi-turn, audiovisual conversational agent jointly emitting text + speech + video. It frames the digital-human problem as unified multimodal world modeling and adopts a Thinker–Actor dual-transformer (à la Qwen2.5-Omni): a frozen GLM-4-Voice Thinker handles language/speech understanding and conversation, while a learnable Actor — initialized from GLM-4-Voice weights — autoregressively emits interleaved discrete text/audio tokens and continuous LTX-VAE video latents under chunk-wise diffusion forcing. Distributed across two A100 GPUs, the system streams 25 fps × 384² video with phoneme-level lip sync and up to 8K-token conversational context (~10 min of dialogue), sustaining hours-long video calls from arbitrary portraits.

Key claims

The recipe unifies multimodal understanding and generation across text, speech, and video in a single architecture trained end-to-end on unlabeled human talking videos — text/speech via a frozen pretrained LSM, video via an autoregressive diffusion transformer cross-attending to LSM hidden states [§1, §3.1].
Thinker–Actor dual-transformer: Thinker = frozen GLM-4-Voice (preserves conversational intelligence); Actor = parallel video transformer also initialized from GLM-4-Voice, plus a flow-matching text-speech head and HiFi-GAN vocoder; only the Actor’s video transformer is trained on multimodal sequences [§3.1, §4.1].
Interleaving rhythm: every chunk emits 13 text tokens + 26 speech tokens (a ~2 s window at 12.5 Hz) + 64 video tokens (8 frames at 384² via LTX VAE), so video generation is fully guided by text–audio semantics with minimal buffering latency [§3.1 Time-Aligned MultiModal Generations].
Audio–visual sync is enforced by (a) a cross-modal attention layer after each self-attention in every Actor block, conditioning video tokens on the corresponding text–audio chunk; (b) 3D RoPE on video + 1D RoPE on text-audio aligned along the temporal axis [§3.1].
Audio-Visual Context Attentions: video tokens use bidirectional self-attention within each 8-frame chunk and causal attention across chunks, with full cross-attention into the chunk-aligned Thinker hidden states; conversational text/audio context goes up to 8K tokens in the Thinker (~10 min dialogue) [§3.1].
Chunk-wise autoregressive diffusion with diffusion forcing (independent noise level $k_n$ per chunk) is used for the continuous video latent stream — closing the train/test gap and mitigating both inter- and intra-chunk drift over long horizons [§3.2 Diffusion Forcing].
Long-range identity is preserved by treating the reference portrait as a global condition placed at the start of the context sequence rather than via a heavyweight reference network; the model dynamically balances global identity tokens against historical context [§3.2 Global Identity Reference].
A chunk-wise pyramid denoising scheduler reduces the per-chunk diffusion cost from $N \cdot S$ to $N + S$ forward passes (N chunks, S steps), enabling real-time inference; combined with KV cache, no CFG (CFG=1), 2K-token visual context (~10 s window), and per-GPU placement of Thinker vs Actor across two A100s, the system sustains 25 fps multimodal streaming via WebRTC/LiveKit [§3.2 Real-Time Inference].
Training data: 2.7M talking-head clips / 4248.6 hours, curated from HDTF + CelebV-HQ + licensed sources with scene-cut detection and lip-sync validation; pretraining 3 epochs on 256 A100s, finetuning 200K steps on 220K high-quality long-form clips. No instruction finetuning of the full model — Thinker handles dialogue, Actor handles synchronized multimodal rendering [§4.1].
Against real-time audio-driven portrait baselines (JoyVasa, SadTalker, PD-FGC) at 384² with shared audio: X-Streamer leads on CPBD 0.55, FVD 573.36, and on Glo/ID/Lip/Div/VQ human-eval axes, while remaining at parity on lip-sync SyncC/SyncD [Table — Section 4.2].
Ablations: removing diffusion forcing collapses quality (FVD 573 → 1989, ID-Sim 0.75 → 0.29); removing global ID reference also hurts (FVD 794, ID-Sim 0.41); token-wise causal attention is worse than chunked bidirectional+causal (FVD 628, ID-Sim 0.70) [Table — Section 4.2 ablations].

Method

X-Streamer factors the digital-human problem as $p(T_o, A_o, V_o \mid I_{\text{ref}}, T_i, A_i)$ — given a portrait and streaming text/audio queries, emit synchronized text/audio/video responses — and solves it with a Thinker–Actor pair on a unified token sequence. The Thinker is a frozen GLM-4-Voice that reasons over up to 8K tokens of text+speech dialogue. Its hidden states are consumed by the Actor, which interleaves three modalities at a fixed cadence (13 text + 26 speech + 64 video tokens per ~2 s chunk). The Actor’s video transformer — architecturally a parallel GLM-4-Voice-initialized model — receives Actor self-attention plus cross-modal attention after every self-attention layer into the Thinker’s text-audio hidden states, with 3D-RoPE on video and temporally-aligned 1D-RoPE on conditional embeddings.

Video latents come from the LTX VAE (32×32×8 spatiotemporal compression), which keeps the per-second video-token budget tractable. The Actor predicts the next 8-frame video chunk by iterative denoising under a v-prediction diffusion objective, conditioned on the previous chunks’ clean latents and the chunk-aligned Thinker hiddens. Diffusion forcing — each chunk in a training mini-batch gets an independent noise level $k_n$ — closes the teacher-forcing gap that otherwise causes irreversible drift in continuous-latent AR video; bidirectional intra-chunk attention + causal inter-chunk attention is the chosen attention structure. Identity is anchored by placing $I_{\text{ref}}$ ‘s latent at the start of the sequence as a global reference rather than re-injecting it via an explicit reference network.

For real-time inference the team distributes Thinker and Actor across two A100s (Thinker hosts dialogue and emits hiddens; Actor renders video and streams via WebRTC/LiveKit), uses a KV cache, restricts visual context to 2K tokens (~10 s window) and disables CFG, and replaces the naïve $N\cdot S$ pyramid schedule of N chunks × S denoising steps with a chunked schedule of cost $N+S$ , sustaining 25 fps at 384².

Results

Real-time audio-driven baseline comparison (384² fixed): X-Streamer 0.55 CPBD / 573 FVD / 0.75 ID-Sim vs JoyVasa 0.37 / 749 / 0.73, SadTalker 0.20 / 778 / 0.78, PD-FGC 0.17 / 1184 / 0.42 — leads on perceptual sharpness and FVD while staying competitive on identity. Human-eval rates Globality 0.081, Expression 0.033, Identity 0.77, Lip 0.70, Diversity 0.85, Visual Quality 0.86, dominating prior real-time portrait animators on the non-lip axes.
Long-horizon stability ablations: removing diffusion forcing → FVD 1989, ID-Sim 0.29 (catastrophic drift); removing global identity reference → FVD 795, ID-Sim 0.41; replacing chunked-bidirectional-and-causal attention with token-wise causal → FVD 629, ID-Sim 0.70.
Throughput: on a single GPU, a 1-minute multimodal response takes 51.2 s in the Thinker and 58.2 s in the Actor; distributing the two transformers across two A100s yields 25 fps multimodal streaming. Peak VRAM 53 GB on a single GPU.
Reported support: hours-long consistent video chat from arbitrary portraits, with phoneme-level lip-sync and long-range identity/scene preservation.

Why it’s interesting

X-Streamer is the first wiki entry that unifies the Audio-Driven Character Animation frontier (talking-head video from portrait + audio) with Joint audio-video generation (synchronized text + audio + video output from a single unified model) and Autoregressive Video Generation long-horizon mechanisms (diffusion forcing, chunk-wise AR) — all in a single 18B system targeting real-time multi-turn conversation. It is structurally different from the open audio-driven-character-animation cluster (MultiTalk / Wan-S2V / HuMo / TalkingMachines), which freezes a pretrained Wan-family video DiT and only adds an audio cross-attention path: X-Streamer instead grows a new video transformer out of a language-speech LSM (GLM-4-Voice), with text/audio generation participating in the same autoregressive sequence rather than being supplied externally. Compared to TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models, which distills MultiTalk to a 2-NFE real-time student but still depends on externally provided audio, X-Streamer generates the audio itself from a conversational prompt and emits it in the same interleaved stream. The diffusion-forcing recipe is the same one used by Context Forcing: Consistent Autoregressive Video Generation with Long Context / MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation / FlowAct-R1: Towards Interactive Humanoid Video Generation for long-horizon AR video, and the global-identity-reference design is a minimalist alternative to FantasyTalking-style reference networks. In the joint-A+V picture (LTX-2, MOVA, SkyReels-V4, OmniForcing), X-Streamer is the first filed model that adds language understanding — and hence multi-turn dialogue memory — to the joint A+V stack, replacing the dual-stream-DiT-with-bidirectional-cross-attention recipe with a Thinker–Actor language-model-trunk recipe inherited from Qwen2.5-Omni.