Skip to content
Bud's Wiki

RealVideo: Real-time Streaming Conversational Video System (Z.ai)

TL;DR

Section titled “TL;DR”

RealVideo is a Z.ai system release (blog + GitHub + HF/ModelScope weights) that stitches an LLM, a TTS model, and an autoregressive-distilled audio-driven video model into a WebSocket-based real-time video calling pipeline. Text-in → LLM reply (GLM-4.5-AirX) → speech (GLM-TTS) → lip-synced video (Wan2.2-S2V-14B with Self-Forcing-style autoregressive diffusion) → streamed frames over WebSocket. It is not a new model or a paper — it is a deployable demo of the audio-driven-character-animation + AR-video-distillation stack glued together with an LLM/TTS frontend. Hits real-time (~500 ms per DiT block) on 2× H100 with sequence parallelism and 2 denoising steps.

Key claims

Section titled “Key claims”
  • The system is a composition of existing open models, not a new model: GLM-4.5-AirX for the LLM brain, GLM-TTS for voice synthesis, Wan2.2-S2V-14B for audio-driven lip-synced video, and Self-Forcing for the autoregressive diffusion distillation that makes it real-time [README, Acknowledgements].
  • Input is text only; output is a continuous streamed video of a user-supplied reference image lip-syncing to the LLM’s spoken reply [README Features].
  • Voice cloning is supported via a 3-second-or-longer reference audio clip uploaded alongside the image [README Usage].
  • Minimum hardware is 2× 80 GB GPUs (H100/H200); one GPU runs the VAE service, the rest are sharded for parallel DiT inference [README §4].
  • DiT block latency table: at sequence-parallel size 4 with 2 denoising steps, one block takes 306.39 ms uncompiled; at sp=4 / 4 steps, 480.68 ms compiled — both inside the ~500 ms budget the team cites for “smooth real-time generation” [README §4 table].
  • Compilation provides a consistent ~20–25 % latency reduction across the sp / step grid [README §4 table].
  • Communication is WebSocket-based bidirectional streaming so frames can be pushed as they are denoised, not after a full clip is rendered [README Features, Usage].

Method

Section titled “Method”

The pipeline has four stages. (1) A text message arrives over WebSocket. (2) GLM-4.5-AirX (the small/fast variant of the GLM-4.5 reasoning family) generates the textual reply. (3) GLM-TTS converts that reply to audio, optionally using a cloned voice from the user-uploaded reference audio. (4) Wan2.2-S2V-14B — the audio-driven cinematic video model released by the Wan/Tongyi team — generates the lip-synced video conditioned on the reference image and the TTS audio.

The video stage is what makes this real-time. Wan2.2-S2V-14B on its own is a 14B DiT designed for offline generation; the acknowledgements credit Self-Forcing as the second key dependency, meaning the deployed checkpoint is a Self-Forcing-style autoregressively distilled student of Wan2.2-S2V that supports causal/chunked rollout with a KV cache. The blog body itself is empty — the technical detail lives in the README, not the marketing page.

VAE decode runs on a dedicated GPU. DiT inference is sequence-parallelized across the remaining GPUs. The README ships a latency table parameterized by sp_size × denoising_steps, with sp=4 / 2 steps reported as the sweet spot for real-time.

Results

Section titled “Results”

The only quantitative claim is the latency table:

sp size / steps2 (uncompiled / compiled)4 (uncompiled / compiled)
1563.84 / 442.61943.13 / 723.06
2384.86 / —655.92 / 527.11
4306.39 / —513.72 / 480.68

Bolded entries are inside the 500 ms-per-block budget. With 2 GPUs (sp=1 + VAE), 442.61 ms compiled is achievable; with 4+ GPUs (sp=4), uncompiled inference is already inside budget at 306.39 ms.

No video-quality benchmarks (Sync-C, FID, FVD, CSIM) are reported in the blog/README — this is a system release, not a model paper. Quality is implicitly inherited from Wan2.2-S2V-14B and degraded by the few-step Self-Forcing distillation. The Wan-S2V tech report (Wan-S2V: Audio-Driven Cinematic Video Generation Table 1) is the closest available quality reference for the underlying model.

Why it’s interesting

Section titled “Why it’s interesting”

RealVideo is the first filed artifact on the wiki to productize the full stack — LLM + TTS + audio-driven video distillation — as a deployable real-time avatar/video-call system. The components are all already filed: the video backbone is Wan-S2V: Audio-Driven Cinematic Video Generation (Wan2.2-S2V-14B), the real-time recipe is Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (Self-Forcing), and the LLM is part of the GLM-4.5 family (GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models). The cleanest precedent in the Audio-Driven Character Animation cluster is TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models (TalkingMachines), which independently built a real-time FaceTime-style avatar on Wan2.1-I2V-14B with 2-NFE sparse causal AR distillation — RealVideo is essentially the Wan2.2-S2V analogue with an LLM brain bolted on.

For Luma the practical signal is the latency table: at the 14B Wan2.2-S2V scale, real-time (~500 ms/block at chunked rollout) needs 2-step inference and sequence parallelism, and is GPU-bound rather than algorithmic. The bottleneck is no longer the recipe — it’s the dollars per real-time avatar session.

See also

Section titled “See also”