FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space

TL;DR

FSVideo is a transformer-based image-to-video (I2V) diffusion stack that pushes the latent-space compression ratio to 64×64×4 spatial-temporal — substantially deeper than the ~8×8×4 typical of contemporary video VAEs — and then runs a two-stage DiT (14B base + 14B few-step upsampler) on top. The VAE is trained with REPA-style representation alignment and a GAN loss to keep reconstruction competitive at that compression. A 720×1280 5-second clip generates in ~19s of DiT compute on 2 H100s (~42× faster than WAN 2.1 per the project page TLDR), making this a useful datapoint on how far latent compression can be pushed before quality collapses.

Key claims

The video autoencoder uses a 64×64×4 spatial-temporal downsampling ratio (vs. ~8×8×4 in mainstream video VAEs) and “achieves competitive reconstruction quality” per the project page [Project page §1].
The DiT base model uses a “layer memory” design intended to enhance inter-layer information flow and context reuse within the DiT stack [Project page §1].
Generation is multi-resolution: a base DiT denoises at low resolution, then a few-step DiT upsampler runs at high resolution to recover fidelity [Project page §1].
End-to-end DiT inference for 720×1280×5s on 2× H100 takes 18.8s total — 14.2s base DiT (60 NFE at low resolution) + 4.6s SR DiT (8 NFE at high resolution) [Project page §2].
The combined 14B+14B model is reported to be competitive with popular open-source video models while being “an order of magnitude faster” [Project page §1].

Method

The system is structured around aggressive latent compression. A 3D VAE encodes raw frames into a latent tensor with 64× spatial downsampling and 4× temporal downsampling, an 8× spatial compression beyond what WAN-class VAEs do. To prevent reconstruction collapse at that ratio, the VAE is trained with REPA-style representation alignment (matching internal features to a pretrained self-supervised vision backbone) and an adversarial (GAN) loss on the decoder output — both are standard ingredients for keeping high-compression VAEs perceptually sharp.

On top of this latent, a 14B image-to-video DiT denoises at low resolution using 60 sampling steps (NFE), with a “layer memory” mechanism that the authors describe as a context-reuse path between transformer layers (the project page is sparse on specifics; the full paper text on arxiv was unavailable at filing time). The denoised low-resolution latent is then handed to a second 14B DiT acting as a few-step (8 NFE) super-resolution / upsampler stage that produces the final 720×1280 latent, which the VAE decodes to RGB. The two-stage split lets the bulk of compute (60 NFE) operate on the most compressed representation, where every FLOP is leveraged across the 64×64×4 compression.

Results

End-to-end DiT inference: 720×1280, 5s, 2× H100 → 18.8s total (base 14.2s + SR 4.6s) [Project page §2].
Step budget: 60 NFE base + 8 NFE upsampler [Project page §2].
Speed claim: ~42× faster than WAN 2.1 at 720p for the same clip length, per the originating Slack note (not on the project page text; presumably from the paper itself).
Quality claim: “Competitive performance against other popular open-source models” while being an order of magnitude faster [Project page §1]; specific benchmark numbers (VBench / human eval) are in the full paper, which was not retrievable at filing.

Why it’s interesting

The interesting bet here is the VAE: pushing spatial compression from 8× to 64× per axis is a ~64× reduction in DiT token count for the same video, which is where almost all of the claimed 42× speedup over WAN 2.1 has to come from. Whether the REPA + GAN combo is actually sufficient to keep reconstruction usable at that ratio is the open empirical question — Luma’s video stack has converged around lower compression for exactly this reason. If FSVideo’s VAE quality holds up under scrutiny (motion fidelity, fine textures, text rendering, faces), this is a strong existence proof that the current compression-quality frontier is movable, and the two-stage low-res-DiT + few-step-SR-DiT split is a natural recipe to copy. If the VAE underperforms, the layer-memory and multi-resolution recipe are still independently studyable.