MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation

TL;DR

MALT (Memory-Augmented Latent Transformers) tackles minutes-long video generation by carving the video into short latent segments and rolling them out autoregressively, with each new segment conditioned on a single compact memory vector that recurrently encodes the hidden states of every prior segment. The memory vector is read by added “memory attention” layers inserted into a W.A.L.T-style diffusion-Transformer; values and keys come from the memory, queries from the current denoising segment, with relative positional encoding for length extrapolation. Trained on ≤5s clips, MALT generates >120s — up to 1024 frames at 8 FPS — without the rapid quality drop seen in fixed-window AR baselines, by adding noise augmentation to memory vectors during training to make the model robust to its own rollout-time errors. The artifact reached the wiki via a US patent filing (US20250363171A1), which corresponds to the same research and shares the figures, claims, and Gupta-et-al. W.A.L.T citations.

Key claims

The denoising network factorises p(z₁:N | c) as an autoregressive product over segment latents, with each segment generated by reverse diffusion conditioned on (i) the text/conditioning signal and (ii) a memory vector summarising prior segments [§3].
The memory vector is built by recurrent attention over hidden states from prior segments — a small, fixed-size representation regardless of how many segments have been generated, breaking the quadratic blow-up of standard KV-cache conditioning [§3, Fig. 2A].
Memory attention layers (one per Transformer block in the diffusion-Transformer backbone) cross-attend from the current denoising segment’s queries to keys/values derived from the memory vector; the rest of the block keeps W.A.L.T-style spatial / spatiotemporal self-attention and text cross-attention unchanged [Fig. 2A, Fig. 2B].
Memory attention uses relative positional encoding between keys and queries, which the paper credits for letting inference generalise to context lengths far longer than the ≤5s clips seen during training [§3].
Memory attention is computed patch-wise at the same spatial location across data elements rather than over all tokens, keeping per-block compute bounded as the temporal context grows [§3, Fig. 2B].
Trained on videos ≤5s, MALT generates videos >120s at 8 FPS without significant frame-quality degradation; with sufficient compute it scales to ≥1024 frames at 8 FPS [§Detailed Description, Fig. 6].
The training objective adds noise augmentation to the memory vector — gradient is computed on memory vectors derived from ground-truth latents plus injected noise — so the model is trained on the kind of noisy memory it will actually see at rollout time, closing the train/inference distribution gap [§Training, Fig. 5B].
Gradients are not backpropagated through all preceding latent representations: the memory-vector recurrence is treated as a stop-grad input, so hardware memory and compute do not scale strictly with video length during training [§Training].
The architecture is conditioning-agnostic: the same model supports unconditional video generation, text-to-video (T5 / BERT text-embedding conditioning), and video continuation (image-frame conditioning) by swapping the conditioning signal c [§Detailed Description].
The decoder is a MAGVIT-v2-style causal 3D CNN autoencoder (without quantisation) run per segment in parallel, so the decoding step does not serialise the inference pipeline [§Decoder, Fig. 1B].
Long-horizon stability comes specifically from the noise-augmented memory training objective — the patent calls out that without it, the latent denoising network would be brittle to its own rollout drift, recovering the same forgetting-vs-drifting failure mode that motivates the broader AR-video-distillation literature [§Training].

Method

MALT’s pipeline: a long video is encoded by a MAGVIT-v2 video autoencoder into a sequence of latent tokens, which are then sliced into segments of ≥5 frames each. A diffusion-Transformer (W.A.L.T variant) denoises each segment by reverse diffusion; for every segment after the first, the denoiser additionally cross-attends, at every block, to a memory vector h that is updated recurrently as each segment is finalised. The memory vector encodes the hidden states of all previously generated segments in a compact, fixed-size form, so the network can condition on essentially-unbounded temporal context while staying within standard attention compute and memory budgets.

The W.A.L.T backbone is preserved: each “MALT block” (Fig. 2A) is a memory-attention layer, followed by self-attention layers (alternating spatial and spatiotemporal under the W.A.L.T windowed-attention schedule, Fig. 2B), followed by a text cross-attention layer. Memory attention uses relative positional encoding for length extrapolation and is computed per spatial-location patch across temporal tokens, keeping the compute bounded as the temporal context grows.

Training-time tricks. The memory vector is built from ground-truth segments rather than student rollouts, with added Gaussian noise (noise augmentation) so the model learns to denoise under imperfect memory — the paper’s stated mechanism for closing the train/inference gap and preventing quality decay over long rollouts. Gradients are not backpropagated through prior segments, so training cost does not grow with video length.

Results

The patent reports qualitative text-to-video generation at >120s / 8 FPS with no significant quality degradation, and headline scalability to >1024 frames at 8 FPS, trained only on ≤5s clips. The corresponding arXiv paper (2502.12632, Yu et al. 2025) reports quantitative wins on long-video benchmarks including Kinetics-600 long-video prediction (Fig. 4 in the arXiv version), but the patent body does not include the numeric tables in extractable form; further benchmark details are in the arXiv paper rather than this patent filing.

Why it’s interesting

MALT is the originating method in the lineage the wiki’s Autoregressive Video Generation concept has been tracking: it introduced the “compress prior segments into a fixed-size memory vector + cross-attend during denoising + add noise to memory during training” recipe that Pretraining Frame Preservation in Autoregressive Video Memory Compression (Pretraining Frame Preservation), Context Forcing: Consistent Autoregressive Video Generation with Long Context (Context Forcing), PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference (PackForcing), and FlowAct-R1: Towards Interactive Humanoid Video Generation (FlowAct-R1) all evolve in different directions — explicit per-frame retrievability, functional KV partitioning, learned mid-buffer compression, and slot-structured memory banks respectively. Two further connections worth noting. First, MALT’s “noise-augmented memory” training objective is the direct ancestor of Context Forcing’s Error-Recycling Fine-Tuning and LongVie 2’s degradation-aware training — same idea (perturb the context the way rollout would), different injection sites. Second, the artifact came to the wiki via a Google patent filing dated 2025-05-22, a year after the arXiv preprint; the patent route is unusual for the ingest pipeline. Note: one of the listed authors is affiliated with Luma AI on the arXiv paper.