Pretraining Frame Preservation in Autoregressive Video Memory Compression

TL;DR

PFP pretrains a lightweight neural network that compresses 20 seconds of video history into ~5k tokens of DiT context, using an explicit objective: reconstruct frames at arbitrary random temporal positions with PSNR/SSIM-grade fidelity. The pretrained encoder then drops in as a memory module when fine-tuning autoregressive video DiTs (Wan, HunyuanVideo) via LoRAs, giving long-range character/scene consistency without blowing up context length. The “preserve any single frame perfectly” objective is the key trick — it isolates the compression problem from motion/semantics learning, so the encoder can be trained on millions of in-the-wild videos cheaply, then plugged in. Compression at 4×4×2 (spatial × spatial × temporal) on top of the existing VAE/patchifier is the sweet spot the authors recommend.

Key claims

The context-length / quality trade-off in autoregressive video DiTs can be turned into an explicit, white-box learning problem by training an encoder φ(H) to compress history H and then measuring frame-retrieval quality at arbitrary indices Ω [§3.1].
Pretraining on millions of in-the-wild videos with a random-index frame-reconstruction objective produces a memory encoder that, by itself, reconstructs history frames with PSNR ≥17 at 4×4×2 compression and ≥20 at 2×2×1 [Table 1].
Index randomness during pretraining is required: fixing the retrieval positions (e.g. only ending frames) leads to a “cheating” solution where the encoder spends its 5k tokens on a few frames and ignores the rest [§3.1].
The encoder should not go through the VAE’s 16-channel bottleneck — its residual high-resolution branch outputs directly at DiT inner dimensions (3072 for WAN-5B, 5120 elsewhere) to preserve fidelity [§3.1, Fig. 3].
“Large Patchifier” (i.e. FramePack-equivalent compression via enlarged patchifier kernels, no new layers) gets PSNR 12.93, LPIPS 0.365 at 4×4×2 — significantly worse than the proposed dual-branch design at PSNR 17.41, LPIPS 0.171 [Table 1].
Pretraining vs. random-init at the same architecture / 100k finetune steps is qualitatively different: the pretrained variant maintains face / clothing / scene consistency, the random-init variant fails to attend to relevant history frames [§4.2, Fig. 6].
ELO from human user study: proposed 4×4×2 = 1216, proposed 2×2×2 = 1218, vs. WanI2V+QwenEdit baseline at 1198 [Table 2].
The compressed history can be concatenated on-the-fly without recomputation because the encoder is almost fully convolutional [§3.2].
Combining two encoders at different compression rates (e.g. 4×4×2 plus a 2×2×4 that prioritizes temporal compression) preserves fine spatial details like text on newspapers / billboards, at the cost of more context [§4.5, Fig. 10].
For Short-style videos with dense camera shot shifts, error accumulation during AR rollout is “absent or invisible”; long-single-shot videos still need extra design tricks [§4.2].

Method

The system has two stages. Stage 1 (pretraining the memory encoder φ). A long video history H is downsampled spatially and temporally and fed through the DiT’s VAE + patchifier + first-layer projection to produce a low-resolution context. In parallel, the original high-resolution video goes through a small 3D-conv + attention encoder that produces a residual vector added to the LR context after the first DiT projection (so the encoder bypasses the VAE’s narrow channel bottleneck and writes directly into DiT-inner channel widths like 3072/5120). The training task: given the compressed φ(H), randomly pick a subset Ω of frames to “reconstruct” (concretely, by adding noise-as-mask at levels sampled from [0,1] to all other frames and treating the picked clean frames as diffusion targets), then run the DiT to denoise. Crucially, Ω is sampled randomly at every step — fixed-position training collapses to a cheating solution that ignores most of the video.

Stage 2 (autoregressive video fine-tuning). The pretrained φ is plugged into a Wan or HunyuanVideo DiT as a history memory encoder, with rank-128 LoRAs on the DiT itself. The DiT now sees [memory tokens from φ(H)] + [current-window noisy latents] and generates the next second(s) of video. Inference repeatedly concatenates each new generated section into H. Two optional augmentations: a short sliding window of 3 latent frames (reduces shot-change frequency in dense scenes), and IP-Adapter-style cross-attention from φ’s penultimate features into each DiT block (helps with extreme-detail consistency, e.g. ordered items on a supermarket shelf).

Compression notation <sx><sy><st> means latent-space rates beyond the VAE+patchifier rate; the recommended sweet spot is 442 (4×4 spatial, 2× temporal latent compression on top of the existing pipeline).

Results

Pretrained-encoder-only reconstruction at 4×4×2 latent compression: PSNR 17.41, SSIM 0.596, LPIPS 0.171. At the gentler 2×2×1: PSNR 20.19, SSIM 0.705, LPIPS 0.121. Ablation “Only LR” (no residual high-res branch): PSNR 15.21. “Without LR” (no low-res branch): PSNR 15.73. “Large Patchifier” (FramePack-equivalent): PSNR 12.93 [Table 1].
Headline number from the paper’s framing: a 20-second history at ~5k context length, runnable on an RTX 4070 12GB consumer GPU [Fig. 1 caption].
AR-finetune consistency (Wan 2.2 5B base, 4×4×2): Cloth 96.12, Identity 70.73, Instance 89.89; user-study ELO 1216 vs. 1198 for the WanI2V+QwenEdit storyboard baseline [Table 2].
Without pretraining (same architecture, 100k finetune steps): Cloth drops to 87.12, Identity to 66.99, Instance to 81.13 — pretraining is responsible for roughly a +9 cloth-consistency / +4 identity points gap [Table 2].
Base-model scan: Wan 2.2 14B (high-noise) beats Wan 2.2 5B and HunyuanVideo 12.8B on Aesthetic (67.22) and Clarity (69.37) at the same 4×4×2 compression; user ELO 1231 [Table 3].
Training cost: 8×H100 cluster for pretraining; LoRA finetuning runs on single H100/A100 nodes with batch 64 at 480p on HunyuanVideo 12.8B — “personal or laboratory-scale” [§4.1].

Why it’s interesting

The “preserve any random frame perfectly” objective is a clean way to isolate the compression sub-problem from the joint motion-semantics-compression objective an end-to-end AR video DiT would otherwise tangle together — which is why pretraining transfers so cleanly into the AR fine-tune. For Luma’s long-video work, the most actionable bits are (a) the architectural recipe of bypassing the VAE channel bottleneck by writing into DiT-inner channel widths via a residual branch, (b) the explicit demonstration that FramePack-style large-patchifier compression is dominated by a dual-branch design with proper pretraining at the same compression rate, and (c) the casual claim that error accumulation vanishes for dense-shot-shift training data, which is a reasonable design heuristic for any Sora2/Veo-style storyboard model. The 5k-context / 20-second-history / 4070-class-GPU operating point also sets a useful target for consumer-deployable AR video.