RELIC: Interactive Video World Models with Long-Horizon Memory

TL;DR

RELIC is a 14B-parameter interactive video world model that targets all three of real-time streaming, long-horizon spatial memory, and precise user control in one system. The headline trick is a compact KV-cache memory of historical latent tokens annotated with both relative actions and absolute camera poses, so the model can retrieve 3D-consistent content without an explicit geometry stack. It is built by fine-tuning a bidirectional teacher to exceed its native 5s training horizon, then converting that teacher into a causal student via a memory-efficient self-forcing variant that supports full-context distillation over long rollouts. Reported result: 16 FPS real-time generation with stronger action following, long-horizon stability, and spatial-memory retrieval than prior work, from a single image + text prompt.

Key claims

An interactive world model must simultaneously solve real-time long-horizon streaming, consistent spatial memory, and precise user control; existing systems address only one in isolation, e.g. long-term memory mechanisms typically degrade real-time performance [project-page §1].
Long-horizon memory is represented as highly compressed historical latent tokens stored in the KV cache, each annotated with both relative actions and absolute camera poses — this camera-aware structure is what enables implicit 3D-consistent retrieval without an explicit 3D representation [project-page §2].
The bidirectional teacher (originally trained on 5-second clips) is fine-tuned to generate beyond its training horizon before distillation, so the causal student inherits long-rollout supervision rather than being capped at teacher window length [project-page §3].
A memory-efficient self-forcing variant enables full-context distillation over long-duration teacher and long student self-rollouts, addressing the student–teacher horizon mismatch in self-forcing-style training pipelines [project-page §3].
Implemented at 14B parameters on a curated Unreal-Engine-rendered dataset, generating at 16 FPS in real time, with reported improvements over prior work on action following, long-horizon streaming stability, and spatial-memory retrieval [project-page §4].

Method

RELIC is a causal autoregressive video DiT distilled from a bidirectional teacher. Two ingredients carry the design. First, the long-horizon memory is a compressed historical-latent KV cache in which each token is tagged with (a) the relative action that produced it and (b) the absolute camera pose at which it was rendered — collapsing what would otherwise be an explicit 3D state into a camera-aware token-level annotation that attention can dereference at retrieval time. Second, distillation runs in two stages: the bidirectional teacher is first fine-tuned past its 5s native horizon so that the long-context supervision exists at all, and then a memory-efficient self-forcing recipe converts the extended teacher into a causal student. The student trains on long rollouts of its own generations against the long-context teacher, which the project page argues is what closes the exposure-bias gap that limits the Self-Forcing / CausVid lineage to ~10s before drift. Training data is curated UE-rendered video, supplying paired action–observation–camera-pose tuples that real video does not.

Results

The project page reports 16 FPS real-time generation at 14B parameters from a single image + text description, with qualitative claims of more accurate action following, more stable long-horizon streaming, and more robust spatial-memory retrieval versus prior work. Quantitative benchmark numbers, ablations, and a paper PDF are not posted on the project page at filing time.

Why it’s interesting

RELIC sits squarely in the Autoregressive Video Generation lineage and is structurally a peer of Context Forcing: Consistent Autoregressive Video Generation with Long Context (Slow/Fast memory KV + Bounded PE) and Pretraining Frame Preservation in Autoregressive Video Memory Compression (pretrained memory encoder) — but its memory-annotation idea is sharper: instead of compressing or partitioning the cache, it enriches each token with absolute camera pose so retrieval is geometry-aware without an explicit 3D module. This is the exact design choice The flavor of the bitter lesson for computer vision argues a world model should make, and contrasts with the explicit-3D-Gaussian state used by VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control. The “fine-tune the bidirectional teacher past its training horizon before distilling” move also addresses a problem that Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation frames at the initialization stage — RELIC’s framing is closer to “fix the teacher’s effective horizon first, then distill” rather than “fix the distillation objective.” Real-time + memory at 14B places it as a candidate open counterpart to closed-flagship interactive world models like Genie 3 (Project Genie: Experimenting with infinite, interactive worlds).