Prompt Relay: Inference-Time Prompt Routing for Temporal Control in Multi-Event Video Generation

TL;DR

Prompt Relay is a training-free, inference-time mechanism that gives a pretrained text-to-video diffusion model temporal control over multi-event prompts by modifying cross-attention with a distance-based locality penalty. Each prompt is bound to a temporal segment [t_start, t_end]; query tokens whose latent frame index falls outside that segment have their attention to the segment’s key tokens smoothly down-weighted by a Gaussian-tail penalty subtracted from the QKᵀ logits. The decay rate σ is chosen so the multiplicative prior exp(−C) hits a user-defined value ε at segment boundaries, ensuring smooth handoffs between adjacent prompts. No retraining, no architecture change — just a per-token logit bias inside cross-attention — so it composes with any DiT backbone whose cross-attention is accessible.

Key claims

Temporal locality of multi-event prompts is enforced by subtracting a distance-based penalty C(Q, K) from cross-attention logits before softmax, so each prompt only influences query tokens within its temporal interval [§ Temporal Cross-Attention Routing].
The penalty C(i, j) = 1[j ∈ K_s] · ReLU(|f(i) − m_s| − w)² / (2σ²) uses the latent frame index f(i) of the query and the midpoint m_s = (t_start + t_end)/2 of the segment, with w as a free-attention half-window and σ as the decay scale [§ Temporal Cross-Attention Routing].
Boundary-aware decay: σ is set in closed form from exp(−(L−w)²/(2σ²)) = ε so the attention prior reaches a small user-defined value ε at the endpoint distance L, eliminating one hyperparameter [§ Boundary-Aware Decay].
Because the penalty acts multiplicatively on unnormalized attention scores as exp(−C(i,j)), the prior is identically 1 inside the free-attention window and decays smoothly toward 0 outside it, preventing destructive interference at segment joins [§ Boundary-Aware Decay].
The method is fully inference-time: no fine-tuning, no extra parameters, no auxiliary networks — only a per-cross-attention-head logit bias keyed on (latent frame index, prompt-segment membership) [§ Temporal Cross-Attention Routing].

Method

The setup: a pretrained T2V diffusion model whose cross-attention attends video query tokens (indexed by latent frame f(i)) to text key/value tokens. The user supplies a sequence of prompts p_1, …, p_S, each associated with a temporal interval [t_start^s, t_end^s] and contributing a token set K_s in the key/value sequence.

At each cross-attention call, Prompt Relay constructs a penalty tensor C over (query, key) pairs and replaces softmax(QKᵀ/√d) V with softmax(QKᵀ/√d − C) V. The penalty has three parts:

A gating indicator 1[j ∈ K_s] that only penalizes attention into prompt p_s’s tokens.
A ReLU(|f(i) − m_s| − w) term that is zero inside a “free-attention” window of half-width w centered on the segment midpoint m_s = (t_start^s + t_end^s)/2, and grows linearly with distance outside it.
A 1/(2σ²) quadratic envelope that turns the linear distance into a Gaussian tail on the multiplicative attention prior exp(−C).

σ is set analytically from an endpoint constraint: at endpoint distance L, the prior should equal a small chosen ε ∈ (0,1), giving σ = (L − w) / √(2 ln(1/ε)). ε becomes the only user-facing knob — it controls how strictly the model isolates prompts from neighbouring segments, with smaller ε giving sharper handoffs and larger ε giving more cross-segment bleed.

The construction is per-cross-attention-layer and per-head; no architecture or weights change. Because the penalty enters the logits additively before softmax, it composes with standard CFG, sampler choice, and existing attention kernels.

Results

The fetched project page describes the mechanism but does not include quantitative results (no benchmark numbers, no ablations on w, σ, or ε, no comparison baselines surfaced on the landing page). Open for follow-up if the paper or arxiv preprint is found.

Why it’s interesting

This is the cleanest inference-time recipe filed so far for the “different prompts for different parts of a single generated video” problem. It contrasts sharply with the model-side approaches the wiki has been tracking: OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning retrains a unified video DiT with an MLLM reasoning trace to handle compositional prompts, and Co-Director: Agentic Generative Video Storytelling orchestrates a multi-agent pipeline of frozen black-box generators to stitch a multi-scene story from a single high-level prompt — Prompt Relay sits at the opposite extreme, doing the same job with a single per-head logit bias and no agentic loop. It also complements Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video‘s “context-aware keyframe → I2V” UX recipe by giving the underlying generator a primitive for “respect this prompt only during this time window,” which the Vidmento canvas could exploit directly.

Mechanically, the closest sibling on the wiki is OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models — both are training-free interventions inside a frozen video diffusion model that exploit attention to localize a semantic effect (Omnimatte spatially, Prompt Relay temporally). The Gaussian-tail-of-distance formulation is also reminiscent of relative position bias / ALiBi for length extrapolation, applied here to the cross-modal axis. The main open question is empirical — without numbers it is unclear how sharply ε needs to be tuned to avoid destructive interference for fast cuts versus how much cross-segment leakage helps with smooth transitions; the paper presents a principled boundary condition but not an evaluation curve.