Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners

TL;DR

LILA (Linear In-Context Learning) from Google + TU Munich trains an encoder-decoder to produce pixel-level feature maps from a single image, bootstrapped from a frozen DINOv2 backbone with a DPT decoder, using only uncurated videos and noisy off-the-shelf cue maps (depth from DepthAnythingV2, optical flow from SEA-RAFT, plus PAMR-refined encoder self-distillation). The novelty is the training objective: for adjacent frame pair (t, t+τ), solve a closed-form ridge regression that finds the optimal linear projection from the decoder’s feature map at frame t to the concatenated cue map at frame t, then apply that same projection to the feature map at frame t+τ and require it to match the cue map at frame t+τ. This in-context consistency suppresses frame-specific noise that standard ERM distillation would absorb. The resulting features beat FlowFeat, FeatUp, LoftUp, and bare DINOv2 across DAVIS-2017 VOS, NYUv2 surface normals, and COCO-Stuff semantic segmentation by 3–9 points without using any mask supervision.

Key claims

Standard ERM distillation of the same cue maps (depth + flow + PAMR-refined encoder features) into a DPT decoder is worse than LILA’s cross-frame linear-consistency objective, because ERM absorbs the noise in the cue maps in expectation while LILA’s projection-must-transfer constraint requires temporally invariant features and suppresses frame-specific noise [Table 5 row A, §4.4 discussion].
The optimal linear projection in Eq. 2 has a closed-form ridge-regression solution at size C×C (feature dim, ~32) rather than HW×C (number of pixels), making the in-context objective cheap [§3, Eq. 2].
Cue diversity is synergistic: removing all geometric/motion cues (self-distillation only) drops DAVIS k-NN from 73.9 to 68.6; adding only depth recovers most of it; adding only flow recovers most of it; combining all three is strictly best [Table 4].
Cropping context and query separately is necessary — without it DAVIS k-NN drops to 72.4 because the network sees identical inputs and the consistency constraint becomes trivial [Table 5 row C].
LILA + DINOv2-S14 (trained on YouTube-VOS, no mask supervision) beats LoftUp (which uses SAM/SA1B mask supervision) on DAVIS-2017 linear-probe VOS [Table 1].
LILA + DINOv2-L14 (trained on Kinetics-700) reaches NYUv2 surface-normal RMSE 0.182, ADE20K-comparable semantic-segmentation mIoU on COCO-Stuff of 63.3, and DAVIS-2017 VOS gains over FlowFeat across both linear probing and local k-NN [Tables 1, 2].
Zero-shot semantic-segmentation probing on COCO-Stuff (CLIP-initialized probe, train on seen classes, test on seen+unseen) shows LILA improves unseen-class mIoU over the DINOv2 baseline, indicating the dense features preserve CLIP-aligned semantic structure rather than degrading it [Table 3].
The temporal gap τ trades off between context-query similarity (small τ → trivial task, weak features) and cue predictability (large τ → noisier flow/depth correspondence); accuracy degrades gracefully for large τ [Fig. 7].
PAMR refinement of encoder features in the self-distillation target is critical — removing it drops DAVIS k-NN by 2.0 points [Table 5 row B].
LILA generalizes out-of-domain to chest X-rays but fails on aerial scenes where the off-the-shelf depth/flow networks produce nonsense — the framework inherits its training-data assumptions from those auxiliary models [Fig. 6b, §4.4 limitations].

Method

The setup is a frozen DINOv2 ViT encoder + trainable DPT decoder with four skip connections from intermediate ViT blocks. For each training step, sample adjacent frames (t, t+τ) at varying temporal gaps from YouTube-VOS or Kinetics-700, compute forward and backward optical flow with SEA-RAFT and per-frame depth with DepthAnythingV2, and compute PAMR-refined DINOv2 features for each frame as a third “cue” modality. Concatenate flow + depth + PAMR features along channels to form the cue maps C_t and C_{t+τ}.

Independently random-crop the two frames (aspect-preserving) and pass both crops through the encoder-decoder to produce pixel-resolution feature maps F_t and F_{t+τ}. Solve a closed-form ridge regression W* = argmin ‖W F_t − C_t‖² + λ‖W‖² (Eq. 2, C×C system, ~32-d feature dim makes this trivial). Apply the same W* to F_{t+τ} and minimize ‖W* F_{t+τ} − C_{t+τ}‖² plus an edge-aware gradient-matching loss that rewards alignment of feature/cue discontinuities. At inference time, the depth and flow networks are discarded; LILA consumes a single image and emits a pixel-resolution feature map.

Results

DAVIS-2017 VOS linear probing with DINOv2-S14: LILA reaches J&F on par with LoftUp (which uses SAM mask supervision) and beats FlowFeat by ~1 point; with DINOv2-L14 + Kinetics, LILA is the new SOTA across both linear-probe and local k-NN columns [Table 1]. NYUv2 surface-normal estimation: RMSE drops monotonically across S14/B14/L14 backbones, with LILA-L14-Kinetics at 0.182 vs DINOv2-L14 bare backbone at ~0.21; δ-accuracy thresholds (11.25°/22.5°/30°) all improve [Table 2 left]. COCO-Stuff semantic segmentation (27 coarse categories, attention-based probe): LILA improves mIoU by 3.0/3.0/4.6 over the DINOv2 backbones (S14/B14/L14); pAcc improvements are 2.6/3.0/3.3 [Table 2 right]. ADE20K mIoU goes from DINOv2-S14’s number to a 2-point improvement under LILA, and 1.4-point under LILA-B14 [Table 3 left]. Zero-shot COCO-Stuff (CLIP-initialized linear probe): seen, unseen, and harmonic-mean mIoU all improve over the DINOv2 baseline [Table 3 right].

Why it’s interesting

LILA is the third entry in the “dense pixel features from frozen vision foundation models” cluster Luma now has filed, and it complements the other two by a clean axis split. AnyUp: Universal Feature Upsampling (AnyUp) and JAFAR: Jack up Any Feature at Any Resolution (JAFAR) both leave the encoder untouched and learn an attention-based upsampler over the coarse patch grid; V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning (V-JEPA 2.1) goes the other direction and retrains the encoder with a denser objective; LILA sits between them — it freezes the encoder but trains an encoder-decoder head whose objective is itself the novelty, using cheap noisy auxiliary cues from off-the-shelf depth/flow networks instead of human masks. The “ERM distillation is worse than LILA at the same target” result in Table 5 is the cleanest statement to date that how you distill noisy targets matters as much as which targets you pick — cross-frame projection-transfer is a credible new primitive that’s worth knowing about outside the dense-features setting too (e.g. as a regularizer for any task with paired noisy supervision). For Luma’s video pipeline this is concretely useful: every place we’re currently using a DINOv2 / DINOv3 readout as a representation target for REPA-style alignment or as a perceptual loss is operating on patch-resolution features; LILA would give pixel-resolution features at the same parameter cost as adding a small DPT head, and unlike LoftUp/SAM-based upsamplers it doesn’t require mask supervision.