JAFAR: Jack up Any Feature at Any Resolution

TL;DR

JAFAR is a lightweight cross-attention-based feature upsampler that takes the coarse spatial feature map from any frozen vision encoder (CLIP, SigLIP, DINOv2, RADIO, etc.) and produces a high-resolution feature map at arbitrary target resolution. The key idea is asymmetric query/key construction: queries carry high-resolution low-level image cues (color, texture), while keys are downsampled image features modulated by the encoder’s semantic features via SFT, producing semantically-enriched but spatially-aligned keys. Trained task-agnostically with only multi-resolution self-supervision (no high-res labels, upsampling factors ≤×3 during training), JAFAR generalizes to much higher ratios at test time and beats LiFT, FeatUp, CARAFE, SAPA, ReSFU, and DySample on linear-probe semantic segmentation (ADE20K, COCO-Stuff, VOC, Cityscapes), depth (RMSE / δ₁.₂₅), CAM faithfulness, zero-shot open-vocabulary segmentation, and BeV segmentation.

Key claims

The query/key asymmetry — queries from low-level image features, keys built from the same image features but modulated by deep encoder semantics via SFT — is what makes cross-resolution attention work; a Linear-Projection baseline (keys from the encoder feature directly) and a concatenation variant both lose to SFT modulation [§3.1, Table 5].
Training at low resolutions and low upsampling ratios (≤×3 from 224² inputs) is sufficient to generalize to much higher inference-time ratios (e.g. ×16) without high-resolution supervision, thanks to the resolution-agnostic cross-attention design [§3.2, Tabs. 1–2].
Multi-head attention helps up to 4 heads (more robust averaged attention kernels); beyond that the per-head dimension becomes too low and quality degrades [§4.4, Table 5].
The value projection is removed entirely — attention weights directly interpolate the low-resolution encoder features — to preserve the original feature content and keep the design resolution-agnostic [§3.1, Similarity Based Upsampling].
RoPE positional embeddings on queries and keys are essential — without them spatial grounding disappears and the model generalizes poorly to unseen resolutions [§3.1].
JAFAR consistently outperforms LiFT and FeatUp on linear-probe semantic segmentation: average +mIoU gain over the next-best method across COCO-Stuff/ADE20K/VOC/Cityscapes, with peak +1.50 mIoU (+8.39%) on Cityscapes versus FeatUp [Table 1].
On Class Activation Map faithfulness JAFAR achieves the highest ADCC aggregate score, beating the second-best method by 8 points (a 12.5% relative improvement) [Table 2].
On zero-shot open-vocabulary segmentation with CLIP-ViT-B/16, JAFAR is the highest performer; FeatUp is reported as the next-best task-agnostic baseline [§4.3.4].
Plugging JAFAR into BeV-segmentation pipelines (SimpleBev, PointBeV, BevFormer) with a frozen DINOv2 backbone improves mIoU by up to several points regardless of the BeV architecture [§4.3.5].
Training cost is modest: a single A100, 100k steps, AdamW at LR 1e-4, batch size 4, on ImageNet train; only the upsampler is trained, encoder is frozen [§4.1].

Method

JAFAR sits as a post-hoc module on top of a frozen vision encoder. Given a high-res image I and the encoder’s coarse feature map F_lr, JAFAR (i) projects I through a lightweight image encoder into intermediate representation Z, (ii) adds RoPE positional embeddings, (iii) builds queries Q by passing Z through a small query encoder followed by adaptive average pooling to the target resolution, (iv) builds preliminary keys K_pre by encoding Z and downsampling to the encoder’s spatial resolution, (v) applies Spatial Feature Transform modulation: K = γ(F_lr) ⊙ K_pre + β(F_lr) where γ, β are linear projections from the encoder feature map. The output is a cross-attention map between Q and K (no value projection, no learned V), which interpolates the original low-resolution F_lr to produce the high-res output F_hr. Training uses multi-resolution views of the same image: downsample I by a random factor s ∈ [1, 3], encode both, train JAFAR to upsample the low-res features to match the high-res features under a combined cosine + L2 loss. Despite training only at moderate ratios, inference handles ×16+ upsampling cleanly because the design is resolution-agnostic (no fixed-scale CNN, RoPE-based positional alignment).

Results

Headline numbers on linear-probe semantic segmentation with DINOv2 ViT-S/14 at 224² input: JAFAR averages +mIoU over next-best across COCO-Stuff (27 cls), ADE20K (150 cls), Pascal VOC (21 cls), Cityscapes (27 cls). vs FeatUp specifically: +1.50 mIoU on Cityscapes (+8.39%) [Table 1]. On depth with Depth Anything V2 pseudo-labels on COCO, JAFAR is second among baselines, beating FeatUp, LiFT, CARAFE, SAPA, DySample — only ReSFU edges it [§4.3.2]. CAM faithfulness: highest ADCC, +8 points (+12.5%) over the second best [Table 2]. Zero-shot open-vocabulary segmentation with CLIP-ViT-B/16: highest mIoU and accuracy on both Pascal VOC and ADE20K vs FeatUp [§4.3.4]. BeV segmentation: consistent mIoU gains across three architectures when using JAFAR vs no-upsampler or other upsamplers [§4.3.5]. Qualitative PCA visualizations show JAFAR is the only task-agnostic method that suppresses positional-embedding artifacts in upsampled feature maps [Fig. 3, 5]. Limitation called out by the authors: a separate upsampler must be trained per backbone — backbone-independence at inference time is left to future work [§5].

Why it’s interesting

JAFAR addresses the same problem V-JEPA 2.1 (V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning) attacks from the other side: foundation vision encoders produce semantically rich but spatially coarse features (×14–16 downsampled), which hurts dense prediction. V-JEPA 2.1 retrains the encoder with a context-loss objective so its features come out spatially coherent; JAFAR keeps the encoder frozen and learns a tiny upsampler on top. The two solutions answer different deployment constraints — JAFAR is a drop-in module for production CLIP/DINOv2 stacks; V-JEPA 2.1 is the option when you can afford to re-pretrain. For Luma’s semantic-AE direction (Terrance’s framing in the share), JAFAR is the cheap path to upsample frozen CLIP/DINOv2 features back to native image resolution before reconstruction or alignment losses — relevant because the autoencoder-for-diffusion literature (Autoencoders for Diffusion: A Deep Dive, What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion) increasingly aligns latents to VFM features, where coarse alignment loses fine spatial detail. The architectural insight (asymmetric Q/K with shared low-level provenance + SFT modulation) is also reusable wherever cross-resolution attention is needed; it’s the same problem-shape as upsampling DiT activations or aligning a coarse latent with a high-res spatial conditioning signal.