AnyUp: Universal Feature Upsampling

TL;DR

AnyUp is a learned vision-feature upsampler that is trained once and can be applied at inference to features from any image encoder at any spatial resolution and channel dimensionality — no per-backbone retraining. It adopts the attention-based upsampling formulation of JAFAR/LoftUp but replaces the dimensionality-specific input projection with a feature-agnostic convolution layer (a per-channel learned filter basis with softmax-aggregation across channels), adds local window attention in place of global attention to suppress distant-patch outliers, and trains via random-crop supervision with self- and input-consistency regularization. The result is SOTA on linear-probe semantic segmentation (COCO-Stuff, PASCAL-VOC, ADE20k), depth and surface-normal estimation (Probe3D / NYUv2), and feature-space preservation, while a single DINOv2-ViT-S-trained AnyUp transfers cleanly to SigLIP 2 and DINOv3 features at test time. ICLR ‘26 Oral; code and weights at github.com/wimmerth/anyup.

Key claims

A single per-channel convolutional filter basis followed by softmax-over-bases and average-over-input-channels yields a feature-agnostic layer whose output dimensionality is fixed regardless of input channel count, enabling one model to consume DINOv2 (384-d), SigLIP 2, DINOv3, CLIP, and ResNet features without retraining [§4.1, Eq. 1].
Restricting JAFAR-style global cross-attention to a local window around each query pixel both improves quality (the global form sometimes attends to “vastly unrelated and distant image areas”) and reduces compute [§4.2].
Training on random local crops — supervising the upsampler against features computed only on a smaller crop, while running the upsampler on the full feature map — is faster and more memory-efficient than JAFAR’s full-image low-res supervision and than LoftUp’s EMA-target high-res supervision [§4.3.1].
Self-consistency regularization plus input-consistency regularization (downsampling the prediction and comparing to the input features) improve locality preservation, which matters for sub-patch-level tasks like surface normals [§4.3.2].
On COCO-Stuff / PASCAL-VOC / ADE20k linear-probe semantic segmentation, AnyUp beats Bilinear, FeatUp, LoftUp, and JAFAR — 62.16 / — / 42.43 mIoU vs JAFAR’s 61.82 / 84.36 / — and Bilinear’s 59.48 / 81.43 / 40.54 [Table 2].
On Probe3D depth + surface normals at 224×224, AnyUp posts the best RMSE in every column it competes in — 31.17 RMSE on normals vs LoftUp 33.94 and Bilinear 32.70; absolute-depth RMSE 0.4755 vs JAFAR 0.4906 and Bilinear 0.4925 [Table 3].
On feature-space preservation (DINOv2 linear probes pre-trained on the original low-res features applied directly to upsampled high-res features), AnyUp gets 40.83 mIoU on ADE20k vs LoftUp’s 4.27 — LoftUp’s affinity-matrix-based training destroys the feature space [Table 5].
An AnyUp trained on DINOv2 ViT-S features, when tested on SigLIP 2 features, matches or outperforms a LoftUp / JAFAR specifically trained on SigLIP 2 (e.g. 54.45 mIoU vs LoftUp-on-SigLIP-2’s 40.73) [Table 6].
A DINOv2-ViT-S–trained AnyUp tested on DINOv3 ViT-S+ achieves 62.96 mIoU, matching the DINOv3-trained version (62.99) — feature-agnosticism transfers to genuinely unseen larger backbones [Table 6].
Ablations: removing window attention, the crop-based data sampling, the input-consistency regularizer, or all regularization each costs measurable mIoU and RMSE [Table 7].

Method

AnyUp inherits JAFAR’s overall shape: image and low-res feature map are processed in parallel, queries are formed from image features at the target high resolution, keys are formed from (downsampled image features + low-res feature map), values are the raw low-res patches, and the upsampled feature at every pixel is an attention-weighted combination of low-res patch features (no value projection — this preserves the input feature space). Two structural changes turn this into an encoder-agnostic model. First, the input feature map passes through a feature-agnostic convolutional layer before being mixed into the keys: each of C_in channels is convolved independently with a shared learned filter basis of K filters; the per-filter activations are softmaxed across the basis dimension and averaged over input channels, yielding a fixed K-channel output regardless of C_in. Second, the attention is restricted to a window around each query pixel, which both speeds inference and prevents global-attention “outlier” matches between visually similar but semantically unrelated regions.

Training samples a random local crop of the high-res image, downsamples it to the encoder’s native low-res resolution, and uses the crop’s locally-extracted features as ground truth for the corresponding region of the full-image upsampled output. Loss is an MSE between predicted and target features plus a self-consistency regularizer (detailed in App. B.1) and an input-consistency regularizer (downsample the prediction, compare to the input feature map). Training is on ImageNet only; the DINOv2 ViT-S backbone is the default for reported numbers, but the released checkpoint is the same one used in the cross-backbone evaluation.

Results

Linear-probe semantic segmentation (DINOv2 ViT-S, 448² input, 14× upsampling) — COCO-Stuff: 62.16 mIoU / 81.37 Acc (vs JAFAR 61.82 / 81.07, Bilinear 59.48 / 79.32); ADE20k: 42.43 mIoU / 75.85 Acc (vs LoftUp 42.02, Bilinear 40.54 / 74.12); PASCAL-VOC: AnyUp top mIoU in Table 2 [Table 2]. Probe3D depth + normals at 224²: surface-normal RMSE 31.17 (best); δ-accuracies 0.29 / 0.57 / 0.69 (best across thresholds); absolute-depth RMSE 0.4755 and δ 0.8216; relative-depth RMSE 0.3378 and δ 0.9233 — best on every column [Table 3]. Varying-resolution evaluation: AnyUp wins at 16→112, 32→224, and 32→112 on COCO segmentation and on NYUv2-style depth, slightly behind LoftUp on one 16-pixel input case [Table 4]. Feature-space preservation: AnyUp 40.83 ADE20k mIoU / 74.94 Acc using frozen DINOv2 linear probes, vs LoftUp 4.27 / 46.58 (collapse) and JAFAR 39.06 / — [Table 5]. Cross-backbone generalization: trained on DINOv2 ViT-S, tested on SigLIP 2 — 54.45 mIoU vs LoftUp-on-SigLIP-2 40.73 [Table 6]; tested on DINOv3 ViT-S+ — 62.96 mIoU vs DINOv3-trained AnyUp 62.99 (parity) [Table 6]. Ablations confirm all four design choices (feature-agnostic layer, window attention, crop sampling, input-consistency) contribute [Table 7].

Why it’s interesting

AnyUp is the natural follow-up to JAFAR: Jack up Any Feature at Any Resolution — it inherits JAFAR’s resolution-agnostic cross-attention upsampler and addresses JAFAR’s explicit “limitation” (one upsampler per backbone) with a feature-agnostic input layer and a smarter training-time crop strategy. That combination matters for Luma in two ways: (a) anywhere a frozen vision foundation model is used as a representation target — REPA-style alignment, semantic autoencoders, MET3R-style multi-view consistency scoring (Asim et al., a co-author paper, is cited) — the encoder choice can now change without re-paying upsampler training cost; (b) AnyUp’s “no value projection, attention weights interpolate raw low-res features” design is the same architectural primitive used in conditioning DiTs on coarse spatial signals, so the feature-agnostic-layer trick is worth knowing about even outside of the upsampling task. The companion direction in V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning — retrain the encoder to produce dense features — remains the alternative when you control the backbone; AnyUp is the right tool when you don’t.