SPDA-SAM: A Self-prompted Depth-Aware Segment Anything Model for Instance Segmentation

TL;DR

SPDA-SAM is a domain-adaptive variant of SAM (ViT-B backbone) for instance segmentation that removes SAM’s reliance on human prompts and injects monocular-depth features. It pairs a Depth-Aware Dual-Path Encoder (two LoRA-finetuned SAM backbones — RGB and depth — fused by a Coarse-to-Fine RGB-D Fusion Module, C2FFM) with a Self-Prompted Decoder that generates semantic + spatial prompts from internal features and an initial coarse mask (Semantic-Spatial Self-Prompt Module, SSSPM). Depth maps are estimated with Depth Anything v2; only LoRA adapters are trained. The system reports state-of-the-art mAP on twelve instance-segmentation datasets spanning salient, urban-driving, underwater, and waste-sorting domains, beating prior SAM-based methods (RSPrompter, USIS-SAM) and Mask2Former across the board.

Key claims

Self-prompting + depth-fusion together beat 22 instance-segmentation baselines (including Mask2Former, USIS-SAM, RSPrompter) across twelve domain-shifted datasets, with mAP gains of +1.7 to +2.4 over the best prior method on salient-object datasets COME15K-E/H, DSIS, SIP [§IV-B1, Table I].
On Cityscapes with the ViT-Base backbone, SPDA-SAM improves mAP by +11.3 over the best SAM-based baseline (USIS-SAM/RSPrompter); with ViT-Huge the gain is +11.1 mAP [§IV-B2, Table II].
The Semantic-Spatial Self-Prompt Module (SSSPM) draws semantic prompts from the image encoder (Transformer Block 2 + Fine Fusion Block 4 features) and spatial prompts from a coarse foreground probability map produced by an initial mask-decoder pass, then fuses them with channel-attention gating [§III-B2].
The Coarse-to-Fine RGB-D Fusion Module applies one coarse fusion at patch-embedding resolution (depth → Sigmoid gate × RGB + residual) and four fine-fusion cross-attention blocks at Transformer Blocks 2/5/8/11, modulating RGB features with depth-derived structural cues at multiple semantic levels [§III-A3].
Both encoder paths are initialized from SAM ViT-B weights and fine-tuned via LoRA only — the SAM backbone weights are frozen — so domain adaptation requires only a small target-domain dataset [§III-A1–A2, §IV-A3].
Training uses a three-term loss: RPN loss + cross-entropy on the coarse mask + cross-entropy on the refined mask, with all three weights set to 1; the mask decoder is invoked twice per image (coarse → SSSPM → refined) [§III-C].
The pipeline depends on a pretrained monocular depth estimator (Depth Anything v2) at inference time — depth is predicted, not sensed [§IV-A1].

Method

SPDA-SAM has two main pieces. The Depth-Aware Dual-Path Encoder runs two parallel ViT-B encoders initialized from SAM — one on the RGB image, one on a Depth-Anything-v2 depth map of the same image — both adapted via LoRA only (the SAM backbone weights stay frozen). Their features are merged by a Coarse-to-Fine RGB-D Fusion Module (C2FFM): a single Coarse Fusion Block at patch-embedding resolution downsamples → dilated-conv → ReLU → 1×1 → upsample on the depth features, passes them through a Sigmoid to form a gate, multiplies element-wise into the RGB features, and adds a residual; then four Fine Fusion Blocks applied at Transformer Blocks 2/5/8/11 do bidirectional cross-attention between RGB and depth feature sequences. The Self-Prompted Decoder has a Mask Decoder applied twice. First pass: an RPN proposes initial geometric prompts, the mask decoder produces a coarse foreground probability map. Second pass: the Semantic-Spatial Self-Prompt Module (SSSPM) generates new prompts — a Feature Alignment Block fuses low-level (Block 2) and high-level (Fine Fusion 4) features into semantic prompts, a Mask Encoder turns the coarse mask into spatial prompts, and a Prompt Fusion Block concatenates them and channel-gates via global-average-pool + Sigmoid attention. The mask decoder re-runs with the refined prompts to emit the final mask. Loss = RPN loss + pixel-CE on the coarse mask + pixel-CE on the refined mask, all weighted 1. ViT-B, AdamW, lr 2e-4, weight-decay 0.05, linear warm-up 50 iters + cosine decay; batch size 2 on 2× 3080 Ti or 1× 4090.

Results

Salient instance segmentation: On COME15K-E, COME15K-H, DSIS, SIP — beats next-best by +1.7/+2.4/+2.0/+2.4 mAP (mAP=68.4/64.4/79.7/81.3); also leads mAP-50 and mAP-75 [§IV-B1, Table I]. Prior best baselines were USIS-SAM and RSPrompter.
Cityscapes (ViT-B): mAP gain +11.3 / mAP-50 +15.4 / mAP-75 +11.6 over best SAM-based baseline; ViT-H: +11.1 / +17.2 / +10.5 [§IV-B2, Table II].
Ablations are referenced but truncated in the available text; the headline claim is that both SSSPM and C2FFM contribute independently to the gain.
The mAP gain is consistent across underwater (LIACI, UIIS, USIS10K/16K, UIIS10K) and terrestrial (KITTI, ZeroWaste) domains, supporting the “drop-in domain adapter for SAM” framing.

Why it’s interesting

The paper’s structure — frozen SAM ViT-B + dual-path encoders (RGB + estimated depth) + LoRA-only finetuning + a self-prompting module that closes SAM’s prompt-dependence loop — is the same recipe template recurring across the wiki’s Parameter-Efficient Finetuning cluster: keep the pretrained backbone frozen, add a small adapter (LoRA) and a domain-specific input/conditioning transform. It’s a direct counterpart to DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models and HDR Video Generation via Latent Alignment with Logarithmic Encoding (LumiVid) on the diffusion side: frozen 14B / LTX video DiT + LoRA + Log-Gamma/LogC3 input transform. On the perception side, SPDA-SAM is interesting next to SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning (the production SAM 3.1 release that focuses on speed via Object Multiplex rather than depth/self-prompt) — SAM 3 already integrates an “SSSPM-like” self-prompting mechanism into its detector, but SPDA-SAM is a far cheaper academic recipe to retarget the older SAM ViT-B/H checkpoints to depth-aware instance segmentation. Useful as a reference for any Luma pipeline that needs occlusion-respecting per-instance masks (Alex’s snake/hand/glass example) without a 3D-from-monocular detour through full scene reconstruction.