Sharp Monocular View Synthesis in Less Than a Second

TL;DR

SHARP (Single-image High-Accuracy Real-time Parallax) is Apple’s feed-forward, regression-only network that maps one photograph to a ~1.2M-Gaussian 3DGS scene in under a second on an A100, suitable for real-time AR/VR rendering of nearby views with metric absolute scale. The architecture is a Depth Pro backbone (low-res ViT + patch ViT) feeding a DPT-style 2-layer depth decoder, a learned depth-adjustment U-Net that resolves monocular depth ambiguity during training, a Gaussian initializer that unprojects the depth into base Gaussians in normalized space, and a DPT-style Gaussian decoder that emits per-pixel deltas for position/scale/rotation/color/opacity. End-to-end training is a two-stage curriculum: 100K steps on synthetic data with ground-truth depth, then 60K steps of self-supervised finetuning that swaps the input view and a model-rendered pseudo-novel view. SHARP cuts LPIPS by 25–34% and DISTS by 21–43% against the strongest prior method (Gen3C) on six zero-shot datasets, while running ~1000× faster.

Key claims

A purely regression-based feed-forward network — no diffusion, no per-scene optimization — can produce high-resolution 3DGS scenes from a single photo at SOTA visual fidelity for nearby views [§1, §5].
Freezing a pretrained monodepth model degrades view-synthesis fidelity, particularly on transparent/reflective surfaces; SHARP unfreezes the low-resolution Depth Pro image encoder and the full depth decoder during training so depth adapts to view-synthesis losses [§3.1].
Monocular depth ambiguity (the network can resolve uncertainty by predicting the mean of possible depths, which is fine for depth metrics but causes artifacts under reprojection) is resolved at training time by a small (2M-param) C-VAE-style depth-adjustment U-Net that takes both predicted disparity and GT disparity and emits a scale map, replaced with the identity at inference [§3.1, §3.2, §3.4].
A two-layer depth map (duplicated last conv of the DPT decoder), where layer 1 is the visible surface and layer 2 absorbs occluded regions / view-dependent effects, is used to initialize base Gaussians; the Gaussian decoder then emits per-attribute deltas composed via attribute-specific activations [§3.1, Eq. 3.1–3.2].
The Gaussians are initialized in normalized space without using the input image’s intrinsics — the source-camera transform is folded into the target-view projection matrix, so the network does not have to adapt its predictions to the field of view [§3.1].
A two-stage curriculum — synthetic-data pretraining then self-supervised finetuning that uses the network’s own pseudo-novel views as the input and the real photo as the supervision target — adapts the model to real images without requiring stereo pairs (cf. AdaMPI’s warp-back) [§3.3].
Loss configuration matters: an L1 + perceptual + alpha-BCE on rendered views, plus regularizers for floaters (large-disparity-gradient Gaussians), offset magnitude, screen-space projected variance, and total-variation on the second depth layer, together with an MAE + multiscale-TV bottleneck on the depth-adjustment map [§3.4, Eq. 3.3–3.11].
Zero-shot LPIPS / DISTS gains over the prior SOTA Gen3C across six metric-pose datasets (Middlebury, Booster, ScanNet++, WildRGBD, Tanks and Temples, ETH3D): 25–34% LPIPS reduction and 21–43% DISTS reduction, with three-orders-of-magnitude faster synthesis [Table 1, §1].
Once produced, the 3D representation renders at >100 FPS on a standard GPU, supporting natural posture-shift AR/VR viewing of static photo collections — the explicit target application [§1, §5].

Method

The pipeline is a single feed-forward pass through five modules (Fig. 3, ~340M trainable / 702M total parameters). A pretrained Depth Pro encoder — two ViTs, one on the downscaled image and one on patches — emits four feature maps at multiple resolutions, from which a DPT-style depth decoder outputs a two-channel depth map (visible surface + occluded/view-dependent layer). During training a small U-Net depth-adjustment network takes the predicted inverse depth and ground-truth disparity and emits a scale map that resolves monocular ambiguity; it is replaced by the identity at inference, which is what makes the test-time pipeline a pure single-pass regressor. The adjusted two-layer depth and the input image are unprojected to initialize ~1.2M base Gaussians in a normalized camera frame (scale ∝ depth, color from the downsampled image, rotation = identity, opacity = sigmoid⁻¹(0.5)). A second DPT-style Gaussian decoder (≈8M params, trained from scratch) emits per-attribute deltas for position / scale / rotation / color / opacity, which a Gaussian composer combines with the base Gaussians via attribute-specific activations. Rendering uses an in-house differentiable splatter where the source-view transform is folded into the target projection matrix so the network can predict in a canonical frame. The training curriculum is synthetic-data first (perfect input + novel view, 100K steps on 128 A100s) then a self-supervised finetune (60K steps on 32 A100s) in which the model’s own rendered pseudo-novel view becomes the input and the real photograph becomes the supervision target — adapting to real images without paired stereo data.

Results

On six zero-shot, metric-pose datasets, SHARP achieves the best LPIPS and DISTS on every dataset against Flash3D, TMPI, LVSM, Stable Virtual Camera, ViewCrafter, and Gen3C [Table 1]. Headline gains vs Gen3C: LPIPS 0.358 vs 0.545 on Middlebury (-34%), 0.270 vs 0.384 on Booster (-30%), 0.154 vs 0.227 on ScanNet++ (-32%), 0.190 vs 0.285 on WildRGBD (-33%), 0.421 vs 0.566 on Tanks-and-Temples (-26%), 0.554 vs 0.734 on ETH3D (-25%) [Table 1]. DISTS gains in the same direction (21–43%). Synthesis runs in under a second on an A100 (the Apple researcher cited in the Slack pointer measured ~5–10s on an M1 Max with the released checkpoint), three orders of magnitude faster than the diffusion baselines (Gen3C/SVC/ViewCrafter), and the resulting 3DGS renders at 100 FPS for nearby-view AR/VR use. Ablations (in supplement) confirm: perceptual loss is what produces sharpness, regularizers are what suppress artifacts, the depth-adjustment module materially improves detail, and SSFT improves crispness on real photos. PSNR/SSIM are reported but downweighted because they are sensitive to 1%-scale shifts that don’t correspond to perceptual quality.

Why it’s interesting

SHARP is a counter-datapoint to the diffusion-dominated single-image-to-3D direction the wiki has been accumulating in 4d-scene-generation: FlashWorld: High-quality 3D Scene Generation within Seconds (cross-mode DMD2-distilled MV-DiT + 3D-Gaussian student, 9s), WonderJourney: Going from Anywhere to Everywhere (training-free LLM + SD-inpaint + MiDaS depth + VLM validator), and Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation (fine-tuned video DiT + 3D point-cloud world cache) all sit on the static-3D-from-single-image axis but route through generative priors. SHARP shows that with a strong monodepth backbone (Depth Pro) and careful loss engineering, regression still wins on near-view fidelity at three orders of magnitude lower latency — at the cost of giving up large-baseline / faraway-view synthesis that the diffusion methods get for free. The two-layer depth + Gaussian-initializer + delta-decoder factoring is also a cleaner version of the The flavor of the bitter lesson for computer vision “explicit 3D as human-machine interface” position than FlashWorld’s MV/3D dual mode: SHARP commits fully to the explicit-representation branch and validates it on perceptual quality. The depth-adjustment C-VAE trick (information-bottleneck the residual that resolves monocular ambiguity at training time, identity at inference) is also a reusable recipe for any task where you want a frozen-at-inference geometry predictor that was nonetheless trained against a downstream rendering loss.