Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model

TL;DR

Lotus-2 adapts a pretrained FLUX image diffusion model into a deterministic two-stage estimator for monocular depth and surface-normal prediction. The argument: the stochastic multi-step generative formulation that FLUX was trained under is the wrong objective for dense geometric inference, which needs stable and accurate outputs, not diverse ones. A single-step clean-data core predictor (with a small local continuity module to fix Pack-Unpack grid artifacts) plus an optional constrained multi-step rectified-flow detail sharpener gets SOTA depth and competitive normals on just 59K training images — under 1% of MoGe-2’s data.

Key claims

The pretrained FLUX rectified-flow model carries a usable “world prior” for geometry, but the stochastic multi-step sampling protocol it was trained under introduces structural variance that hurts deterministic dense prediction; a single-step ( $t=1$ ) clean-data parameterization is both more stable and more accurate [Project page §Lotus-2 method].
A lightweight local continuity module ( $\Lambda$ ) wrapped around the diffusion transformer mitigates grid artifacts produced by FLUX’s non-parametric Pack–Unpack patchify operations [Project page §Adaptation protocol of the core predictor].
The full pipeline is two-stage and decoupled: a deterministic single-step core predictor gives globally coherent geometry; an optional detail sharpener applies a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor, with refinement steps $T'_\text{inf} \le 10$ adjustable to taste [Project page §Inference pipeline].
Trained on 59K samples — less than 1% of large-scale depth/normal datasets, and ~0.66% of MoGe-2’s training set — Lotus-2 sets new SOTA on zero-shot affine-invariant depth and is highly competitive on zero-shot surface-normal estimation [Project page §Quantitative comparison; README teaser].

Method

Lotus-2 starts from a pretrained FLUX.1-dev rectified-flow diffusion transformer and rewires its sampling protocol for deterministic regression. The core predictor sets the noise variable equal to the image latent ( $\mathbf{z}_t = \mathbf{z}_1 = \mathbf{z}^x$ ) and predicts clean target geometry in a single forward pass with a clean-data loss — eliminating the stochasticity and multi-step accumulation of the original generative formulation. A local continuity module $\Lambda$ is inserted around the transformer to smooth grid artifacts introduced by FLUX’s Pack–Unpack operations.

The detail sharpener is a second-stage model trained to map “structurally correct but coarse” predictions (produced by the core predictor) to “accurate and fine-grained” predictions via a deterministic multi-step rectified-flow trajectory, all confined to the manifold the core predictor already lives in. At inference, the two stages compose: single-step prediction → up to ~10 deterministic refinement steps. Inference uses clean-data prediction and ≤10 rectified-flow steps for the sharpener.

Results

New SOTA on zero-shot affine-invariant monocular depth across the Marigold benchmark suite, with “best overall performance than all other methods” per the project-page quantitative table [Project page §Quantitative comparison on zero-shot affine-invariant depth estimation].
Highly competitive zero-shot surface-normal results on the DSINE evaluation protocol, with the qualitative win concentrated in fine-grained boundary detail (Fig. 1 teaser, including hard cases like oil paintings and transparent objects) [Project page §Quantitative comparison on zero-shot surface normal estimation].
Training data: 59K samples — less than 1% of large-scale datasets, and ~0.66% of MoGe-2 [Project page §Abstract; README §Lotus-2 intro].
Inference hardware: ≥40GB GPU (validated on A800-SXM4-80GB), with rectified-flow refinement at ≤10 steps [README §Setup].

Why it’s interesting

The wiki has been tracking two adjacent threads that this paper bridges. First, the “diffusion as deterministic estimator” pattern: Lotus-2’s single-step clean-data core predictor sits very close in spirit to The Geometry of Noise: Why Diffusion Models Don't Need Noise Conditioning (which argues noise-conditioning isn’t actually necessary for diffusion) and to the broader “few-step / one-step generative” effort tracked under Diffusion Distillation — except here the goal isn’t faster image generation but stabler regression, a different motivation for the same collapse. Second, it’s a concrete data point for the bet that pretrained image-generation backbones (FLUX in this case) carry transferable geometric priors usable for dense-prediction tasks far from the original training objective — complementing Image Generators are Generalist Vision Learners (Vision Banana) (“image generators are generalist vision learners”) which makes the broader claim, and contrasting with discriminative depth lines like WonderJourney: Going from Anywhere to Everywhere that use MiDaS-style regressors. The 0.66%-of-MoGe-2 data-efficiency result is the headline: it suggests the FLUX prior is doing the heavy lifting and the adaptation recipe matters more than scale here.