SONIC: Spectral Optimization of Noise for Inpainting with Consistency

TL;DR

SONIC is a training-free inpainting method that ignores the denoiser’s sampling trajectory and instead optimizes the initial seed noise of an off-the-shelf flow / diffusion text-to-image model so that — when denoised — it reconstructs the unmasked pixels of the input image. Two ideas make this tractable: (i) the denoiser is treated as a single linear map (a stop-gradient linearization of the flow trajectory), so no backprop through the denoiser is needed; and (ii) the optimization is performed in the spectral (Fourier) domain of the latent noise to level convergence rates across spatial frequencies. With ~40 Adam steps on the initial latent — then a standard BLD inpainting pass on top — SONIC outperforms FLAIR, FlowChef, FlowDPS, BLD-SD3.5, and the trained inpainter BrushNet on SSIM, LPIPS, and FID across FFHQ, DIV2K, and BrushBench at roughly the runtime of FLAIR-50.

Key claims

A single initial seed noise largely determines the global structure of a flow-model generation; for inpainting this means a uniformly-random seed will produce inpainted content whose structure is inconsistent with the unmasked image regardless of how the denoising trajectory is guided [§1, Fig. 2].
The denoising trajectory of modern flow models is approximately linear, so the relationship between initial noise and final image can be approximated by x ≈ A·z + b with A, b held fixed (stop-gradient) during the optimization step — eliminating the need to back-propagate through the denoiser [§3.2, Eq. 2].
Direct spatial-domain optimization of the seed noise diverges from the Gaussian initial-noise manifold (over-saturation, frame-level artifacts); optimizing the spectral representation ẑ = F(z) instead converges stably because different latent-space frequencies are rebalanced explicitly [§3.3, Fig. 3].
Gradient updates must be masked to the observed (unmasked) latent pixels — otherwise updates “seep” into masked regions and corrupt the seed-noise manifold [§3.4, Fig. 6].
The entire optimization can be done in latent space using nearest-neighbor fills inside the mask before VAE encoding; ablation shows this matches the upper-bound of running the optimization against the ground-truth full image (SSIM 0.857 / LPIPS 0.121 vs GT-upper-bound 0.856 / 0.121 on FFHQ) [§3.4, Table 4].
On FFHQ with the FLAIR protocol, SONIC reaches SSIM 0.857, LPIPS 0.121, FID 12.88, HPS v2 0.233 — best in class on every structure / perceptual metric vs. FLAIR (0.823 / 0.266 / 18.98), BrushNet (0.759 / 0.237 / 17.95), BLD-SD3.5 (0.824 / 0.180 / 25.84) at matched 400 NFEs [Table 1].
On DIV2K SONIC reaches SSIM 0.803, LPIPS 0.115, FID 17.70 — best in LPIPS/FID against the same baselines [Table 2].
On BrushBench SONIC reaches SSIM 0.861, LPIPS 0.153, FID 48.07 — best on LPIPS/FID even against BrushNet, which is the trained inpainter targeted at this dataset [Table 3].
Runtime is roughly one minute per image on an RTX 5090, comparable to FLAIR-50 NFE despite SONIC running 400 NFE-equivalent — the optimization steps are cheap because no denoiser backprop is involved [§4.1].
Concurrent work (Sec. 2 “concurrent”) proposes a similar seed-noise optimization for image editing but optimizes in the spatial domain and requires an initial inversion step; SONIC’s ablation shows spatial optimization underperforms spectral on FFHQ (FID 14.58 vs 12.88) [§4.3, Table 4].

Method

The problem is framed as a least-squares fit on the initial latent noise z: find z* that minimizes ‖M·(G(z) − y)‖² where G is the T-step flow denoiser, M is the binary mask, and y is the masked input. Naively this requires unrolling G (e.g. Stable Diffusion 3.5 cannot back-prop through more than one step on a 5090). SONIC’s first move is to linearize: treat one full denoising pass as G(z) ≈ A·z + b with A, b recomputed each outer step but stop-gradient’d within it — so each outer step does one full denoise forward, derives (A, b), and applies a gradient step that is effectively a least-squares projection.

The second move is to optimize the spectral coefficients ẑ = F(z) (2-D Fourier of the latent), not the spatial latent. Different spatial frequencies in the latent converge at different rates under the linearized step, and optimizing spatially produces flickering solutions that drift off the Gaussian-noise manifold; optimizing spectrally levels these rates. The Fourier coefficients are trainable parameters; spatial-domain z is obtained by inverse FFT each step. Gradient updates are masked to the unmasked-region pixels so the masked region keeps its original Gaussian sample (otherwise optimization corrupts the seed manifold). Inside the mask the encoded image is filled by nearest-neighbor color from the unmasked border before VAE encoding — empirically as good as using the ground-truth image during optimization.

After ~25–40 Adam steps (lr=3.0, default β₁/β₂), the optimized z* is fed to a standard BLD inpainting pipeline (StableDiffusion 3.5, CFG=2.0) which performs the actual inpainting. SONIC’s contribution is entirely on the seed-noise side; the denoising procedure is unchanged.

Results

FFHQ, FLAIR protocol [Table 1]: SONIC vs second-best — SSIM 0.857 vs 0.829 (FLAIR-400) (+0.028), LPIPS 0.121 vs 0.180 (BLD-SD3.5) (−33%), FID 12.88 vs 17.95 (BrushNet) (−28%), IR 0.114 vs 0.100 (BrushNet) (+0.014). HPS v2 0.233 best. PSNR 22.56, third-best — the paper argues PSNR alone is misleading for inpainting.
DIV2K, FLAIR protocol [Table 2]: SSIM 0.803 (best, +0.014 over BLD-SD3.5 0.789), LPIPS 0.115 (best, −20% vs BLD-SD3.5 0.144), FID 17.70 (best, −22% vs FLAIR-400 22.15). BrushNet’s mask-trained inpainter scores particularly poorly here (SSIM 0.575, FID 29.00), confirming the paper’s mask-generalization-failure claim.
BrushBench (BrushNet’s own dataset) [Table 3]: SONIC SSIM 0.861, LPIPS 0.153, FID 48.07 — best on all three even though BrushNet is trained on this exact benchmark. CLIP, IR, AS are 2nd/3rd. The result demonstrates that an off-the-shelf SD3.5 + 40-step noise optimization can outperform a model specifically trained for the task.
Ablations [Table 4]: Spatial-domain optimization SSIM 0.853 / LPIPS 0.127 / FID 14.58 — substantially worse than spectral 0.857 / 0.121 / 12.88. Removing gradient masking: catastrophic — FID 33.27, IR drops to −0.389. Optimizing against ground-truth image (oracle upper bound): SSIM 0.856 / LPIPS 0.121 / FID 12.77 — within noise of the nearest-neighbor-fill variant, confirming the simple fill strategy is sufficient.
Compute: ~1 minute per image on RTX 5090, comparable to FLAIR at 50 NFE despite the 400-NFE-equivalent setting, because the linearization sidesteps denoiser backprop.

Why it’s interesting

SONIC is the third independent datapoint in a small but growing “initial noise is a control surface” cluster on the wiki, and the one with the cleanest mechanical contribution. The other two: Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models amortizes per-prompt noise optimization into a frozen LoRA (training-time move) so a distilled generator’s output distribution tilts toward a reward; Astropulse: sending special noise patterns alongside images to manipulate FLUX Kontext exploits non-Gaussian noise as a side-channel control surface on FLUX.1 Kontext at inference time (no optimization). SONIC sits exactly in between: it does per-sample inference-time noise optimization but avoids amortization because the linearization makes the per-sample optimization cheap enough to run at deploy time. The three together make the case that the “initial noise slot” is now a first-class control surface alongside text prompts and reference images — a claim Classifier-Free Guidance variants has been collecting from multiple angles (Ostris’s training-side noise-augmentation, NAG’s attention-space negative guidance, ALG’s frequency-scheduled conditioning).

The linearization trick — A·z + b with stop-gradient on A, b — is also a small but reusable building block. The same trick should in principle work for super-resolution, deblurring, and any other linear-measurement inverse problem on flow models (the paper flags this as future work). The spectral-domain optimization is the more surprising piece: it implies that “the gradient is the wrong shape” on the spatial latent for noise-side optimization, and that a fixed spectral basis is enough to fix it without learning anything. This echoes Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation‘s observation that different DiT tokens have different convergence rates per timestep, and contrasts cleanly with Back to Basics: Let Denoising Generative Models Denoise‘s argument that the denoiser itself is what needs frequency-aware design.

Practically, SONIC is the first “off-the-shelf SD3.5 outperforms a trained inpainter on the trained inpainter’s own benchmark” result the wiki has filed, and it costs only ~40 optimization steps on the initial latent.