Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds

TL;DR

Block-Sparse Featurizers (BSFs) replace the “one direction per concept” atom of a sparse autoencoder with a block of directions that spans a low-dimensional subspace, chosen to match a data-generating process in which activations are a sparse sum of low-dimensional concept manifolds. Three concrete BSF variants — Vanilla (block-TopK on norms), Grassmannian (tied encoder = decoder with orthonormal per-block frames), and Group Lasso (block soft-threshold) — are trained on DINOv3, Stable Diffusion XL, and InceptionV1, and evaluated with a minimum-description-length (MDL) analysis at a task-tolerant reconstruction distortion. BSFs describe activations in fewer bits than TopK SAEs across every dictionary width, per-concept stable rank saturates at 2–4 dimensions (independent of allotted block size up to 128), and the recovered blocks form dense clouds interpretable as concept manifolds — including a single continuous curve manifold that unifies InceptionV1’s classic curve-detector neurons and exposes previously undocumented higher-order Fourier harmonics. In SDXL, the same blocks support manifold steering: walking a 2D grid inside a block sweeps a concept’s factors of variation (pretzel shape, hat brim/crown, coffee milk pattern), providing a continuous knob where SAE atoms provide only a one-dimensional on/off.

Key claims

Under an additive-mixture-of-manifolds data-generating process (each activation = sum of $k$ manifold contributions), the maximum-a-posteriori-matched sparsity prior is block sparsity — a penalty on the number of active blocks, not active coordinates — and the block-norm penalty is the unique frame-invariant reduction of that MAP objective [§2.2, Prop. 1].
On a synthetic superposition-of-manifolds toy (circles, spheres, tori embedded via random orthonormal maps into ℝ⁵¹²) all three BSF variants recover the ground-truth factors at 0.85–0.94 per-block R² against an oracle ceiling of 0.94, while a TopK SAE shatters each multidimensional factor across atoms and collapses to R² ≈ 0.30 [§3.1, Fig. 4-5].
Two recent multi-dimensional-unit featurizers (SMixAE, MFA) fail on the same toy (per-block R² 0.31 and 0.53); a small change — signed codes + block-TopK for SMixAE, additive decoding instead of convex posterior for MFA — lifts recovery to 0.86 and 0.95 respectively, isolating the block-sparsity principle as the causal ingredient [§3.1].
Task-tolerance study on DINOv3: classification and segmentation retain 99% performance down to reconstruction R² = 0.20, but monocular depth estimation requires R² ≥ 0.70 to keep 99% accuracy — grounding the MDL analysis’s chosen distortion bar [Fig. 6 top].
At R² = 0.7 the Grassmannian BSF describes DINOv3 activations in fewer bits than a TopK SAE across every dictionary width tested, with the shortest description at moderate block dimension $g = 8$ – $16$ that eases downward as the dictionary widens; the effect holds for all three BSF variants at multiple noise levels [§3.3, Fig. 7, App. C].
Stable-rank measurement on DINOv3 BSF blocks: average per-block effective rank saturates at 2–4 dimensions and stays there even when block capacity is raised to 128, so blocks don’t use the room they’re given — visual concepts are intrinsically low-dimensional in a way that pre-committing to $g = 1$ (SAEs) or $g = 128$ (naive block width) both miss [§3.4, Fig. 8].
Every InceptionV1 curve-detector neuron (Cammarata et al. 2020) reads off a single continuous curve manifold recovered by a BSF block; the block additionally exposes higher-order Fourier harmonics that were not documented in the original circuits work — 1st harmonic tracks the full 360° orientation, 2nd harmonic tracks 180° flip-symmetric curves, and the harmonic ladder explains prior anomalies noted by Olah et al. (2020) [§ curve detectors, Fig. 9].
In DINOv3 the BSF surfaces coherent multidimensional blocks including a “shadows/lighting” abstraction and an “arch” region whose principal components sweep from arch-bottom (red channel) to arch-crown (green channel) across viaducts and church naves — concepts that no single-direction SAE atom captured [Fig. 1, § DINOv3 examples].
In SDXL the same block structure supports manifold steering: laying a 2D grid over a block and walking from cell to cell smoothly varies a concept’s factors — pretzel shape, hat brim angle / crown shape / orientation, coffee milk pattern / spoon location / cup design — providing a continuous knob unavailable to one-dimensional SAE steering [§ SDXL, Fig. 10-12].

Method

The paper first argues that the right featurizer architecture is a hypothesis about the data-generating process (DGP), then adopts an additive mixture of manifolds DGP (Def. 1, following Bhalla et al. 2026a): an activation $\mathbf{h} = \sum_{f \in \mathcal{S}} \varphi_f(z_f) + \boldsymbol\eta$ is the sum of $k$ contributions, each of which lies on a low-dimensional concept manifold immersed in activation space. Under a spike-and-slab prior on which manifolds are active and an assumption that each factor spans a low-dimensional linear subspace in activation space (the block analogue of block-RIP), the MAP objective for recovering the code reduces to a block- $\ell_0$ penalty on the number of active blocks — with per-block penalties depending only on the block norm, since any orthonormal rotation within a block leaves the concept invariant.

The three BSF architectures instantiate this principle with different approximations to the NP-hard block- $\ell_0$ problem:

Vanilla BSF — free linear encoder $\mathbf{x}\mathbf{W}_{\text{enc}} + \mathbf{b}$ , free linear decoder, block-TopK activation function that keeps the $g_{\text{active}}$ blocks of largest $\ell_2$ -norm and zeroes the rest (block analogue of absolute TopK / JumpReLU).
Grassmannian BSF — tied encoder-decoder ( $\mathbf{W}_{\text{enc}} = \mathbf{W}_{\text{dec}}^\top$ ), each block constrained to an orthonormal frame on the Stiefel manifold, one learned scalar per block to compensate for the energy lost by tying, block-TopK activation.
Group Lasso BSF — free linear encoder, block soft-threshold activation function (the proximal operator of the $\ell_{2,1}$ mixed norm), no hard sparsity constraint but a block- $\ell_{2,1}$ penalty added to the reconstruction loss.

Evaluation uses two rulers. On the synthetic toy where ground truth is known, per-primitive $R^2$ against an oracle matches BSFs to primitives. On real activations where ground truth is unknown, MDL under Eq. 5 splits the transmission cost into support bits, code bits, residual bits (at a task-set distortion floor), and amortized dictionary bits — with the support term $\log\binom{n}{n_{\text{active}}}$ being the axis on which block sparsity beats unstructured sparsity (a single block index stands in for $g$ coordinates). Concepts are visualized at two resolutions: block-norm heatmaps for “where is this concept present” (same as an SAE), and principal-component colorings of the block contribution $\mathbf{W}_f \mathbf{z}_f$ for “how does the concept vary once active” — a per-token RGB rendering of the top-3 PCs.

Results

Numbers are all from the paper’s tables and figures:

Toy recovery (per-block R² against oracle 0.94): Vanilla BSF 0.85, Grassmannian BSF 0.87, Group Lasso BSF 0.94, TopK SAE 0.30, SMixAE 0.31 → 0.86 (with block-TopK + signed codes), MFA 0.53 → 0.95 (with additive decode) [§3.1, Fig. 4-5].
DINOv3 task-tolerance floor: classification & segmentation stable down to R² = 0.20; depth estimation requires R² ≥ 0.70 to keep 99% accuracy — the MDL analysis is anchored at R² = 0.7 as a result [Fig. 6 top].
DINOv3 MDL at R² = 0.7: Grassmannian BSF < TopK SAE in total bits across every dictionary width tested; optimal block dimension $g$ ≈ 8–16 for small dictionaries, drops as the dictionary widens [Fig. 7].
Per-block stable rank on DINOv3: saturates at 2–4 dimensions even at block capacity $g = 128$ — the observed intrinsic dimensionality of a visual concept is 2–4, independent of the room the featurizer allocates [Fig. 8].
InceptionV1 curve detectors: all previously-identified curve-detector neurons read off a single continuous curve manifold with a Fourier-harmonic decomposition — 1st harmonic (360° orientation), 2nd harmonic (180° flip symmetry), higher harmonics; ties together the Cammarata et al. 2020 and Gorton 2024 findings under one representation [§ curve detectors].
SDXL manifold steering: 2D grid walks within a single BSF block produce smooth, interpretable sweeps over a concept’s factors of variation (pretzel shape, hat parameters, coffee cup / milk / spoon parameters) with no per-parameter supervision [§ SDXL].

Why it’s interesting

BSF is the first activation-decomposition primitive on the wiki that is natively multidimensional per concept, and it lands as a matched-prior response to the growing pile of “concepts aren’t single directions” evidence — Emergent Representation Geometry has been collecting that evidence (multi-hop graph geometry, cross-model SAE atoms, sensory-cue-steered kernel geometry) but every filed method still decomposes those geometries with one-dimensional atoms. BSF is the first tool the wiki has that reads out what those papers argue is in. Sharpens two open questions on Mechanistic Interpretability: (1) SAE atoms are demonstrably fragmenting what BSFs recover as single coherent concepts (the InceptionV1 curve manifold shattered across many SAE atoms; the DINOv3 arch region collapsed to a single line), which reframes the “how many features does a model have?” debate as partly an artifact of the featurizer’s dimensional prior; and (2) the block-norm-plus-PCA-of-contributions reading gives a cleaner steering interface than any single direction — the SDXL manifold-steering result is the constructive complement to Emotion Concepts and their Function in a Large Language Model‘s single-direction emotion-vector steering, showing that when a concept genuinely has internal geometry, a subspace-native tool lets you steer along the geometry rather than turning a scalar knob. Also worth noting: the paper is a Goodfire (+ Harvard, Stanford, Brown) release with a slate of authors overlapping Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment and Introducing Silico — Goodfire's platform for building AI models with the precision of written software — this is the technical follow-through to the Silico launch pitch that mech-interp needs primitives beyond one-dimensional SAE atoms.