Your DeepSeek mHC Might Not Need the "m"

ListenThrow Away the "m": When Identity Beats the Fancy Residual Mix10:36

Show notes

A translated Zhihu post by Shuanyueliang de Zhexianren runs a direct ablation against DeepSeek's mHC and finds that replacing its Sinkhorn-Knopp doubly-stochastic residual-mixing matrix with a plain identity matrix works *better* — on Qwen3 1.7B and 8B dense models over 150B tokens. We unpack why: the cumulative product of those mixing matrices collapses to a rank-one uniform bottleneck, and the real cross-stream fusion was living in the input-dependent φ projection all along. We also connect it to Hyperloop as a second independent datapoint that the manifold projection isn't pulling its weight. **Sources** - Your DeepSeek mHC Might Not Need the "m" — 涮月亮的谪仙人 (Shuanyueliang de Zhexianren), translated by Tyler Romero

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

An English translation (by Tyler Romero / Opus 4.6) of a Zhihu post by Shuanyueliang de Zhexianren reporting a direct ablation against mHC: Manifold-Constrained Hyper-Connections: on Qwen3 1.7B and 8B dense models trained for 150B tokens with Megatron, replacing mHC’s Sinkhorn-Knopp-projected doubly-stochastic residual-mixing matrix H_res with the plain identity matrix beats mHC, mHC-lite, and mHC-orthogonal variants. The author’s measured ordering is Identity-HC > mHC > mHC-lite > mHC-orthogonal (Cayley). Mechanism: the single-layer H_res that Sinkhorn-Knopp learns is already close to identity (diagonal ~0.96, off-diagonal ~0.01), but by Dobrushin-ergodicity the layer-wise product of strictly-positive doubly-stochastic matrices collapses geometrically to the rank-1 uniform 1/n · 1ᵀ1 matrix (measured minimum eigenvalue of the 56-layer composite H_res product on Qwen3-1.7B: 9.2 × 10⁻¹⁸, matching the geometric-decay estimate). Setting H_res = I locks in what the model was learning anyway, eliminates the collapse, drops 20-step Sinkhorn iterations + a specialized TileLang kernel, and the “cross-stream mixing” the mHC paper credits to H_res is actually happening inside the input-dependent φ projection that generates H_pre and H_post — not in H_res at all.

Key claims

On Qwen3 1.7B and 8B dense models, 150B tokens of Megatron pretraining, the measured ordering is Identity-HC > mHC > mHC-lite > mHC-orthogonal (Cayley) [§opening].
Single-layer H_res learned by mHC’s Sinkhorn-Knopp projection is empirically close to identity (diagonal ~0.96, off-diagonal ~0.01); the depth-cumulative product collapses to the uniform 0.25 · 1ᵀ1 matrix for n=4 streams after about 10 layers [§“Single-layer vs cumulative”].
The collapse is a theorem, not an accident: for any strictly-positive doubly-stochastic matrix sequence with uniform positive lower bound δ, the Dobrushin ergodicity coefficient τ ≤ 1 − nδ < 1 and the cumulative product’s row differences decay as τ^L, forcing convergence to the rank-1 uniform matrix [§“Mathematical reason”].
Measured minimum eigenvalue of the mHC composite H_res product across 56 HC modules on Qwen3-1.7B is 9.2 × 10⁻¹⁸, matching the geometric estimate (0.49^56 ≈ 10⁻¹⁸); the Sinkhorn version’s measured mean minimum eigenvalue per layer is 0.49 [§“Perron-Frobenius decomposition”].
Sinkhorn-Knopp’s 20-iteration finite approximation adds error: measured row-sum standard deviation of 0.12 per layer, and ~27.9% of inputs have relative range > 10³ where 20 iterations give column-sum deviation up to 100% [§“Sinkhorn approximation error”, citing mHC-lite].
When H_res = I, the cross-stream mixing the mHC paper credits to H_res actually happens in the φ projection (streams flattened to [s, b, n·C], projected to [s, b, 2n], split into h_pre / h_post dynamic coefficients) — φ’s input contains information from all n streams, so h_pre = sigmoid(α_pre · proj[:n] + b[:n]) aggregates across streams dynamically even without any H_res mixing [§“Is φ the Real Key”].
With H_res = I, each stream maintains its own residual with no reshuffling across depth, so H_pre / H_post learn consistent “which stream to read from / write to” mappings instead of chasing permutations that compose into a new permutation at every layer [§“Advantage of identity”].
Setting H_res = I eliminates φ’s projection weights for the H_res branch, drops the 20-iteration Sinkhorn-Knopp forward and the custom backward recompute kernel, and signals propagate losslessly across depth instead of decaying exponentially [§“Comparison table”].
Alternative constraint families fail for different reasons: orthogonalization (Cayley / Givens) guarantees spectral norm exactly 1 but allows negative entries, causing some streams to be negated and capacity to collapse; mHC-lite with softmax weighting ran its α coefficient up from 0.01 to ~2, sharpening the softmax toward one-hot and reducing mixing [§“Failed attempts”].
Under H_res = I, the effective communication channel between any two layers (l, k) simplifies to depending only on H_post_l and H_pre_k without cumulative decay from intermediate layers’ mixing matrices [§“Analysis plots”].

Method

The author’s codebase extends Megatron with the full mHC primitive (φ projection → split into H_pre, H_post, H_res logits → Sinkhorn-Knopp on H_res → stream update), then switches a single flag that forces H_res to the identity matrix (dropping both the H_res logits output from φ and the Sinkhorn-Knopp forward/backward). All other mHC machinery is preserved: n = 4 parallel streams, the H_pre / H_post dynamic coefficients, the sigmoid activation (not the original HC’s tanh — the author credits mHC with this change), and the non-negative constraint on the read-in and write-out mappings.

Four variants were raced head-to-head on Qwen3-1.7B and Qwen3-8B dense models for 150B tokens: (a) original mHC with 20-step Sinkhorn-Knopp, (b) mHC-lite with softmax-weighted convex combinations for exact double stochasticity, (c) mHC with orthogonal H_res via Cayley transform and Givens rotations, and (d) Identity-HC with H_res = I. Diagnostic measurements include the per-layer min eigenvalue, min singular value, and row-sum standard deviation of the Sinkhorn output, plus the cumulative-product spectral statistics across 56 HC modules (28 layers × 2 per attention/MLP). The analysis plots at the end — including an effective-channel-size plot measuring how much of layer k’s transformation output propagates to layer l’s transformation input — were drawn by Opus; the author flags that some surrounding analytical text (notably the comparison table) was AI-generated and may contain overstatements, but the experimental numbers were measured.

Results

Headline: Identity-HC beats mHC on both Qwen3-1.7B and Qwen3-8B at 150B tokens, with the author summarizing the ordering as Identity-HC > mHC > mHC-lite > mHC-orthogonal [§opening]. No per-benchmark table is published — the result is qualitative (“one look and you just go quiet”) but is the headline the author came to the post to share.

Mechanistic measurements that back the claim: single-layer mHC H_res has diagonal entries averaging ~0.96 with off-diagonals ~0.01 (near-identity); cumulative-product H_res collapses to the uniform 0.25 · 1ᵀ1 matrix for n=4 by depth ~10; measured min eigenvalue of the full 56-module cumulative product is 9.2 × 10⁻¹⁸ (matching the 0.49^56 ≈ 10⁻¹⁸ estimate); measured per-layer row-sum std is 0.12 under the finite 20-iteration Sinkhorn [§“Perron-Frobenius decomposition”, §“Sinkhorn approximation error”].

Negative results on alternative constraint families: mHC-lite with softmax weighting underperforms original mHC and shows α rising from 0.01 to ~2, which the author diagnoses as softmax sharpening into one-hot (less mixing); orthogonal H_res via Cayley/Givens stays near α ≈ 0.01 and loses capacity because negative entries let streams cancel [§“Failed attempts”].

Comparison table (reproduced from the post, author flags this specific table as AI-generated prose but experimental numbers measured):

Metric	Sinkhorn	Identity
Cumulative product	rank-1 collapse (δ_min → 0)	identity (δ_min = 1)
Approximation error	row-sum std ≈ 0.12	exact
Extra computation	20-step iteration + backward recompute	zero
Extra parameters	φ projection weights for H_res branch	none
Signal propagation	shallow signals decay exponentially	lossless propagation

Why it’s interesting

This is a direct ablation that contradicts the central claim of mHC: Manifold-Constrained Hyper-Connections — DeepSeek’s December-2025 paper arguing the residual-stream geometry is the load-bearing stability primitive, framing n=4 doubly-stochastic residual mixing as a new scaling axis for V4 — on the paper’s own benchmark regime (dense Transformer pretraining, Qwen3-size backbones, Megatron). The surprise is where the mixing turns out to live: the author’s H_res = I ablation shows the cross-stream aggregation that mHC credits to the Birkhoff-projected H_res is actually happening inside the input-dependent φ projection that generates the H_pre / H_post dynamic coefficients, so the Sinkhorn-Knopp iteration and its specialized TileLang kernel are solving a problem that is not load-bearing — and under depth composition they are actively harmful, forcing information through a rank-1 uniform-mixture bottleneck that identity avoids.

This slots alongside Hyperloop Transformers (Zeitoun / Torroba-Hennigen / Kim, April 2026) as the second independent datapoint undermining the Sinkhorn piece of mHC: Hyperloop found that a data-dependent diagonal B beats both the identity baseline and the Sinkhorn-Knopp parameterization (PPL 14.40 vs 14.59 vs 14.61) in the loop-level HC placement, and now this post finds that at mHC’s own every-layer placement, plain identity beats Sinkhorn-Knopp outright. The two results triangulate the same conclusion from different placements — the manifold projection is not pulling its weight — and together they argue that the “H_res must be constrained to the Birkhoff polytope” claim in mHC overstates the role of H_res and understates φ’s input-dependence as the real cross-stream fusion mechanism. This is a Conflict-level finding for the training-stability-at-scale concept page where mHC is currently cited as the headline residual-geometry stability result.

The post also has a secondary methodological interest for the wiki’s growing library of informal / AI-assisted research notes: the author is explicit about which parts of the text are Opus-generated (the comparison table, some analytical plots) versus measured in-codebase, and flags the risk that AI-generated prose may be overstating things — a transparency pattern worth adopting for similar informal drops.