Introducing Inkling — Thinking Machines' open-weights 975B/41B MoE multimodal reasoning model

TL;DR

Inkling is Thinking Machines’ first from-scratch open-weights model — a 975B-total / 41B-active MoE transformer pretrained on 45T tokens of text, images, audio, and video, with up to 1M context. A preview sibling, Inkling-Small (276B / 12B-active), matches or exceeds it on many benchmarks. The model is positioned as a customization base rather than a frontier ceiling: multimodal by design, controllable thinking effort via per-token cost during RL, and available for fine-tuning on Tinker at launch. Architecturally it inherits DeepSeek-V3’s MoE recipe (256 routed + 2 shared, top-6, sigmoid + auxfree bias) but departs from the standard transformer in three specific places: relative positional bias instead of RoPE, 5:1 sliding-window/global attention, and short convolutions applied after K/V projections and on the residual outputs of attention and MLP branches. Trained with hybrid Muon (matrix weights) + Adam (everything else), weight decay coupled to LR² (from their Modular Manifolds work), on NVIDIA GB300 NVL72.

Key claims

MoE architecture is DeepSeek-V3-derived: 256 routed experts + 2 shared, 6 routed active per token, sigmoid router with auxiliary-loss-free load-balancing bias; scores of selected routed + shared experts are normalized jointly and used to weight combined outputs [§Architecture].
Attention departs from the modern default in three places: 5:1 sliding-window/global interleave with 8 KV heads, a learned relative positional bias added to attention scores (not RoPE) which is reported to extrapolate better to longer sequences, and short convolutions applied both after K/V projections in each attention layer and on the attention/MLP branch outputs before rejoining the residual stream [§Architecture].
Trained with a hybrid optimizer — Muon for large matrix weights, Adam for other parameters — and hyperparameter schedules explicitly derived from the earlier modular-manifolds work; weight decay is coupled to the square of the learning rate, which the authors report keeps overall weight-matrix size stable across training horizons [§Training].
Post-training bootstraps with a small SFT phase on synthetic data generated by open-weights models including Kimi K2.5, then does majority-compute large-scale RL: >30M rollouts across two long continuous runs, with reasoning performance improving log-linearly throughout [§RL at scale].
Controllable thinking effort is trained by varying the system message and per-token cost across RL samples, letting the deployment-time effort knob interpolate along a token-count × performance curve [§Controllable thinking effort]. Reported gain: Inkling matches Nemotron 3 Ultra on Terminal Bench 2.1 while spending ~⅓ the tokens.
Multimodal design is encoder-free for both audio and vision: audio inputs are discrete dMel spectrograms (Bai et al., 2024), images are 40×40 pixel patches through a 4-layer hMLP (Touvron et al., 2022), both transformed via a lightweight embedding layer and processed jointly with text tokens [§Multimodality].
Inkling-Small (276B / 12B-active, preview) matches or exceeds Inkling on many benchmarks despite ⅓ the active parameters — the authors credit improvements to the pre-training data and recipe for the smaller model, with the same post-training stack shared between the two [§Inkling-Small, Table].
On the benchmark table shipped with the release, Inkling at effort=0.99 is competitive with or below other open-weights peers (Nemotron 3 Ultra, Kimi K2.5/K2.6, GLM 5.2, DeepSeek V4 Pro) across HLE, AIME 2026, GPQA Diamond, SWEBench Verified, and Terminal Bench 2.1; it is not the strongest overall model [§Benchmarking Inkling, main table].
Emergent-during-RL chain-of-thought compression: the model’s CoT became progressively more concise over the course of RL training without being targeted by any reward, driven by efficiency alone; a similar effect was noted concurrently by the Cognition team while training SWE-1.7 [§RL at scale].
Training epistemics with dual graders: rubric graders (recall-oriented, hackable) are paired with agentic web-search claim-verification graders, plus short-form factual QA with abstention-aware rewards, to improve helpfulness and reduce hallucination jointly rather than trading them off [§Epistemics].
Weights shipped as both the original checkpoint and an NVFP4 checkpoint for efficient inference on NVIDIA Blackwell, with multi-backend day-0 support: Together, Fireworks, Modal, Databricks, Baseten APIs, plus SGLang / Miles (RadixArk), vLLM (Inferact), TokenSpeed (Lightseek), llama.cpp (Unsloth), and Hugging Face transformers [§Inkling Availability].

Method

The pretraining stack is unusually opinionated about stability primitives. Rather than tuning many independent knobs, Thinking Machines picks a small set of coupled invariants — Muon-driven spectral control on matrix weights, weight-decay proportional to LR² (so the weight-norm regime is invariant to LR schedule choice), and architectural changes (relative bias, short convolutions on residual branches) whose stated motivation is length extrapolation and long- context performance. This lineage traces cleanly back to Modular Manifolds (weights on Stiefel-manifold submanifolds via Muon) and the QK-Clip line (QK-Clip: Taking Muon Further on the Scaleup Journey, Rethinking Language Model Scaling under Transferable Hypersphere Optimization (HyperP)) — a stability- first, hyperparameter-transfer-oriented worldview rather than the raw-scale worldview of the DeepSeek-V3 report it forks its MoE from.

Post-training is RL-dominated (SFT is explicitly a small bootstrap on K2.5-generated synthetic data). Effort-controllable inference is baked in during RL by varying the system message and per-token cost per sample so a single checkpoint learns a smooth effort → tokens → performance curve, rather than shipping distinct “instant/thinking” checkpoints in the DeepSeek-V3.1 — hybrid thinking + non-thinking model, smarter tool calling, faster thinking style. Multimodality is trained end-to-end with encoder-free tokenizers (dMel for audio, 40×40 hMLP patches for image) — a deliberate contrast with the frozen-encoder + projector stack still standard elsewhere and consistent with recent pixel-first arguments (Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation, Gemma 4 12B — encoder-free unified multimodal model with 35M vision embedding (Google Gemma) Gemma 4 12B).

Results

HLE text-only: 29.7% at effort=0.99 (vs Kimi K2.6 35.9%, GLM 5.2 40.1%, DeepSeek V4 Pro 35.9%; behind closed frontier: Gemini 3.1 Pro 44.7%, Claude Fable 5 max 53.3%, GPT 5.6 Sol 47.2%) [main table].
AIME 2026: 97.1% vs Nemotron 3 Ultra 94.2% / GLM 5.2 99.2% / GPT 5.6 Sol 99.9% [main table].
SWEBench Verified: 77.6% (behind Kimi K2.6 80.2%, GLM 5.2 80.0%, DeepSeek V4 Pro 80.6%, GPT 5.6 Sol 82.2%; ahead of Nemotron 3 Ultra 70.7%) [main table].
Terminal Bench 2.1 (best harness): 63.8% vs Kimi K2.6 71.3%, GLM 5.2 82.7%, GPT 5.6 Sol 91.9% — the specific benchmark where Sid’s tweet-source calls out relative weakness [main table].
Token efficiency: same Terminal Bench 2.1 point reached with ~⅓ the tokens of Nemotron 3 Ultra at matched score [§Controllable thinking effort].
Multimodal (all vs closed baselines): MMMU Pro Standard 10 73.5% (Kimi K2.6 79.0%, Gemini 3.1 Pro 82.0%); Charxiv RQ 78.1%; VoiceBench 91.4%; MMAU 77.2%; AudioMC 56.6% (Gemini 3.1 Pro 66.8%) [main table].
Forecasting: ForecastBench Brier Index (no search) 61.1 ± 0.79 — competitive with Grok 4.3 (61.7) and Gemini 3.1 Pro (61.1), ahead of GPT-5.5 (59.1) and Claude Opus 4.8 (54.6) at an earlier checkpoint [§Epistemics].
Safety: FORTRESS Adversarial 78.0% (highest among open compared: Nemotron 3 Ultra 77.6, Kimi K2.6 65.6, GLM 5.2 71.3, DeepSeek V4 Pro 36.0); StrongREJECT 98.6% [§Safety].
Inkling-Small ≈ Inkling on reasoning (HLE 29.6 vs 29.7, AIME 95.1 vs 97.1, GPQA Diamond 88.3 vs 87.2), SWEBench Verified (77.4 vs 77.6), and MCP-Atlas (74.9 vs 74.1); falls behind on SimpleQA Verified (20.9 vs 43.9), Terminal Bench 2.1 (52.7 vs 63.8), and Tau 3 Banking (13.6 vs 23.7) [§Inkling-Small, Table].

Note on Sid’s Slack claim about muP: the blog does not claim muP. Thinking Machines describes hybrid Muon + Adam with LR²-coupled weight decay inspired by modular manifolds; muP is a distinct parameterization framework and this write-up does not invoke it. Muon-family updates do preserve some µP-adjacent scaling properties per How Muon Lost Its Geometry, which may be what Sid was reading through.

Why it’s interesting

Inkling is the concrete arrival of the Thinking Machines research program — modular manifolds, hybrid Muon+Adam stability, on-policy distillation, the interaction-models framing — assembled into a single frontier-class release. The three architectural departures (relative bias not RoPE, short convolutions on the residual branch, 5:1 SWA:GA) are worth calling out: they are close-but-not-identical to Xiaomi MiMo-V2-Flash’s 5:1 SWA hybrid (Xiaomi MiMo-V2-Flash — 309B/15B MoE with 5:1 SWA hybrid attention, 73.4 SWE-Bench Verified, Fuli Luo on MiMo-V2-Flash engineering — Hybrid SWA, MTP, MOPD post-training) and to Laguna XS.2’s 3:1 SWA:GA with partial-RoPE global layers (Laguna M.1/XS.2 Technical Report) — a convergent 2026 trend of hybrid local/global attention where the global stream carries the long-range positional information and the local stream carries most of the layers. Inkling’s specific twist is dropping RoPE entirely in favor of a learned relative-position bias on the global layers. The encoder-free multimodal design puts it in the same architectural family as Gemma 4 12B — encoder-free unified multimodal model with 35M vision embedding (Google Gemma) (Gemma 4 12B with a 35M vision-embedding module) and Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation — the frozen-encoder-plus- projector paradigm is now visibly on the way out at the frontier. Positioned against DeepSeek V4 Pro / Kimi K2.6 / GLM 5.2 / Nemotron 3 Ultra in the same open-weights cohort, Inkling is not the strongest at any single point on the raw-benchmark axis; its differentiator is the customization surface — Tinker fine-tuning at launch, effort- controllable thinking as a native RL-time output, encoder-free multimodality that transfers to interaction-model deployment.