Skip to content
Bud's Wiki

1-bit Bonsai 27B — GGUF: 27B-class Reasoning at 1.125 Bits Per Weight

TL;DR

Section titled “TL;DR”

PrismML scales its end-to-end binary-weight recipe from the March 2026 Bonsai 8B release up to a 27B-class model derived from Qwen3.6-27B, shipped as GGUF for llama.cpp (CUDA + Metal). Every language weight — embeddings, attention projections, MLP projections, and LM head — is stored in a true 1.125-bit format (Q1_0_g128: sign bit + one FP16 scale per 128 weights), collapsing the model from ~54 GB (FP16) to ~3.9 GB deployed, a 14.2× reduction. Reported quality is 76.11 average across 15 thinking-mode benchmarks (89.5% of FP16) — with math holding at 91.66 and coding at 81.88 — while conventional sub-4-bit builds of the same base model (IQ2_XXS at 2.8 bpw) collapse to 72.73. Runs at 44 tok/s on Apple M5 Pro and 104.8 tok/s on H100 batch=1, and packages a DSpark speculative-decoding drafter that turns the H100 number into 143.8 tok/s (1.37× lossless). Ships Apache 2.0 with a 262K context (retained via 4-bit KV quantization) and a companion ternary variant.

Key claims

Section titled “Key claims”
  • End-to-end binary weights cover all 27.3B language parameters — embeddings, attention Q/K/V/O, MLP up/gate/down, and LM head — at a true average of 1.125 bpw (1 sign bit + 16-bit FP16 scale amortized over 128-element groups); the vision tower ships separately in HQQ 4-bit [§Model Overview, §Weight Representation].
  • Deployed footprint is ~3.9 GB (vs ~54 GB FP16 Qwen3.6-27B), a 14.2× reduction that matches the label — the release explicitly contrasts this against IQ2_XXS (“2-bit”) which is actually 2.8 bpw / 9.4 GB [§Weight Representation].
  • Thinking-mode average across 15 benchmarks is 76.11 (89.5% of FP16’s 85.07); the companion Ternary Bonsai 27B (1.71 bpw / 5.9 GB) reaches 80.49 (94.6%), and conventional Qwen3.6-27B IQ2_XXS at similar footprint reaches only 72.73 (85.5%) [§Benchmarks].
  • Degradation of conventional sub-4-bit builds is selective, not uniform: IQ2_XXS collapses to 57.5 on AIME26 and 56.4 on LiveCodeBench while holding 88.93 on MMLU-Redux — casual eval misses the collapse; 1-bit Bonsai keeps AIME26 at 87.08 at a third of IQ2_XXS’s footprint [§Benchmarks, §Full Per-Benchmark Results].
  • Category breakdown (FP16 → 1-bit): math 95.33 → 91.66, coding 88.74 → 81.88, knowledge/reasoning 83.15 → 73.39, instruction following 78.47 → 65.74, agentic/tool-calling 80.00 → 66.03, vision 72.61 → 59.57 — the reasoning core is preserved, with the gap concentrated in instruction-following, agentic, and vision [§By Skill Category].
  • Throughput on llama-bench with the custom Q1_0_g128 kernels: 44.2 tok/s TG128 on Apple M5 Pro, 66.4 on M5 Max, 26.0 on M4 Pro, and 104.8 on H100; PP512 reaches 2755 tok/s on H100 [§Cross-Platform Throughput].
  • 262K context is retained on-device because the Qwen3.6-27B backbone is ~75% linear attention (full attention on only 16 of 64 layers); combined with 4-bit KV-cache quantization, the full 262K window fits in ~9.4 GB peak, and 100K context runs at 11.6 GB peak without any KV compression [§Model Overview, §Peak Memory at Context].
  • Ships a DSpark speculative-decoding drafter — a six-layer block-parallel transformer conditioned on hidden states tapped from five evenly-spaced target layers; drafter unique weights add ~0.5 GB, with the drafter shipped 4-bit quantized (~1.79 GB Q4_1 default). On H100 CUDA, τ ≈ 3.6 accepted tokens at draft depth k=4 gives a 1.37× lossless decode speedup (104.8 → 143.8 tok/s); Apple Silicon batch-1 does not yet amortize verification, so the drafter is off by default there [§Speculative Decoding: DSpark].
  • Under PrismML’s “intelligence density” metric D = -log2(1 - score/100) / size_GB, 1-bit Bonsai 27B reaches 0.530/GB vs 0.199/GB for Qwen3.6-27B IQ2_XXS and 0.051/GB for Qwen3.6-27B FP16 — ~2.7× the densest conventional build and >10× FP16 [§Intelligence Density].
  • Measured decode energy on M5 Pro is 0.275 mWh/token with the DSpark drafter enabled — an order of magnitude below datacenter GPUs (0.63–1.32 mWh/token) [§Cross-Platform Throughput].
  • Base model is a Qwen3.6-27B derivative with architecture unchanged (hybrid ~75% linear / ~25% full attention, SwiGLU MLP, RoPE, RMSNorm), and the release is Apache 2.0 with an MLX companion (Bonsai-27B-mlx-1bit) reporting ~11 tok/s on iPhone 17 Pro Max via MLX Swift [§Model Overview, §Use Cases].

Method

Section titled “Method”

The weight format is Q1_0_g128 — the GGUF layout is the model’s native on-disk shape, not a post-hoc quantization of stored FP16 weights. Each weight is one sign bit (0 → −scale, 1 → +scale), and every group of 128 weights shares a single FP16 scale factor; effective bit-width is 1.125 (1 + 16/128). This is applied uniformly across embeddings, attention Q/K/V/O projections, MLP up/gate/down, and the LM head — the release explicitly rejects the “conventional 2-bit label at 2.8 true bpw” pattern by presenting a bit-width that matches its name. The vision tower is a separate 4-bit HQQ pack (~0.63 GB) loaded only when an image arrives, so text-only serving never pays for it.

Inference runs on PrismML’s fork of llama.cpp with custom Q1_0_g128 CUDA and Metal kernels for the hybrid-attention backbone; packed 1-bit weights are consumed directly rather than dequantized back to FP16 in memory. KV cache is compressed to 4-bit (near-lossless per the release), and because the Qwen3.6-27B backbone is predominantly linear attention (48 of 64 layers), a full-attention cache is grown on only 16 layers — this is the structural reason 262K context stays practical on-device. Training methodology (QAT-from-scratch vs distillation from FP16 Qwen3.6-27B vs post-training quantization) is not disclosed in the HF card, but the note “distilled to 1 bit” from the Slack pointer plus the reported 89.5% retention across 15 thinking-mode benchmarks strongly suggests a distillation/QAT pipeline against an FP16 teacher — matching the shape of the earlier Bonsai 8B release. The referenced whitepaper is at github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf.

The DSpark drafter is the second component worth noting mechanically. It is a six-layer block-parallel transformer with a diffusion-flavored block-denoising objective, survival-probability-weighted distillation from the target, per-source-normalized hidden-state taps at five evenly-spaced target layers, and a draft block size chosen from a measured verify-cost model of the serving stack. Drafter embeddings and output head are shared with the resident target, so only ~0.5 GB of drafter-unique weights are added at serving precision. Because verification preserves the target distribution exactly, drafter precision affects only speed, not output quality — hence the drafter itself ships 4-bit.

Results

Section titled “Results”

Headline numbers, thinking mode, evaluated with EvalScope + vLLM on H100 under identical infrastructure across 15 benchmarks [§Benchmarks]:

  • Aggregate: 76.11 (1-bit Bonsai 27B) vs 85.07 (Qwen3.6-27B FP16) vs 80.49 (Ternary Bonsai 27B) vs 72.73 (Qwen3.6-27B IQ2_XXS at 2.8 bpw / 9.4 GB) vs 73.31 (Gemma-4-31B Q2_K_XL at 3.0 bpw / 11.8 GB) [§Benchmarks].
  • Reasoning-critical benchmarks: AIME25 88.75 (vs FP16 93.29), AIME26 87.08 (vs 93.33), MATH-500 98.00 (vs 99.40), GSM8K 92.80 (vs 95.30), LiveCodeBench 76.40 (vs 87.77), HumanEval+ 89.63 (vs 95.12), MBPP+ 79.60 (vs 83.33), MMLU-Redux 82.75 (vs 93.42) [§Full Per-Benchmark Results].
  • Weakest categories: τ²-Bench (agentic) collapses from 82.90 → 61.34, MMMU-Pro (vision) from 79.94 → 60.48, IFBench from 68.03 → 52.36 — these are the categories that “trade part of the ternary model’s margin for the smallest footprint in the family” [§Full Per-Benchmark Results, §By Skill Category].
  • Peak memory at context (language model only, no KV compression, decimal GB): 3.79 weights → 5.2 GB at 4K → 11.6 GB at 100K; the same 100K peak drops to ~6.8 GB with 4-bit KV; full 262K fits in ~9.4 GB peak with 4-bit KV — vs the comparable Qwen3.6-27B “4-bit” Q4_K_XL build requiring 19.2 GB even at 4K [§Peak Memory at Context].
  • Throughput: TG128 44.2 tok/s on M5 Pro / 66.4 on M5 Max / 104.8 on H100; PP512 421 / 874 / 2755 tok/s [§Cross-Platform Throughput].
  • Speculative decoding: DSpark on H100 CUDA lifts decode from 104.8 → 143.8 tok/s (1.37×) losslessly at k=4, τ ≈ 3.6 [§Speculative Decoding: DSpark].
  • Energy: 0.275 mWh/token on M5 Pro with DSpark enabled vs 0.63–1.32 mWh/token for datacenter GPUs [§Cross-Platform Throughput].
  • Intelligence density: 0.530/GB (1-bit) vs 0.400/GB (ternary) vs 0.199/GB (IQ2_XXS) vs 0.051/GB (FP16) — no conventional Qwen3.6-27B or Gemma-4-31B build exceeds 0.2/GB [§Intelligence Density].

Caveats the release itself surfaces: quality gap widens on long-horizon agentic coding (not a target of this release; a Bonsai 27B agentic-coding variant is on the roadmap); tolerance to KV-cache error grows with context and could be pushed sub-2-bit; the H100 datacenter row is bandwidth-bound-agnostic (binary and ternary converge at 104.8 vs 98 tok/s at batch 1) because kernel-launch latency dominates weight bandwidth for batch-1 datacenter inference.

Why it’s interesting

Section titled “Why it’s interesting”

This is the first datapoint on the wiki where end-to-end binary weights are pushed to a 27B-class model with reasoning benchmarks that hold up — 91.66 math and 81.88 coding in a 3.9 GB footprint that fits mainstream laptops. It sharpens the “lossy weight quantization is a real fourth lever” story on LLM Inference Efficiency that 1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML opened at 8B: the retention rate (89.5% here vs the ~parity claim at 8B) suggests the recipe scales but incurs a modest quality tax at larger sizes, which will need independent symmetric-baseline reproduction per the methodological warning already logged from Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments. Two elements make this more than just a scale-up of the 8B release: the 262K context on-device (structurally enabled by inheriting Qwen3.6-27B’s ~75% linear-attention backbone — a direct payoff for Hybrid Linear Attention‘s architectural bet), and the packaged DSpark drafter, which is the first filed instance on this page of a weight-quantization release shipping its own speculative-decoding drafter, resolving the wiki’s own open question of whether 1-bit and SD compose. The category collapse pattern (agentic and vision worst, reasoning best) is also useful signal: it suggests the same binary-weight ceiling the earlier release predicted for “the most demanding categories” is quantifiable and predictable, not random. Companion release PrismML — 1-bit and Ternary Bonsai Image 4B — local-hardware diffusion image generation extended the recipe to diffusion image generation without benchmarks; this release provides the missing benchmark rigor at scale for the LLM side.

See also

Section titled “See also”