1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML

TL;DR

PrismML, a Caltech spinout (Babak Hassibi et al.), emerges from stealth with the 1-bit Bonsai model family — 8B, 4B, and 1.7B LLMs whose entire network (embeddings, attention projections, MLPs, and LM head) uses 1-bit weights with no higher-precision escape hatches. The flagship 1-bit Bonsai 8B fits in 1.15 GB (vs ~16 GB for FP16 8B), runs 8× faster on edge hardware, is 4–5× more energy-efficient per token, and reportedly matches full-precision 8B instruct models on a 6-category benchmark average (70.5). All three sizes ship under Apache 2.0 with MLX (Apple Silicon) and GGUF (llama.cpp / CUDA / Metal) checkpoints day-0. Notable for being a true end-to-end 1-bit deployment story on commodity hardware, not just a quantized-weights post-training pass.

Key claims

1-bit Bonsai 8B is end-to-end 1-bit across all 8.2B parameters — embeddings, attention layers, MLP layers, and LM head — with no FP16 or higher-precision components in the network [§Bonsai 8B technical paragraph].
Memory footprint is 1.15 GB on GGUF Q1_0 and 1.28 GB on MLX 1-bit g128, vs 16.38 GB for FP16 8B — a ~12–14× reduction [HF model cards].
Reported inference is 6.2× faster than FP16 8B on RTX 4090 (GGUF), 8.4× faster on M4 Pro (MLX), and 44 tok/s on an iPhone 17 Pro Max running MLX Swift [HF model cards].
Energy per output token is reportedly 4–5× lower than FP16 8B; on mobile, instantaneous power can be higher but token generation is fast enough that per-token energy still drops [HF MLX card note].
Bonsai 8B reports a 70.5 average across six benchmark categories, claimed competitive with FP16 Llama3 8B / Qwen3 8B at 1/14th the size [blog post, HF model cards].
“Intelligence density” — defined as −log(avg error rate) / model size in GB — for Bonsai 8B is 1.06/GB vs 0.10/GB for FP16 Qwen3 8B, a 10.6× ratio (raw benchmark ratio is ~12.7×) [§Intelligence density paragraph].
The 1-bit format is software-emulated on existing hardware (CUDA / Metal / NPU); current gains come from custom Q1_0 / MLX dequantization kernels, not from 1-bit silicon — PrismML maintains its own forks of llama.cpp and mlx to provide these kernels [HF model card “limitations” sections].
The release is Apache 2.0 across all three sizes (8B / 4B / 1.7B), with a parallel Ternary Bonsai family ({−1, 0, 1} weights, ~9× smaller, ~5× faster) positioned as a higher-quality / less-compressed alternative [PrismML homepage].
The mathematical foundation is described as a Caltech research program on “compressing a neural network without losing its reasoning capabilities” — the blog and HF cards reference a forthcoming whitepaper but do not include training details (data recipe, training compute, distillation-vs-from-scratch) in the public release [§CEO quote; HF card link to “Whitepaper”].

Method

The public release does not disclose the training recipe (whether 1-bit Bonsai is quantization-aware trained from scratch, distilled from a larger FP16 teacher, or post-training quantized with calibration). What is described:

End-to-end 1-bit: every weight matrix in the network (token embeddings, Q/K/V/O projections, MLP up/gate/down, LM head) is stored at 1-bit precision. Inputs/activations remain higher precision; the dequantization kernel materializes per-block scales inline at matmul time rather than expanding the weight tensor to FP16 in memory.
Group-128 quantization: both the MLX (mlx-1bit-g128) and GGUF (Q1_0_g128) variants use 128-element groups, the standard granularity for low-bit LLM quantization. This sets the bit-budget overhead from scales/codes.
Custom kernels in forked runtimes: PrismML ships forks of llama.cpp (with Q1_0 CUDA/Metal kernels) and mlx (with 1-bit kernels for Apple Silicon). Upstream PRs are described as “pending.” Without these forks, the GGUF/MLX files are not loadable.
Training scale: trained on Google TPU v4. No token count, no compute estimate, no curriculum description in the public release.

The “Whitepaper” link on the HF model cards is the intended technical disclosure; it is not yet referenced as an arXiv ID.

Results

Headline numbers from the HF cards (evaluated via EvalScope v1.4.2 + vLLM 0.15.1 on H100, “identical infrastructure, generation parameters, and scoring” across 6B–9B models):

Average across 6 benchmark categories: 70.5 for 1-bit Bonsai 8B, described as competitive with leading FP16 8B instruct models; specific per-benchmark scores and the identity of the 6 categories are referenced but not reproduced in the search-accessible portion of the HF card.
Intelligence-density score: 1.06/GB (Bonsai 8B) vs 0.10/GB (FP16 Qwen3 8B) — a 10.6× ratio under PrismML’s −log(error)/GB metric, vs ~12.7× under a raw benchmark-average / GB metric.
Memory: 1.15 GB (GGUF Q1_0) / 1.28 GB (MLX 1-bit) for Bonsai 8B; ~0.5 GB for Bonsai 4B; ~0.24 GB for Bonsai 1.7B.
Throughput: 6.2× faster than FP16 on RTX 4090; 8.4× faster on M4 Pro; 44 tok/s on iPhone 17 Pro Max via MLX Swift.
Energy: 4–5× lower energy per token on GPU; on mobile, measured via Xcode Power Profiler (“estimated rather than hardware-metered” — explicit caveat).

Caveats the release itself flags: vendor-reported numbers, no native 1-bit silicon, mobile power is estimated, the FP16 frontier continues to advance.

Why it’s interesting

The interesting bit is not that 1-bit weights are possible (BitNet b1.58 and follow-ups established that at smaller scales), but that PrismML claims a production-ready commercial deployment with end-to-end 1-bit including embeddings and LM head, shipped Apache 2.0, with custom runtimes for both Apple Silicon and CUDA at launch. Against LLM Inference Efficiency‘s current frame (lossless speed-ups via speculative decoding, KV-cache reorganization, prefix cacheability) this is a different lever — lossy model compression at the weight level that produces a model small enough that the KV-cache and runtime-overhead concerns those papers worry about become structurally easier on edge hardware. It complements Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments‘s warning that low-bit claims need symmetric baselines: Bonsai’s numbers are vendor-reported and pre-whitepaper, so independent verification will matter. As an open-weights release it slots into Open foundation-model releases as the smallest-footprint datapoint to date — 1.15 GB Apache-2.0 8B with day-0 MLX + GGUF runtimes is a different shape than the 80B-MoE HunyuanImage 3.0 or the four-mode K2.5 release, but it follows the same “multi-variant + multi-backend at launch” packaging convention.