turbovec — Rust vector index implementing TurboQuant with Python bindings

TL;DR

turbovec is an open-source Rust vector index (with Python bindings) that implements Google Research’s TurboQuant (TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate) as a drop-in replacement for FAISS-style PQ indices on local hardware. It claims a 10M-document, 1536-dim corpus fits in 4 GB of RAM (vs ~31 GB float32) and searches faster than FAISS IndexPQFastScan — by 12–20% on ARM (NEON) and matching-or-beating on x86 (AVX-512BW). It adds a TQ+ per-coordinate calibration step (two scalars per coordinate fit on the first add) to recover up to +1.4pp R@1 on low-dim / drift-prone embeddings, and supports SIMD-kernel-level allowlist filtering for hybrid retrieval. The artifact is interesting to the team as the first public reproduction we have of TurboQuant outside the original ICLR paper and the RaBitQ rebuttal — it ships benchmark scripts, a length-renormalization correction borrowed from RaBitQ, and reports symmetric-baseline numbers on M3 Max and Sapphire Rapids.

Key claims

A 1536-dim 10M-vector corpus that occupies 31 GB as float32 fits in 4 GB under turbovec, i.e. ≈16× compression at 2-bit [README “compression” section, Fig. compression.svg].
On Apple M3 Max (NEON), turbovec beats FAISS IndexPQFastScan by 12–20% across every tested single- and multi-threaded configuration at d ∈ {200, 1536, 3072} and bit-width ∈ {2, 4} [README “ARM speed” section, Fig. arm_speed_st.svg / arm_speed_mt.svg].
On Intel Xeon Platinum 8481C (Sapphire Rapids, AVX-512BW), turbovec wins every 4-bit configuration by 1–6%, ties FAISS within ~1% on 2-bit single-threaded, and is the slower side by 2–4% only on the 2-bit multi-threaded d=1536 / d=3072 rows [README “x86 speed” section, Fig. x86_speed_st.svg / x86_speed_mt.svg].
Online ingest with no train phase: each add(...) is index-ready immediately, the random rotation and Lloyd-Max codebook are fixed analytically from the post-rotation Beta distribution rather than fit to data, so adding more vectors does not trigger a rebuild [README “How it works” §1–§5].
TQ+ per-coordinate calibration fits a (shift, scale) pair per coordinate from the empirical 5/95% quantiles on the first add batch, mapping each coordinate’s distribution onto the canonical Beta marginal before Lloyd-Max quantization; calibration is frozen after the first add and reports up to +1.4pp R@1 on low-dim / drift-prone embeddings (GloVe d=200 at 2-bit) [README “How it works” §3].
Length-renormalized scoring stores ‖v‖ / ⟨u, x̂⟩ per vector (one extra d-dim dot product at encode) and multiplies it into each candidate score, removing the systematic downward bias of scalar quantization on inner-product estimation without per-query overhead — borrowed from RaBitQ (SIGMOD 2024) [README “How it works” §6, README “References”].
Recall numbers from the README’s symmetric benchmark vs FAISS IndexPQ (LUT256, nbits=8, float32 LUT) at k=64 over 100K vectors: at OpenAI d=1536/d=3072 turbovec beats FAISS by 0.4–3.4 points at R@1 across 2-bit and 4-bit, both converging to 1.0 by k=4; on GloVe d=200, turbovec beats FAISS by 0.3 points at 4-bit and trails by 1.2 points at 2-bit, with the gap closing by k≈16 [README “Recall” section, Fig. recall_glove.svg / recall_d1536.svg / recall_d3072.svg].
Filtering happens inside the SIMD kernel at 32-vector block granularity — blocks with no allowed slots short-circuit before LUT lookup, and non-allowed slots inside scored blocks are dropped at heap-insert, so selective allowlists avoid most of the SIMD cost rather than paying-and-discarding [README “Hybrid retrieval” section].
The Lloyd-Max codebook achieves distortion within a factor of ~2.7× of the Shannon distortion-rate lower bound, with the length-renormalization step removing the residual inner-product estimator bias [README “How it works” §6, final paragraph].

Method

The implementation follows the original TurboQuant pipeline plus two pragmatic additions. (1) Pre-rotation. Each vector is L2-normalized and the resulting direction is multiplied by a single shared random orthogonal matrix; after rotation each coordinate is approximately Beta-distributed (converging to N(0, 1/d) at high d). (2) TQ+ calibration. Because the asymptotic Beta is loose at finite d (especially on low-dim and word-vector-style embeddings), TQ+ fits a per-coordinate shift+scale during the first add, mapping the empirical 5/95% quantiles onto the canonical Beta marginal. Calibration is frozen after the first add and reused by all subsequent inserts. (3) Lloyd-Max scalar quantization. Each coordinate is quantized against the target Beta marginal Lloyd-Max bins — 4 buckets at 2-bit, 16 at 4-bit. Because the distribution is analytically known, the codebook is precomputed from the math, not fit to the data. (4) Bit-pack. Encoded vectors are nibble-packed (4-bit) or pair-packed (2-bit); a 1536-dim vector goes from 6,144 bytes (FP32) to 384 bytes at 2-bit. (5) Length-renormalized scoring. At encode, one extra dot product ⟨u, x̂⟩ is computed between the unit direction and its centroid reconstruction; the index stores ‖v‖ / ⟨u, x̂⟩ per vector and the SIMD kernel multiplies each candidate’s accumulated inner product by this scalar before heap insertion — making the estimator unbiased at query time. (6) Search. A query is rotated once into the same domain and scored via SIMD: NEON intrinsics on ARM, AVX-512BW with nibble-split lookup tables on modern x86, and an AVX2 fallback for older x86. Hybrid retrieval allows an allowlist: np.uint64[] argument to search(...); the kernel skips entire 32-vector blocks whose allowlist intersection is empty.

Results

Memory: 1536-dim, 10M vectors → 4 GB at 2-bit (vs 31 GB float32) [README, intro].
ARM speed (M3 Max): +12% to +20% over FAISS IndexPQFastScan across {d=200, 1536, 3072} × {2-bit, 4-bit} × {ST, MT}, k=64, 100K vectors, 1K queries [README “ARM speed”].
x86 speed (Sapphire Rapids, 8 vCPU): +1% to +6% over FAISS on every 4-bit row; ±1% on 2-bit ST; −2% to −4% on 2-bit MT at d=1536 and d=3072 (the only configs slower than FAISS, attributed in the README to short inner-accumulate loops where FAISS’s AVX-512 VBMI path wins) [README “x86 speed”].
Recall vs FAISS IndexPQ (LUT256, nbits=8) at k=64, 100K vectors: +0.4 to +3.4 points R@1 on OpenAI d=1536/3072 across 2-bit and 4-bit; +0.3 points on GloVe d=200 4-bit; −1.2 points on GloVe d=200 2-bit; both converge to FAISS by k≈16 (GloVe) or k=4 (OpenAI) [README “Recall”].
Available framework drop-ins: LangChain InMemoryVectorStore, LlamaIndex SimpleVectorStore, Haystack InMemoryDocumentStore, Agno LanceDb — same surface, same persistence semantics [README “Framework integrations”].
Distribution: published to PyPI (turbovec) and crates.io (turbovec); built x86_64 wheels target x86-64-v3 (AVX2 baseline, Haswell 2013+), with the AVX-512 kernel runtime-gated via is_x86_feature_detected!.

Why it’s interesting

This is the first public, packaged reproduction of TurboQuant the team has seen outside the original ICLR paper TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate and the RaBitQ-authors’ technical-note rebuttal Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments — and it’s pragmatically interesting because the implementation already concedes the central point of the rebuttal: it adopts RaBitQ’s per-vector length-renormalization at encode time (cited explicitly in the README References) on top of TurboQuant’s data-oblivious rotation+Lloyd-Max scheme. The 12–20% ARM speedup over FAISS IndexPQFastScan is also a useful symmetric-baseline datapoint of the kind Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments argued was missing from the original TurboQuant claims, since FAISS FastScan is the same reference both papers compare to. For a local-RAG / private-embedding-index use case (no managed service, no codebook training, no rebuild as the corpus grows) this is the most directly usable artifact tracked under LLM Inference Efficiency so far — complementing the KV-cache quantization thread on that page rather than the speculative-decoding thread.