InstinctFlash — High-Performance Serving Runtime for Robotics Models

TL;DR

InstinctFlash (General Instinct, YC P26, AGPL-3.0) is a unified open-source serving runtime for robotics models — VLAs (π0.5, LingBot-VLA-4B, LingBot-VLA-V2-6B, GR00T N1.7), video-action WAMs (LingBot-VA), and video-world/policy models (Cosmos3 Edge/Nano DROID, DreamZero DROID) — that lets eight model families ride one Runtime API on Jetson Thor, RTX 5090, and RTX 4090. The framing bet is that end-to-end robot policies do not fit on the edge under naive PyTorch: VLAs are already slow, WAMs take 20–50 denoising steps per action chunk, and the vendor-native serving paths are per-model one-offs. Reported Jetson Thor speedups vs PyTorch reach 33.78× for LingBot-VA when combining FP8 with a 25V/50A → 2V/4A step schedule, 7.88× for π0.5 at FP8, and 3.24×/2.13× for Cosmos3 Edge/Nano DROID (NUMERIC compile + UniPC4 / CFG3); GR00T N1.7 is a low-headroom outlier at 1.19× under strict BITEXACT. The architecture is layered by what each optimization changes — MODEL / GRAPH / CACHE / ATTENTION / KERNEL / HARDWARE — with a planner that reads the checkpoint declaration, decides which passes are provably valid, and shows its work. This is now the third VLA-runtime release on the wiki, after Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots (llama.cpp-shaped C++/GGUF portable runtime) and APXinf-robo — Edge Inference Engine for VLA/WAM Models on Jetson Thor / Orin (Rust engine with hardware-locked FP8/INT8 kernels), and the first to span VLAs, WAMs, and video-world policies under one Runtime API.

Key claims

Eight model families ride one Runtime API across Jetson Thor, RTX 5090, and RTX 4090: LingBot-VA (5B WAM), LingBot-VLA-4B, LingBot-VLA-V2-6B, π0.5, GR00T-N1.7-3B, Cosmos3 Edge/Nano DROID, and DreamZero DROID [README §Load a model].
Headline 33.78× LingBot-VA speedup on Jetson Thor comes from stacking FP8 with a 2V/4A sampling schedule vs the native 25V/50A PyTorch baseline (15506.32 ms → 459.10 ms P50) [README §Results, Notes].
Per-family single-schedule speedups on Jetson Thor P50: π0.5 FP8 7.88× (408.58 → 51.85 ms), LingBot-VA FP8 25V/50A 5.36×, LingBot-VA FP8 2V/4A 4.51×, Cosmos3 Edge DROID NUMERIC UniPC4/CFG3 3.24×, LingBot-VLA-4B FP8 2.82×, Cosmos3 Nano DROID NUMERIC UniPC4/CFG3 2.13×, DreamZero DROID FP8 16-step + dynamic cache 1.98×, LingBot-VLA-V2-6B FP8 1.86×, GR00T N1.7 BITEXACT 1.19× [README §Results, measured 2026-09-15].
Precision policy is a strict three-tier lattice, not a per-model flag. The Runtime defaults to precision="native" with a BITEXACT transformation ceiling; tier_ceiling="numeric" permits numerical changes; precision="fp8" (CLI: --fp8) explicitly enables FP8. DreamZero’s dynamic-step cache requires tier_ceiling="behavioral" [README §Load a model, “precision policy” link, §Get actions].
OpenPI wire compatibility means unmodified openpi-client connects unchanged — serve hosts the same Runtime behind msgpack-over-websocket, so existing pi0/openpi robot-side clients don’t change [README §Get actions, “Over the network”].
Optimization is organized in six layers by what each one changes — MODEL (distillation, step reduction, checkpoint compression via InstinctCompress and instinct-pdd), GRAPH (prefill extraction, CUDA-graph capture, memory planning), CACHE (KV reuse, cross-attention and episode caches), ATTENTION (FlashAttention, hybrid and linear attention), KERNEL (backend/layout dispatch, fusion), HARDWARE (fp8/int8, TensorRT, Jetson-class edge devices) [README §Architecture].
The planner decides which optimizations are provably valid for the loaded weights without a GPU or weights, then applies them — the pipeline is checkpoint → adapter → planner → engine passes → actions, with the adapter declaring what the model is and the planner deciding what is safe to change [README §Architecture].
Serving surface has four flags: --serve.dry_run (preflight only, no weights/GPU), --serve.smoke (load, produce one action, exit), --serve.seed (paired-comparison seeding; rejected under FP8 serving), --serve.viz (stream observations/actions/latency to a Rerun viewer) [README §Get actions].
A validate <dir> verb certifies non-inferiority given --validate.teacher_outcomes/.student_outcomes/.margin, stamping the certificate into the package — a shippability contract on distilled/quantized packages, not just latency numbers [README §Get actions].
Fine-tuned-checkpoint-first UX: instinctflash serve /path/to/your/checkpoint detects the family, writes an instinctflash.json declaration from what the checkpoint proves, and starts serving; anything unprovable is asked explicitly, never guessed [README §Load a model, “Your fine-tuned checkpoint”].
External adapter registration via instinctflash.adapters entry point makes model-family support pluggable — new model families are pip installed rather than forked [README §Add a model, examples/external_plugin/].
Companion tweet reports a 5.75× wall-clock advantage over GPT-6 Astra on the same task under detailed prompting + ground-truth trajectory + video, with Astra failing to one-shot the task after 4:48 [Companion tweet, 2102442309938024765]. Vendor-reported, single side-by-side, no protocol disclosed — carries the standard Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments symmetric-baseline caveat.

Method

InstinctFlash factors robotics-model serving into an inspectable pipeline checkpoint → adapter → planner → engine passes → actions. The adapter declares what a model is (family, tokenizer, image/state/action encoding, schedule); the planner decides which optimizations are provably valid for those weights without loading them or touching a GPU; the engine passes apply and measure each optimization. The six-layer optimization taxonomy is by what each pass changes rather than by priority: MODEL changes the weights (distillation, step reduction, checkpoint compression) and lives in the companion InstinctCompress and instinct-pdd repos; GRAPH changes when work is issued (prefill extraction, CUDA-graph capture, memory planning); CACHE changes what is recomputed (KV reuse, cross-attention and episode caches); ATTENTION changes how tokens mix (FlashAttention, hybrid/linear attention); KERNEL changes how a kernel is written (backend/layout dispatch, fusion); HARDWARE changes what it executes on (fp8/int8, TensorRT, Jetson-class edge devices under serving/). The runtime measures where time actually goes and starts there.

Precision is a strict three-tier lattice enforced by the planner: BITEXACT (the default, precision="native"), NUMERIC (tier_ceiling="numeric" — numerical changes allowed, e.g. shared BF16 fusion, exact caching), and BEHAVIORAL (tier_ceiling="behavioral" — required for e.g. DreamZero’s dynamic step cache). FP8 is a separate explicit selection (--fp8) with its own validation surface, and --serve.seed is rejected under FP8 to avoid meaningless paired seeds under quantization-induced nondeterminism. The Python API is deliberately minimal: Runtime.from_pretrained(path_or_hub_id) for load, then with runtime.episode(prompt=...) as episode: episode.predict(observation) for control. For LingBot-VA the caller passes executed_action=... so the next chunk is conditioned on the actions the controller actually executed rather than what the model predicted — direct plumbing for the async-inference contract that Real-Time Robot Execution with Masked Action Chunking REMAC and πR²: Reactive Real-time Flow Policies πR² diagnose as the load-bearing serving failure mode for flow-matching VLAs. Over the network, serve hosts the same Runtime behind msgpack-over-websocket in the pi0/openpi wire format — existing openpi-client code connects unchanged, and the prompt rides in the observation with {"reset": True, ...} starting a new episode. The instinctflash validate <dir> verb certifies non-inferiority (teacher vs student outcomes at a specified margin) and stamps the certificate into the checkpoint package.

The install path pins one environment per model family (Python 3.12 for va, vla4, vla2, pi05, groot; Python 3.13 for edge, nano, dreamzero) so upstream weight repos and their compat patches never collide. Model families are declared through instinctflash.adapters entry points and pip installed rather than forked into the runtime tree.

Results

All P50 latencies on Jetson Thor, measured 2026-09-15 [README §Results]. Speedups are vs the same schedule under PyTorch unless noted.

Model	Acceleration line	PyTorch (ms)	InstinctFlash (ms)	Speedup
LingBot-VA	FP8 · 25V/50A	15506.32	2891.74	5.36×
↳ LingBot-VA	FP8 · 2V/4A	2071.29	459.10	4.51×
LingBot-VLA-4B	FP8	624.22	221.53	2.82×
LingBot-VLA-V2-6B	FP8	734.56	394.11	1.86×
Cosmos3 Edge DROID	NUMERIC · UniPC4 / CFG3	3393.78	1048.01	3.24×
Cosmos3 Nano DROID	NUMERIC · UniPC4 / CFG3	10184.68	4772.38	2.13×
pi05	FP8	408.58	51.85	7.88×
GR00T N1.7	BITEXACT	139.50	117.30	1.19×
DreamZero DROID	FP8 · 16 steps · dynamic cache	23563.08	11899.42	1.98×

The 33.78× headline is LingBot-VA moving from PyTorch native 25V/50A (15506.32 ms) to InstinctFlash FP8 2V/4A (459.10 ms) — combining runtime optimizations with a distilled few-step schedule, not a per-schedule speedup [README §Results, Notes].
Companion tweet reports LingBot-VA runtime-only speedups of 1.2×–7.9× and full 33.78× when composing with the few-step scheduler; π0.5 sits near the middle of that range at 7.88× runtime-only [Companion tweet 2102442308151255285].
GR00T N1.7 is a low-headroom outlier at 1.19× under strict BITEXACT — evidence that not every VLA has room to move under BITEXACT precision, and that headroom is a function of the base model’s kernel efficiency and step count [README §Results].
Astra head-to-head: General Instinct’s own GI0 model (diffusion-transformer VLA generating actions conditioned on predicted future videos) running locally on Jetson Thor is reported at 5.75× wall-clock advantage over GPT-6 Astra on the same task, with Astra fed a ~300-word prompt + ground-truth teleoperated trajectory + video and still failing to one-shot after 4:48 [Companion tweet 2102442309938024765]. Single side-by-side, no per-attempt success stats — vendor-reported.
Reproduction surface: benchmarks/regression/reproduce and benchmarks/regression/serve_smoke produce checked JSON/CSV reports and full action arrays; benchmarks/regression/FRAMEWORK_COMPARISON.rst documents pinned recipes for cross-framework comparison [README §Benchmark].
License is AGPL-3.0 — a distinct licensing posture from the Apache-2.0 Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots Embodied.cpp and open-license APXinf-robo — Edge Inference Engine for VLA/WAM Models on Jetson Thor / Orin APXinf-robo, and one that affects downstream use in closed-weight deployments.

Why it’s interesting

InstinctFlash is the third VLA-runtime release on the wiki and stakes out a distinct point on the substrate axis vs the two priors. Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots Embodied.cpp went the llama.cpp route (Apache-2.0 C++, portable GGUF, one backend spanning CPU/CUDA/NPU); APXinf-robo — Edge Inference Engine for VLA/WAM Models on Jetson Thor / Orin APXinf-robo went the TensorRT-LLM route (Rust, hardware-locked kernels, one model family well). InstinctFlash goes the planner-and-optimization-DAG route (AGPL-3.0, Python + Rust vendor envs, planner picks safe passes per checkpoint, eight families on one Runtime API, precision as an explicit three-tier lattice). The three occupy productively different points: Embodied.cpp maximizes portability and minimizes assumptions; APXinf-robo maximizes per-family Jetson performance; InstinctFlash maximizes model-family coverage under one API + auditable precision policy. Also all three converge on the same OpenPI wire protocol — unmodified openpi-client connects unchanged to any of them — which is a small but load-bearing datapoint that the client-side interface is where the VLA-serving field is standardizing first.

The six-layer optimization taxonomy (MODEL / GRAPH / CACHE / ATTENTION / KERNEL / HARDWARE) directly imports the LLM Inference Efficiency and Diffusion serving optimization design vocabulary into the robot-policy regime, and makes explicit a distinction that page has been carrying implicitly: MODEL-layer changes produce a new checkpoint (distillation, step reduction — the Diffusion Distillation axis) and live in the companion InstinctCompress / instinct-pdd repos; the other five change how the weights execute and produce a plan, which is what the Runtime in this repo owns. The 33.78× LingBot-VA number is the composition of both — 5.36× is runtime-only, and 6.3× comes from 25V/50A → 2V/4A distillation — exactly the “algorithmic × systems levers compose” prediction that Diffusion serving optimization‘s open questions have been asking for empirical evidence on. It also joins C³ache: Accelerating World Action Models with Cross Inference Chunk Cache C³ache as a runtime-side answer to WAM-latency, but attacked at the engine-pass level (planner-decided FP8 + step reduction + graph capture) rather than at the residual-cache level.

The validate non-inferiority certificate is unusual and worth flagging — it’s the strictest shippability contract yet filed on a VLA-serving repo, and matches the “distribution-matching correctness gate” that Speeding up end-to-end inference with self-improving agents Asari AI Labs argued was the missing piece for LLM-serving agents. The precision-tier lattice (BITEXACT / NUMERIC / BEHAVIORAL) makes explicit what the LLM-inference-efficiency literature has been arguing implicitly since Speculative Decoding: Performance or Illusion? — that speedups under quantization or scheduler change require an explicit correctness contract, and that different downstream users tolerate different tiers. Extending that to VLAs and WAMs — where “correctness” is a closed-loop success rate rather than token perplexity — is the pattern the field has been converging toward and this release is the first to encode as a first-class option surface.

Two open questions the release raises but doesn’t answer:

No closed-loop task-success numbers. Unlike APXinf-robo’s 500-episode LIBERO-10 evaluation (92.0–92.8% across three precisions vs the pi05_libero_base 92.4% reference), InstinctFlash reports only P50 latency + “no observed loss in task performance in our real-robot tests” and defers to the validate certificate mechanism. Whether FP8 + 2V/4A LingBot-VA preserves policy quality at the same margin APXinf-robo demonstrated is not established here.
AGPL-3.0 vs Apache-2.0. Embodied.cpp is Apache-2.0 and APXinf-robo is under a mixed permissive license; InstinctFlash is AGPL-3.0. For the robotics-startup regime this concept page tracks — where the closed-weight VLA + open-runtime pattern is common — AGPL’s network-copyleft clause is a materially different distribution surface, and the field hasn’t converged.