Skip to content
Bud's Wiki

BackendBench: Ship Correct and Fast LLM Kernels to PyTorch

TL;DR

Section titled “TL;DR”

BackendBench is a Meta/PyTorch evaluation suite (BSD-3) for testing how well LLMs (and humans) can write PyTorch backends. It lets developers drop custom kernels into an organized directory structure that dynamically overrides PyTorch’s core operators at runtime, producing a fully functional pip-installable backend that existing models can use unchanged. Correctness is gated by PyTorch’s own OpInfo and FACTO test suites across 271 operators; performance is measured against eager on real tensor shapes from popular Hugging Face models (124 operators). The accompanying tweet reports that current LLMs pass 53% of correctness tests with all edge cases and some generated kernels run up to 1.2× faster than eager on real HF shapes.

Key claims

Section titled “Key claims”
  • BackendBench evaluates LLM-generated kernels against PyTorch’s OpInfo and FACTO edge-case suites for 271 operators [README].
  • Performance benchmarks use real tensor shapes from popular Hugging Face models across 124 operators [README].
  • Kernels that pass the BackendBench correctness gate are pitched as upstreamable — the explicit framing is that passing the BackendBench tests indicates a kernel is likely correct enough to merge [README].
  • The framework dynamically overrides PyTorch’s core operators at runtime, so a custom-kernel directory becomes a drop-in backend usable with any existing PyTorch model without modeling-code changes [README].
  • Companion launch claim: LLMs pass 53% of correctness tests with all edge cases, and some generated kernels run up to 1.2× faster than eager on real HF shapes [launch tweet].
  • nanoGPT converges end-to-end with LLM-generated kernels via the BackendBench harness [launch tweet, BackendBenchExamples repo].
  • Author lineup (Mark Saroufim, Jiannan Wang, Bert Maher, Sahan Paliskara, Laura Wang, Shahin Sefati, Manuel Candales) sits inside the Meta PyTorch + GPU MODE org [README citation block].

Method

Section titled “Method”

The repo packages three things. (1) An operator-directory layout: python -m BackendBench.scripts.setup_operator_directories lays out empty op files; the user (or an LLM acting as user) fills them in. At import time these stubs override the corresponding torch.* ops. (2) A multi-suite test harness: --suite smoke for a quick sanity check against an aten backend, --suite opinfo for full edge-case correctness using PyTorch’s own OpInfo test infrastructure (also incorporating FACTO), and --suite torchbench for performance testing against real HF tensor shapes from a curated workload. (3) An upstream path: kernels that pass the correctness gate are intended to be plumbed straight into PyTorch core. A companion repo jiannanWang/BackendBenchExamples shows the full nanoGPT-on-LLM- kernels convergence demo end-to-end.

Results

Section titled “Results”

The repo itself does not publish a leaderboard or table; quantitative claims live in the launch tweet and blog. From the launch tweet: 53% correctness pass rate across LLMs on PyTorch’s edge-case suite, and up to 1.2× speedup over eager on real HF tensor shapes for some generated kernels. nanoGPT converges end-to-end on LLM-generated kernels (the bar this clears is “the kernels are correct enough that gradient flow + optimizer state evolve as in a reference run”, not just per-op correctness). No comparison to torch.compile is reported here — that is the territory of the companion CUDA-Agent line of work CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation, which uses KernelBench rather than BackendBench. The 271-operator / 124-operator surface area is the largest filed evaluation footprint for “LLM as PyTorch backend author” the wiki has seen.

Why it’s interesting

Section titled “Why it’s interesting”

BackendBench is the evaluation harness that the CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation line of work has been implicitly asking for: a correctness gate hardened against the shortcut-exploitation failure modes that ByteDance’s CUDA Agent explicitly engineers around (file-permission-protected verifier scripts, banning torch.nn.functional fallbacks, anti-trivial-output checks). Where CUDA-Agent-Ops-6K builds training problems via combinatorial op composition + KernelBench-similarity exclusion, BackendBench builds an evaluation that uses PyTorch’s own OpInfo test suite — so a kernel that passes is by construction correct against every edge case the PyTorch team has codified. This makes it the natural companion benchmark to KernelBench for the Agentic Software Engineering thread: KernelBench measures “can the LLM beat torch.compile on synthetic kernels?”, BackendBench measures “can the LLM produce a kernel that survives PyTorch’s own conformance bar and plug into a real model?”. It also intersects IO-Aware Kernel Design in an indirect way: the recent kernel-design papers (FA4, Gram-NS, Flash-KMeans, FlashSampling) all justify themselves on torch.compile- vs-handwritten gaps, and BackendBench’s 124-op HF-shape benchmark is exactly the workload distribution where those gaps would surface under automated LLM kernel synthesis.

See also

Section titled “See also”