Our eighth generation TPUs: two chips for the agentic era (TPU 8t and TPU 8i)

TL;DR

Google announces an eighth-generation TPU split into two purpose-built SKUs: TPU 8t for training (compute throughput and inter-chip bandwidth maximized) and TPU 8i for inference (memory bandwidth and on-chip SRAM maximized to serve agentic / reasoning workloads). Both chips run on Google’s own Axion Arm-based CPU hosts and ship with the Virgo network fabric, Pathways/JAX software, and fourth-gen liquid cooling. Headline numbers: TPU 8t pods scale to 9,600 chips and 2 PB of shared HBM with 121 ExaFlops per pod, ~3× per-pod compute over Ironwood; TPU 8i adds 288 GB HBM + 384 MB on-chip SRAM (3× prior gen), doubles ICI bandwidth to 19.2 Tb/s, and claims 80% better perf/$ vs the previous generation. The split-SKU framing — “training vs. inference are now distinct enough to warrant different silicon” — is the load-bearing claim.

Key claims

The eighth-generation TPU ships as two SKUs co-designed with Google DeepMind: TPU 8t for training and TPU 8i for inference, on the explicit thesis that agentic workloads (continuous reason-plan-execute-learn loops, multi-agent swarms) demand a separate point in the design space from training [§“Two chips to meet the moment”].
A single TPU 8t superpod scales to 9,600 chips, 2 PB of shared HBM, and 121 ExaFlops, with 2× the interchip bandwidth and ~3× the per-pod compute of the previous generation (Ironwood); the Virgo network plus JAX/Pathways is claimed to enable near-linear scaling to one million chips in a single logical cluster [§“TPU 8t: The training powerhouse”].
TPU 8t targets >97% “goodput” (productive compute time) via real-time telemetry across tens of thousands of chips, automatic rerouting around faulty ICI links without job interruption, and Optical Circuit Switching that reconfigures around failures with no human intervention [§“TPU 8t: The training powerhouse”].
TPU 8i pairs 288 GB HBM with 384 MB on-chip SRAM (3× the previous generation) and uses a NUMA-isolated Axion CPU host topology — the on-chip SRAM is explicitly sized to hold the KV-cache footprint of reasoning models at production scale [§“TPU 8i: The reasoning engine”, §“Co-designed for Gemini, open for everyone”].
For MoE inference, TPU 8i doubles ICI bandwidth to 19.2 Tb/s and introduces a hierarchical “Boardfly” topology (4-chip fully-connected blocks → 8-board fully-connected groups → 36 groups fully connected at the pod level) that the post says reduces the maximum network diameter by >50% [§“TPU 8i: The reasoning engine”, embedded topology figure].
A new on-chip Collectives Acceleration Engine (CAE) offloads global collective operations and is claimed to reduce on-chip collective latency by up to 5× [§“TPU 8i: The reasoning engine”].
Both chips run native JAX, MaxText, PyTorch, SGLang, and vLLM, expose bare-metal access without virtualization, and are paired with open-source MaxText reference implementations and a “Tunix” RL library [§“Co-designed for Gemini, open for everyone”].
Both chips claim up to 2× perf/W over Ironwood, fourth-gen liquid cooling, integrated on-chip network connectivity (to cut data-movement power across the pod), and a system-level claim of 6× more compute per unit of electricity than Google data centers delivered five years ago [§“Designing for power efficiency at scale”].
TPU 8i headline: 80% better performance-per-dollar vs the previous generation, enabling ~2× the customer volume at the same cost [§“TPU 8i: The reasoning engine”].

Method

The post is a product announcement, not a technical paper, so the “method” is the design philosophy. Google’s argument is that hardware development lead times (years) force them to bet on which workloads will dominate by ship date. Several years ago they decided inference and agent serving would diverge sharply enough from training that one chip could no longer serve both well, and the eighth generation is the realization of that bet: split SKUs whose memory hierarchy, interconnect topology, and host CPU are individually tuned.

For TPU 8t the design point is arithmetic + scale: maximize compute throughput, inter-chip bandwidth, and shared-memory pool size, and engineer for goodput rather than peak. Storage access is 10× faster (TPUDirect pulls data straight into the TPU), and the Virgo fabric plus OCS-based fault recovery target the “every percentage point is days of active training time at frontier scale” failure mode.

For TPU 8i the design point is memory + topology for many small bursts: HBM and SRAM are sized to keep a reasoning model’s active working set on-chip; the Boardfly topology compresses network diameter so that the all-to-all-style traffic patterns of MoE experts and multi-agent swarms don’t pay long-tail latency; CAE offloads collectives off the matrix engines. The Axion ARM CPU host is doubled per server and runs NUMA-isolated, giving each agent / replica its own deterministic memory locality.

Results

The post asserts headline numbers without methodology details. TPU 8t: 9,600 chips per superpod, 2 PB shared HBM, 121 ExaFlops, ~3× per-pod compute vs Ironwood, 2× ICI bandwidth, claimed near-linear scaling to 1M chips, target >97% goodput. TPU 8i: 288 GB HBM + 384 MB SRAM (3× SRAM vs prior gen), 19.2 Tb/s ICI (2× prior), up to 5× lower on-chip collective latency via CAE, >50% reduction in network diameter from Boardfly, 80% better perf/$ vs prior generation. System-level: up to 2× perf/W vs Ironwood across both chips, fourth-gen liquid cooling, 6× more compute per unit of electricity vs Google data centers five years ago. Both chips ship “later this year” (2026) as part of AI Hypercomputer.

Why it’s interesting

The split-SKU framing matters for anyone building or serving foundation models. Most of the wiki’s serving-side work — Better MoE model inference with warp decode on warp-decode for MoE, Enabling Up to 41% Faster Pre-training: MXFP8 and DeepEP for DeepSeek-V3 on B200 with TorchTitan on MXFP8 + DeepEP for DeepSeek-V3 on B200, FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling on B200’s softmax-bottlenecked attention — is GPU-centric and assumes one chip family serves both training and inference. TPU 8t/8i is the public artifact of the opposite bet: that the memory/compute/topology balance for “swarms of reasoning agents” is different enough from “frontier pretraining” that a single SKU loses on both axes. The 384 MB on-chip SRAM target — explicitly sized for KV-cache footprint of reasoning models — is the most concrete instance of that thesis and pairs naturally with the KV-cache compression work in Inference-Time Hyper-Scaling with KV Cache Compression and the on-chip-state engineering of MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens. The Boardfly topology’s network-diameter compression is also a direct counterpart to the all-to-all bottleneck that Better MoE model inference with warp decode and the MoE-serving line of work hit on GPU clusters.