1.5 TB of VRAM on Mac Studio — RDMA over Thunderbolt 5

TL;DR

Jeff Geerling benchmarks a four-node M3 Ultra Mac Studio cluster (1.5 TB total unified memory, ~$40k) running large MoE LLMs via Exo 1.0 and llama.cpp, using a new macOS 26.2 feature: RDMA over Thunderbolt 5. RDMA drops inter-node memory access latency from ~300 μs to <50 μs, letting Exo scale up tokens/sec as nodes are added — opposite to llama.cpp’s RPC layer-sharding, which slows with more nodes. The cluster runs Kimi K2 Thinking (1T params, 32B active) at ~30 tok/s and Qwen3-235B at ~32 tok/s on four nodes. The piece is the most concrete public datapoint on whether Mac Studios are a viable shared-office inference rig for frontier open models.

Key claims

macOS 26.2 adds an rdma_ctl enable recovery-mode toggle that lights up RDMA over Thunderbolt 5, dropping inter-node memory access latency from ~300 μs to <50 μs (per MLX PR #2808) [§“Enabling RDMA”].
Exo 1.0 (Apache 2.0, released same day as the post) is the only clustering tool that currently uses RDMA; llama.cpp still uses an RPC layer-sharding scheme [§“Enabling RDMA”].
On Qwen3-235B, Exo speeds up monotonically with more nodes (up to ~32 tok/s on four), while llama.cpp’s RPC scheme slows down as nodes are added — the headline scaling difference [§“Enabling RDMA”, Qwen3-235B chart].
Kimi K2 Thinking (1T total / 32B active) at 600+ GB on disk fits on the cluster (not on a single Mac) and runs at ~30 tok/s [§“Enabling RDMA”].
A single M3 Ultra Mac Studio (~$10k) outperforms a 4-node Framework Desktop (AMD AI Max+ 395) cluster at half the power, and beats a 2-node Dell GB10 (DGX Spark equivalent) cluster while having double the memory [§“M3 Ultra Mac Studio - Baseline”].
M3 Ultra is the first small desktop the author has tested to break 1 TFLOP FP64 on HPL (~1.3 TFLOPS single-node, 3.7 TFLOPS for the asymmetric 4-node cluster — less than 4× because the top two Macs have half the RAM) [§“HPL and Llama.cpp”].
Thunderbolt 5 beats 2.5 GbE for llama.cpp inference even without RDMA (TCP-only); HPL over Thunderbolt currently crashes the Macs [§“HPL and Llama.cpp”].
Thunderbolt 5 switches don’t exist, so clustering tops out at four Macs in full-mesh; Apple states all five TB5 ports are RDMA-enabled, leaving room to grow if mesh topology can be extended [§“A Mini Mac Rack”, §“Stability Issues”].
Idle power draw of the M3 Ultra Mac Studio is under 10 W; the cluster runs near-silent under 250 W per node [§“M3 Ultra Mac Studio - Baseline”, §intro].

Method

The author boots four M3 Ultra Mac Studios (two 512 GB / two 256 GB, totaling 1.5 TB unified memory) in a DeskPi TL1 mini-rack, cross-cables all four in a Thunderbolt 5 full mesh (no TB5 switches exist), and enables RDMA per node via rdma_ctl enable from macOS recovery. He benchmarks (a) Geekbench / HPL FP64 for raw compute, (b) llama.cpp over 2.5 GbE vs Thunderbolt 5 TCP, and (c) Exo 1.0 with RDMA on three model sizes — Qwen3-235B, DeepSeek V3.1 (671B), and Kimi K2 Thinking (1T). Comparisons are run against a 2-node Dell GB10 (DGX Spark–class) cluster and a 4-node Framework Desktop (AMD AI Max+ 395) cluster.

Results

Single M3 Ultra HPL: ~1.3 TFLOPS FP64; four-node asymmetric cluster: 3.7 TFLOPS.
Qwen3-235B on Exo: scales near-linearly to ~32 tok/s at four nodes; llama.cpp RPC slows with added nodes.
DeepSeek V3.1 (671B): llama.cpp shows a small two-node speedup (network overhead apparently tolerable at 2 nodes).
Kimi K2 Thinking (1T / A32B): ~30 tok/s on the four-node cluster — does not fit on any single node.
Power: <10 W idle per Mac Studio, <250 W under load.
Single M3 Ultra beats a 4× Framework AI Max+ 395 cluster at half the power.

Why it’s interesting

This is the cheapest currently-available way to run frontier-scale open MoE models (Kimi K2 Thinking, DeepSeek V3.1) at usable interactive speeds in a single small office, and it’s the first public benchmark of Apple’s new RDMA-over-Thunderbolt path with Exo. For a generative-models team like Luma’s that mostly cares about training, this is mostly a deployment / on-prem-inference datapoint — but the asymmetric scaling result (Exo speeds up, llama.cpp slows down with nodes) is a clean illustration of why interconnect latency dominates MoE-distributed inference once weights spill across nodes. The stability caveats (prerelease software, Macs crashing on HPL-over-TB, 4-node mesh ceiling, no SSH-driven OS upgrades) make this not yet a hands-off appliance.