FaaScale: Unlocking Fast LLM Scaling for Serverless Inference

TL;DR

FaaScale is a serverless LLM inference platform (MLSys 2026) that attacks the cold-start problem under bursty traffic by co-designing model multicast with cross-node pipeline-parallel execution. Production LLM traces show request rates spiking >10× within minutes against models whose cold loads take tens of seconds (host-memory cache miss rates of 36–64% in their replay), while SSD→GPU load for Llama-70B exceeds 30 s. Their core scheme, PipeCast, multicasts fine-grained model blocks via a binomial-pipeline topology and dynamically forms inference pipelines across the receiving nodes while transfer is still in progress, so requests begin generating tokens before any single node holds the full model. End-to-end, FaaScale reports up to 5× lower P90 TTFT and 17.8–31.3% GPU cost reduction versus ServerlessLLM, FaaSNet, and NCCL on real LLM traces.

Key claims

Production serverless LLM workloads are simultaneously bursty (≥10× request-rate swings within minutes), heavy (~140 GB for Llama-70B, multi-minute cold loads), and long-tailed across tenants (one cited cloud hosts ~4,000 fine-tuned LLMs against ~20 TB aggregate CPU cache) — making both remote loading and host-memory caching insufficient [§1, §2.2, §2.3].
Under a realistic multi-tenant cache configuration (3 models in host memory, 12 in SSD per node, 1 req/min per model, LRU), >95% of model instances are evicted from memory within 15 s, and SSD cache misses account for 36% and 64% of loads across two replayed traces — collapsing the host-memory-cache strategy [§2.3, Fig. 2, Fig. 3].
The design principle is pipelined multicast inference: distribute model blocks via a binomial-pipeline multicast over an RDMA/GDR fabric, and concurrently form cross-node pipeline-parallel execution groups so that the first complete model copy is reconstructable from partial transfers and inference can start before transfer finishes [§3, §4.1, Fig. 4].
PipeCast’s k→N scaling generalization partitions destination nodes into k sub-groups, each running an independent 1→L binomial-pipeline multicast, and uses a k-way circular-shift block transfer order so complementary blocks land on different sub-groups in parallel — making the first complete model assembleable across nodes after only b/k time steps instead of b [§4.2, Algorithm 1, Fig. 5].
Block size b is selected offline to sit at the “elbow” of the transmission-time curve: smaller blocks hurt total transfer time, larger blocks add pipeline-execution overhead from intermediate-tensor exchange; the paper treats the optimum as stable across their tested RDMA networks and leaves volatile-network adaptive tuning as future work [§4.2].
Execution pipelines are generated incrementally and prefer cross-sub-group composition (one node from each available sub-group) over within-sub-group pipelines, on the basis that this maximizes how soon a full set of complementary blocks is co-resident on some pipeline [§4.3, Algorithm 2].
Once a node has received the full model replica, PipeCast switches it from pipelined cross-node execution to local execution mode to eliminate the inter-node activation-exchange overhead — i.e., the cross-node pipeline is a transient bootstrap, not the steady state [§4.1].
System-side designs that make PipeCast usable in practice: a multi-level (GPU / host-memory / SSD) locality-driven startup mechanism that lets GPU-resident and host-cached replicas cooperate during a scale-out, and a unified memory manager that consolidates intra-block tensors for bulk RDMA transfer plus pre-allocates GPU memory to remove runtime allocation cost [§5].
End-to-end results on real-world LLM traces: 2.4×–5× P90 TTFT improvement and 17.8%–31.3% GPU cost reduction versus ServerlessLLM, FaaSNet, and NCCL baselines [§7, Abstract].

Method

The system has a cluster manager (request dispatcher + resource manager + model-scaling controller + pipelined-execution controller) and per-node node controllers + model managers; data movement uses GPUDirect RDMA so model blocks flow directly between GPUs across nodes, with RDMA also used to read models from remote host memory.

PipeCast’s transfer plan is a binomial-pipeline multicast: nodes are logically organized into a hypercube, each step every active node forwards one block to a neighbor, and N receivers are saturated in O(log N) steps. For k→N (where k≥1 source nodes already hold the full model — guaranteed in practice by keeping at least one host-memory replica somewhere in the cluster), the N nodes are split into k equal sub-groups each running an independent 1→L multicast. The k-way transmission strategy partitions model blocks into k equal chunks and assigns each sub-group a circularly-shifted ordering, so complementary chunks are in flight simultaneously across sub-groups.

The pipeline-execution controller continuously watches block-arrival state across all nodes and, as soon as some subset of nodes collectively holds all blocks, materializes a dynamic execution pipeline on them and starts serving queued requests. Algorithm 2 prefers cross-sub-group pipelines (which exploit the circular-shift complementarity) and only falls back to a single-sub-group pipeline when only one sub-group still has unassigned nodes. Once a node finishes receiving the full model, it leaves its cross-node pipeline and serves locally; the cluster manager rebalances dispatched requests accordingly.

Model-block boundaries are layer-aligned (each block is a contiguous set of transformer blocks plus their tensors), and the model manager packs intra-block tensors into a single contiguous buffer per block so the RDMA send is one bulk transfer rather than many small transfers — this is what gives the bandwidth-utilization numbers their headroom against NCCL’s all-reduce-tree baseline.

Results

The paper reports against three baselines: ServerlessLLM (host-memory caching with inference-aware loading), FaaSNet (binary-tree-based image multicast), and NCCL (the standard collective communication library, configured for tree-broadcast). On real-world LLM serverless traces (BurstGPT and the anonymized XYZ trace), FaaScale achieves 2.4×–5× P90 TTFT improvement and 17.8%–31.3% GPU cost reduction. The cost-reduction number is the headline serverless metric — it accounts for both reduced over-provisioning (PipeCast scales fast enough that platforms can run with fewer pre-warmed instances) and shorter cold-start tails (fewer SLO-padding instances needed).

The §2.3 motivating measurement — SSD→GPU load of Llama-70B taking 30+ s even with optimized implementations, an order of magnitude slower than host→GPU load — is the lower bound that any caching-only strategy hits, and is what FaaScale’s pipelined-multicast approach is designed to undercut by avoiding the per-node “must hold full model before starting” requirement.

Why it’s interesting

FaaScale is a different point in the LLM inference efficiency design space than what the wiki has filed so far. The existing entries on LLM Inference Efficiency are almost all about making a single already-loaded inference instance go faster — speculative decoding (Speculative Decoding: Performance or Illusion?, Speculative Speculative Decoding), KV-cache quantization (Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments), prefix-cacheable diffusion LMs (WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference), or weight quantization (1-bit Bonsai 8B: End-to-End 1-bit Language Models from PrismML). FaaScale instead targets the cold-start tail under multi-tenant serverless deployment, where the bottleneck is moving 100s of GB of weights to where the request needs them, not running a forward pass faster. The natural companion is Interaction Models: A Scalable Approach to Human-AI Collaboration, which surfaces a related “frequent small prefills + decodes” regime that the existing speculative-decoding stack also doesn’t address.

The mechanism — cross-node pipeline parallelism formed dynamically during weight transfer — also connects to Distributed training parallelism‘s recurring theme that non-uniformity is the missing primitive: FaaScale treats inference instances as dynamically-composed cross-node pipelines whose composition changes mid-transfer, which is structurally similar to SpectraX — True MPMD Pipeline Parallelism for JAX‘s heterogeneous MPMD pipelines and to KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training‘s per-step compute-bag re-routing, but applied to the inference-serving side rather than training. The pipelined-multicast idea also has a methodological cousin in TorchSpec: Speculative Decoding Training at Scale‘s disaggregated inference-cluster ↔ training-cluster RDMA streaming via Mooncake — both are betting that the right primitive in modern GPU clusters is fine-grained streaming over RDMA rather than treating large weight blobs as monolithic objects.