The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

ListenThe Physics of Planning: What Pretraining Puts There and What Distillation Can Move9:35

Show notes

A controlled "Physics of Language Models"-style study of how multi-turn long-horizon planning is acquired in pretraining, shaped in post-training, and integrated across teachers. The authors build a purpose-built agentic environment to isolate three stages, and use a mutual-information split between general "planning patterns" and task-specific "planning knowledge" to explain when OPD beats GRPO and when multi-teacher distillation transfers versus catastrophically interferes. We unpack the mechanism and what it might mean for how we think about training image and video generation models. **Sources** - The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation (Men, Jin, Liu, Zhao)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

A “Physics of Language Models”-style controlled study of how multi-turn long-horizon planning is acquired, shaped, and integrated across the pretraining → post-training pipeline for foundation-model agents. Using a purpose-built multi-turn environment where task structure, data quality, and teacher provenance can be independently manipulated, the paper isolates three stages: (1) which pretraining data formats produce compositional long-horizon generalization; (2) when GRPO vs. OPD (on-policy distillation) each help vs. hurt during post-training; and (3) how multi-teacher OPD (MOPD) integrates capabilities across environments. Headline mechanistic findings: explicit CoT world-model construction during pretraining beats atomic-skill datasets; OPD has a broader effective region than GRPO under low-quality / long-horizon data; and MOPD generalization is governed by whether teachers share compatible planning patterns — compatible → cross-environment transfer, partially shared → continual learning, conflicting → catastrophic interference.

Key claims

Atomic skills alone are insufficient for compositional generalization at long horizons; a small amount of long-horizon trajectory data is required during pretraining to unlock compositional planning [§ Abstract].
Explicit world-model construction via CoT state-transition modeling during pretraining yields stronger long-horizon generalization than trajectory-only formats [§ Abstract].
Suboptimal trajectories in pretraining data severely degrade long-horizon performance because errors amplify across turns [§ Abstract].
Planning ability decomposes via mutual information into general planning patterns and task-specific planning knowledge — two axes that respond differently to post-training [§ Abstract].
For planning patterns, post-training has three application regions — unnecessary, effective, unsupported — and OPD’s effective region is broader than GRPO’s under low-quality and long-horizon settings, attributed to OPD providing more consistent update directions [§ Abstract].
For planning knowledge, distilling unseen procedures from a teacher with different knowledge can impair the student’s prior world modeling without fully installing the new knowledge — a distillation failure mode distinct from the same-origin-teacher failure of standard OPD [§ Abstract].
Multi-teacher on-policy distillation (MOPD) integrates capabilities by converging to a shared planning pattern across environments; the outcome depends on pattern compatibility: compatible → cross-environment generalization, partially shared → continual learning support, completely conflicting → severe interference [§ Abstract].

Method

The paper builds a unified, controlled multi-turn environment where the task specification, data-generation policies, teacher provenance, and reward structure can each be independently varied. This lets the authors run matched-scale ablations across three post-training stages that are usually confounded in production pipelines.

Pretraining ablations vary data format (atomic-skill demos vs. long-horizon trajectories vs. CoT state-transition modeling), data distribution (task-mix), and trajectory quality (optimal vs. suboptimal), and measure long-horizon planning generalization.

Single-teacher post-training ablations compare GRPO and OPD across matched conditions, and use per-token mutual information between input state and output tokens to separate what the student learns as a general planning pattern (transferable) from what it learns as task-specific planning knowledge (localized). This decomposition is what lets the paper identify the three application regions of post-training (unnecessary / effective / unsupported).

Multi-teacher post-training ablations run MOPD with teachers trained on different environments, measure the compatibility of their planning patterns via the same MI-based decomposition, and characterize the resulting student across the compatible / partial / conflicting regimes.

Results

Pretraining: CoT state-transition modeling produces stronger long-horizon generalization than trajectory-only training; atomic skills alone fail to compose; a small quantity of long-horizon data unlocks compositional planning; suboptimal trajectories cause disproportionate long-horizon degradation [§ Abstract].
Single-teacher post-training: OPD has a broader effective region than GRPO under low-quality / long-horizon conditions, with more consistent update directions across the trajectory [§ Abstract]. Distillation of unseen procedures can impair the student’s prior world modeling — a mid-training regression the paper attributes to knowledge mismatch rather than teacher strength.
MOPD: Converges to a shared planning pattern across environments; compatible patterns → cross-environment generalization; partially shared → continual learning; conflicting patterns → severe interference [§ Abstract].

Why it’s interesting

Direct companion to MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training (Xiaomi MiMo’s canonical MOPD paper) and Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe (the OPD failure-mode taxonomy). Where those two papers give empirical recipes and failure diagnoses on production-scale LLMs, this paper builds the controlled testbed that lets you cleanly measure which OPD failure mode is firing and why — including a formal MI-based split between the planning-pattern axis (where OPD’s same-origin constraint bites) and the planning-knowledge axis (where a distinct “unseen-procedure impairs prior world model” failure mode appears). The three-region framing (unnecessary / effective / unsupported) is a sharper diagnostic vocabulary than the wiki currently has for when to reach for GRPO vs OPD; the MOPD compatibility trichotomy (compatible / partial / conflicting) gives a mechanistic story for why MOPD converges cleanly on same-origin multi-domain teachers (MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training §4.4.2) but has never been convincingly demonstrated across genuinely disjoint task families.

Positions naturally against Understanding Reasoning from Pretraining to Post-Training (Shen et al.’s chess-based pretraining→SFT→RL testbed) — both use controlled environments to isolate what pretraining “puts there” that RL then amplifies vs. installs, but this paper’s environment is multi-turn agentic rather than single-shot puzzle-solving, and it targets the OPD family rather than GRPO. Sharpens open questions in Reasoning RL on when OPD dominates GRPO and how MOPD generalizes off-distribution.