FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

TL;DR

FACT argues the missing signal when scaling dexterous manipulation from cheap human-video data is force, not more hours or more views. The Stanford team (Fei-Fei Li, Jiajun Wu, Huang Huang + collaborators) releases EgoForce — a large-scale human manipulation corpus with recorded contact forces — and trains a two-stage foundation model: a Track-Guided Coarse Planner jointly predicts actions, future video, and 2D visual tracks (used as an embodiment-agnostic task representation for cross-embodiment transfer), then a Force-Action Joint Refiner co-predicts future force and uses it to correct the coarse action. On ten force-sensitive tasks (tool use, musical instruments, precision lab procedures), fine-tuning with just 10 robot + 90 human trajectories beats baselines by up to 60 success-rate points, and ablations isolate force supervision in the human pretraining data — not architecture or video co-training — as what carries the gain.

Key claims

Force supervision in the human pretraining data is the load-bearing ingredient for downstream robot force-sensitive task performance after fine-tuning; removing it from the human stream drops task success substantially while removing track or future-video prediction has smaller effects [§Ablations].
Pairing a coarse action planner (actions + future video + 2D visual tracks) with a force-aware refiner (predicts future force + corrects the coarse action) lets anticipated force and action shape each other during refinement, which the paper frames as the correct factorization for contact-rich tasks [§Method].
Visual tracks serve as an embodiment-agnostic task representation shared between human and robot domains that facilitates cross-embodiment knowledge transfer and coarse action generation on top of future-video prediction [§Method].
On ten tasks requiring fine-grained coordination and force feedback (tool use, musical instruments, precision laboratory procedures), fine-tuning with as few as 10 robot + 90 human trajectories outperforms T-Rex by up to 60 percentage points in success rate [§Experiments].
Varying the human:robot data ratio during fine-tuning shows FACT learns from human demonstrations nearly as effectively as from robot demonstrations — a direct quantitative measurement of the human-data exchange rate under this recipe [§Experiments].

Method

FACT is a two-stage robot foundation model trained in two phases:

Track-Guided Coarse Planner — a jointly trained module that from the current observation + language goal predicts (a) a short-horizon action chunk, (b) future video, and (c) future 2D visual tracks on the scene. Future video and visual tracks also act as a shared task representation that works across human and robot embodiments, so visual-track supervision from human data transfers into the planner.
Force-Action Joint Refiner — builds on the coarse plan by jointly predicting future contact force and refining the coarse action. The anticipated force and the refined action are coupled during refinement so that predicted force consequences steer the action correction.

The pretraining substrate is EgoForce, introduced in the same paper: a large-scale egocentric human manipulation dataset where force recordings are captured alongside video, making it the first corpus on the wiki large enough to drive human-video-to-robot force-aware pretraining at foundation scale. During fine-tuning the model consumes a small mix of robot trajectories plus a larger number of human trajectories under matched task labels.

Results

Headline downstream result: on ten contact-rich tasks spanning tool use, musical instruments, and precision laboratory procedures, fine-tuning with only 10 robot + 90 human trajectories outperforms baselines (incl. T-Rex) by up to 60 absolute percentage points in success rate [§Experiments].
Ablations isolate the load-bearing component: force supervision in the human pretraining data is what carries the downstream gain — predicting future video and visual tracks also help dexterous execution but the gap narrows sharply when either is removed, while removing human-side force collapses the downstream advantage [§Ablations].
Human-robot exchange rate: varying the human:robot ratio during fine-tuning shows FACT learns from human demonstrations nearly as effectively as from robot demonstrations — a sharper quantitative statement than the usual “human data helps” claim [§Experiments].

Why it’s interesting

FACT is the first filed foundation-model-scale datapoint putting the Tactile sensing for manipulation “force is the missing modality” thesis directly inside a human-video-pretraining recipe — the sharpest answer yet to the Jitendra Malik: don't let CV researchers in robotics skip the sensorimotor level sensorimotor counter-position that VLA scaling under-weights contact dynamics. It stakes out a distinct position from existing force-integration recipes on the wiki: unlike FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT (per-layer force modulation inside the VLA, same acronym different team) and ReForce — force-aware retargeting via closed-loop residual force tracker ReForce (sim-trained residual force tracker atop a reference), this FACT puts force into the human pretraining corpus via EgoForce and uses coupled future-force + refined-action prediction as the integration pattern — moving the lever one layer upstream from policy architecture to pretraining data.

On the Human-to-Robot Retargeting axis, FACT sits alongside T-Rex: Tactile-Reactive Dexterous Manipulation T-Rex (its headline baseline, beaten by up to 60 pp) and TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning TouchScale (500-hour human bimanual tactile corpus; same “scale a human-side physical-contact channel” thesis but with 880-taxel tactile gloves instead of recorded force), with FACT distinguishing itself by making 6-DoF force the shared channel and using 2D visual tracks as the embodiment-agnostic task representation that bridges human and robot sides — a different interface-location bet from TouchScale’s tactile-mid-training on vision+tactile tuples.

The sibling-tech evidence that visual-track + future-video prediction is a useful co-task is consistent with the WAM-flavored training story that RoboJEPA: Scaling Robotic Latent World Models RoboJEPA and Long-WAM: Scaling the Context of World-Action Models Long-WAM have been developing on the pure-video axis; FACT’s contribution is binding this to force prediction so the shared representation covers both vision and contact.