That was not what I was aiming at! Differentiating human intent and outcome in a physically dynamic throwing task

TL;DR

Surendran and Wagner (Penn State) collected 1227 throws of a ball at a target from 10 participants and observed that 47% of throws are mistakes — i.e., the physical outcome does not match the thrower’s intended target — with 16% completely missing. The paper argues that in physically dynamic collaborative tasks, treating the observed outcome as a proxy for the human’s intent is systematically wrong, and proposes a pipeline that reads the thrower’s facial reaction after the throw to (a) detect that a mistake occurred and (b) recover the actual intended target. The outcome-prediction model beats a prior two-stream front-on video baseline by 38%, and a 1-D CNN combined with mistake-frequency priors gives an end-to-end pipeline for outcome + intent recognition.

Key claims

In a controlled 1227-throw dataset over 10 participants, 47% of throws did not land in the intended target region and 16% missed the target entirely — mistakes are the common case, not the tail [§Dataset].
Facial reaction after the throw carries enough signal to classify the outcome (hit/miss/mistake) and to recover the actual intended target, i.e., intent can be inferred post-hoc from the thrower’s response to their own outcome [§Approach].
The proposed outcome-prediction architecture improves on the previously used two-stream front-on video baseline by 38% on this task [Abstract].
Combining a 1-D CNN with priors learned from the empirical frequency of mistakes yields an end-to-end outcome-and-intent recognition pipeline [Abstract].

Method

The system consumes video of a person throwing a ball at a discrete set of target regions and outputs two labels: outcome (which region the ball actually hit, or “miss”) and intent (which region the thrower was aiming at). The key departure from prior work is treating these as separable — because they disagree ~47% of the time on this task. Two front-on camera streams are processed: one capturing the throwing action and one capturing the face during and after the throw. A 1-D CNN over the temporal facial-feature stream drives mistake detection; a learned prior over mistake type given observed outcome is then used to back out the most likely intended target. The outcome branch replaces the prior two-stream baseline architecture and reports a 38% relative gain on the throwing dataset.

Results

Mistake base rate on the collected 1227-throw dataset: 47% of throws are mistakes, 16% miss the target entirely [Abstract].
Outcome-prediction: +38% over the previously used two-stream architecture on front-on videos [Abstract].
End-to-end outcome + intent pipeline: 1-D CNN + mistake-frequency priors — the paper’s core contribution is that the intent channel is recoverable at all in this regime, rather than a specific accuracy number on a shared benchmark.

Why it’s interesting

The paper stakes out a position most robotics work implicitly assumes away: in physically dynamic manipulation, the observed outcome and the human’s intent are routinely different, and that gap is signal a robot teammate can exploit. This complements LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition, which separates intent (what should change in the scene) from control (how to achieve it) as a design axis in dexterous manipulation — where LUCID uses intent-vs-control to scale supervision, this paper uses intent-vs-outcome to detect that the human made a mistake and recover what they wanted. It also contrasts with EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration and FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models, which are about a robot recognizing its own failures and recovering; here the robot’s job is to recognize a human teammate’s failure via their reaction, then adjust its inferred goal. The ~47% mistake rate on a simple ball-throwing task is also a useful ground-truth number to cite when evaluating whether a VLM-based intent estimator is doing anything real or just parroting the observed outcome.