Introducing Perceptron Egocentric API

TL;DR

Perceptron’s Egocentric API is a robot- and egocentric-video annotation pipeline built on Perceptron’s Mk1 embodied VLM plus a hand pose-estimation sidecar. From raw video it emits (a) temporal segmentation into atomic manipulation events, (b) self-contained subtask labels — with or without the episode’s task instruction — and (c) per-frame per-hand bounding boxes and 21-keypoint skeletons with left/right identity and manipulation-taxonomy captions. On Macrodata’s WGO-Bench it reports 0.280 semantic end-to-end F1 vs 0.158 for the previous SOTA WGO one-pass Gemini-3.5-Flash pipeline (+77% relative), 0.370 segment F1, and costs 10–15× less than the ~$50 per video-hour human-annotation baseline. Perceptron’s claim is that Mk1’s pre-training already encodes hands, contact, object state, and spatial relations as native concepts — no fine-tuning on WGO-style episodes was done — so the pipeline is an inference-time framework on a general embodied model.

Key claims

Perceptron Egocentric segments video into atomic manipulation events, labels each with a self-contained action description, and additionally emits per-frame hand bounding boxes plus 21-keypoint skeletons (wrist + four joints per finger) with left/right identity via a Mk1 hand detection + pose-estimation sidecar [§“Why it matters for robotics labs”].
Segmentation boundaries are anchored to hand behavior (contact, grasp, release) rather than pixel changes, and each hand is captioned independently into a fixed manipulation taxonomy (reaching, grasping/pinching, lifting, holding, placing/inserting, pushing/pulling, rotating, opening/closing, releasing, plus visibility state) [§“Why it matters for robotics labs”].
The system runs in two profiles — with the task instruction as context, or fully instruction-blind — so the same pipeline serves both known-task labeling and cold data curation [§“Why it matters for robotics labs”].
With the task instruction, Perceptron Egocentric reports semantic end-to-end F1 0.280 vs WGO one-pass 0.158 (+77% relative) on WGO-Bench, producing 181 fully correct segments out of 743 gold vs ~101 for WGO one-pass [§“Head-to-head on WGO-Bench”].
Segment F1 0.370 vs 0.302 (+23% relative) at the same evaluation protocol (IoU ≥ 0.75 temporal matching, benchmark-prescribed LLM label judge), with segment precision 0.435 and segment recall 0.322 (239 of 743 gold segments matched from 550 predictions) [§“Head-to-head on WGO-Bench”].
Semantic precision 0.330 vs 0.190 and semantic recall 0.244 vs 0.136: more of the emitted segments are right and ~80% more gold subtasks are recovered end-to-end vs the WGO one-pass baseline [§“Head-to-head on WGO-Bench”].
Without any task instruction, the best-accuracy Perceptron profile still beats the WGO seeded-relabeling no-instruction baseline (0.138 end-to-end F1) on segmentation and end-to-end quality; the cost-optimized no-instruction profile trims cost ~30% and lands at 0.182 end-to-end F1, still above WGO’s no-instruction seeded variant [§“Head-to-head on WGO-Bench”].
WGO’s seeded-relabeling variant reaches 78.1% conditional label accuracy but only 0.168 end-to-end F1 — 40% below Perceptron’s no-relabeling, with-instruction result — and its label accuracy is scored on the 0.144-recall slice of segments its pipeline matches, so the conditional metric is not comparable across pipelines with different recall [§“Head-to-head on WGO-Bench”].
**Cost: 10-15× cheaper than the ~ $50/video-hour human annotation baseline** at higher end-to-end F1; costs computed from logged usage at provider list rates as of July 2026, batch discounts excluded; hand-pose stream runs on Perceptron GPUs and adds ~$ 1/video-hour if enabled [§“Core results”, §“Notes on evaluation and costing”].
Mk1 was not fine-tuned on the WGO harness, taxonomy, or WGO-style episodes; the annotation features are inference-time framing on a general pretrained embodied model whose representation Perceptron argues natively encodes hands, contact, object state, and spatial relations [§“Why it wins”].
Perceptron’s positioning claim: general-purpose VLM pipelines “sample frames and guess what happened” while Mk1 “perceives the manipulation as it happens” — i.e. the win is model-side (Mk1 pre-training) rather than pipeline-side, so it should hold without task-specific training and widen as Mk1 does [§“Why it wins”].

Method

The pipeline sits on top of Perceptron Mk1, a frontier embodied VLM pretrained to perceive the physical world (hands, contact, object state, spatial relationships as native concepts). For each episode, Mk1 runs a segmentation-and-labeling pass that (i) divides the video into atomic manipulation events bounded by hand-driven state changes, (ii) emits a self-contained action label per segment, and (iii) can optionally consume the task instruction as context. In parallel a hand pose-estimation sidecar tracks both hands frame-by-frame, producing bounding boxes, 21-keypoint skeletons, per-hand left/right identity, and per-hand action-taxonomy captions plus visibility. The pose stream is what supplies contact/grasp/release cues to the segmenter. Evaluation on WGO-Bench uses the benchmark’s own protocol: temporal IoU ≥ 0.75 for boundary matching and the prescribed LLM judge for label correctness; semantic end-to-end requires both to pass. Reported numbers are on the full benchmark (all episodes completed) with the task instruction provided except where marked no-instruction.

Results

Semantic end-to-end F1: 0.280 (Perceptron, with-instruction) vs 0.158 (WGO one-pass, with-instruction) — the metric that requires both a correct boundary and a correct label. 181/743 gold segments fully correct vs ~101 [§“Head-to-head on WGO-Bench”].
Segment F1: 0.370 vs 0.302; segment precision 0.435, segment recall 0.322 (239/743 matched from 550 predicted) [§“Head-to-head on WGO-Bench”].
Semantic precision 0.330 vs 0.190; semantic recall 0.244 vs 0.136 — approximately 80% more gold subtasks recovered end-to-end than WGO one-pass [§“Head-to-head on WGO-Bench”].
No-instruction results: Perceptron’s best no-instruction profile beats WGO’s seeded no-instruction variant (0.138 end-to-end F1) on segmentation and end-to-end. The cost-optimized no-instruction profile: 0.182 end-to-end F1 at ~30% lower cost [§“Head-to-head on WGO-Bench”].
WGO seeded-relabeling comparison: 78.1% conditional label accuracy but 0.168 end-to-end F1 (40% below Perceptron’s 0.280), and label accuracy is measured only on the 0.144-recall matched slice — so the seeded pipeline’s headline label number does not compose with its own boundary recall [§“Head-to-head on WGO-Bench”].
Cost: 10-15× cheaper per video-hour than the ~ $50 human-annotation baseline at higher end-to-end quality; hand-pose sidecar adds ~$ 1/video-hour on Perceptron GPUs [§“Core results”, §“Notes on evaluation and costing”].
Outperforms Gemini Robotics ER-1.6 and Gemini 3.5 Flash, described as the previous SOTA for this domain [§“Why it matters for robotics labs”].

Why it’s interesting

This is the most direct answer yet to the specific bottleneck Segmenting Robot Video into Actionable Subtasks (WGO-Bench) flagged: WGO-Bench’s own numbers show boundary discovery is the harder half of subtask annotation (0.306 segmentation F1 vs 61.0% gold-boundary labeling accuracy), and Perceptron argues that boundary discovery is where a pre-trained-on-hands embodied model beats a frame-sampling general-purpose VLM. The +77% relative end-to-end F1 gap comes almost entirely from better segmentation (0.370 vs 0.302 segment F1 combined with 0.330 vs 0.190 semantic precision), not from better labeling — consistent with the “hands, not pixels” thesis.

Two second-order implications for the wiki’s VLA Models and Synthetic Training Data threads. First, if the manipulation-boundary problem is now closer to solved by a general embodied VLM at ~$3–5/video-hour, the data-engine layer downstream of pretraining (RT-H, π0.5, SARM, and the human-video-cotraining stack in What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos? and HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining) gets substantially cheaper supervision at higher fidelity. Second, this is the first filed instance where a pretrained embodied VLM’s representation is itself the annotation engine — a distinct pattern from Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots-style retargeting or dense-caption pipelines. It also gives VLM-as-Evaluator a new data point on the “which VLM makes a good judge/annotator” question that IGenBench’s judge-selection study opened: on manipulation video the answer may be an embodied VLM rather than a general frontier VLM.

The hand-pose sidecar shape also lines up with Pose Estimation and Motion Capture: 21-keypoint skeletons + left/right identity + a manipulation-action taxonomy per hand is the same primitive that HumanScale, ACE-Ego-0, and TriHands consume, packaged as an API.