Getting a Grip on Robotic Data Collection — Co-Designed Koala Gripper for Force-Mediated Manipulation

TL;DR

RAI Institute stakes out a hardware-first position on handheld robotic data collection: the human-operated capture device and the robot-mounted end-effector must be co-designed as a single unified platform — kinematically identical, sharing finger geometry, camera placement, and force sensor suite — so that policies trained on demonstrations transfer without an embodiment gap. The Koala platform (named after the double-thumbed marsupial) implements this with a monolithic pivoting double thumb + two underactuated linkage-based fingers = 3 controllable DoF + 2 underactuated, delivering the range of grasps a parallel-jaw gripper cannot: paper-thin precision pinches, power grasps from a pencil to a two-liter bottle, and caging grasps on irregular objects, all with fingertip force sensors that log operator effort at 100 Hz. The post frames co-design as the answer for force-mediated, contact-rich manipulation (loosening bolts, assembly, disassembly) where visual observation alone cannot distinguish a resting hand from a squeezing one.

Key claims

The four scaling routes for robot learning data — internet-scale video, simulation, egocentric video, teleoperation, and handheld collection — have different physical-signal coverage; internet video and egocentric video cannot capture force or tactile data at training scale, and teleop loses subtle force/contact cues when the operator is removed from the workspace, slowing the demonstration cadence and inheriting that slowness in the learned policy [§Robotic Data Collection Methods].
Handheld collection uniquely preserves both human cadence and force feedback because the operator receives proprioceptive feedback through the mechanism while manipulating real objects, with force sensors in the gripper logging the interaction [§Handheld data collection].
Vision-only policies fail on force-mediated tasks not for resolution reasons but for modality reasons: a cupped hand gently resting on a bottle cap and one squeezing it produce nearly identical images, and the gap between “pressing” and “resting” is not primarily a resolution problem [§Advancing Dexterous Manipulation through Handheld Data Collection].
Retrofitting existing robot grippers for handheld use (adding tracking cameras + encoders to off-the-shelf end effectors) surfaces ergonomic problems that degrade data quality — data collection devices and robot end effectors must be developed in unison [§Co-Designing Grippers].
Co-design eliminates a class of transfer failures at the observation-mapping level: because a policy maps observations → actions, consistency between demonstration and deployment observations is the simplest way to make it reliable — same mechanism, same camera viewpoint, same force profile [§Co-Designing Grippers].
Ergonomics is a data quality issue — if the operator is fatigued or uncomfortable, the resulting policy suffers; three sizes of handheld device were prototyped to accommodate the range of human hands over extended collection sessions [§The Evolution of Co-Designing Grippers, §Koala Platform].
Koala’s design specifically targets force-mediated assembly and tool-use tasks — the earlier UMI-based parallel-jaw variant (UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data-style, per RAI’s reference to Chi et al. 2024) could not wield ratchets, screwdrivers, or hold a nail steady while hammering [§The Evolution of Co-Designing Grippers].
The design mandate was not to replicate every human grasp type but to achieve the functional outcome of each grasp while minimizing the active DoF — e.g. lateral pinch (key between thumb and index-side) becomes a power pinch between double thumb and fingers, same functional result with a single DoF [§Koala Platform].
The handheld variant accommodates full human hand stroke + force without breaking while keeping the force ratio under 3:1 for comfort in long sessions; the actuated variant replaces the human-operated linkage with a custom ballscrew-driven motor but shares finger geometry + sensor suite [§Koala Platform].
Force + proprioception directly address the reactive-policy problem: today’s policies observe the current moment and act without memory, so the instantaneous observation must fully determine task state — force/proprio resolve ambiguities vision alone cannot, making single-step observations rich enough to act on [§Why Richer Observations Matter Even More Without Memory].
Sensor spec: RGB HD wide-angle cameras mounted on the gripper (egocentric workspace view) + gripper state (width, velocity, torque) via motor encoders at 100 Hz + fingertip force sensors [§Koala Platform].
Empirical outcome: training on several hundred demonstrations has yielded successful policies while unlocking capabilities outside the realm of parallel-jaw grippers [tweet].

Method

Koala is a co-designed pair of grippers sharing the exact linkages, finger geometry, kinematics, and sensor suite. The handheld variant is hand-powered — operator squeezes triggers that couple 1:1 to finger motion via a linkage mechanism (rack-and-pinion of earlier UMI-based prototypes replaced by linkages to eliminate backlash). Three triggers control (i) the double thumb pivot, (ii) the coupled index-side finger closure, and (iii) the second finger closure, giving three active DoF plus two underactuated DoF that emerge from linkage compliance. Contoured deformable pad surfaces produce the grasp range (precision pinch → power grasp → caging). The actuated (robot-mounted) variant replaces the human-driven linkage with a custom ballscrew + motor; every downstream mechanical element is identical so the actuator sees the same friction, compliance, and contact geometry the operator saw.

Sensor stack — mirrored 1:1 across handheld and actuated variants — is HD wide-angle RGB gripper cameras (egocentric view of the workspace, camera pointing upward for tracking robustness in cluttered scenes), magnetic position encoders on the ballscrew (100 Hz gripper state: width, velocity, torque), and fingertip force sensors. Because both variants share the sensor placement and the linkage transmits the same forces, the observation streams are geometrically and mechanically matched. The gripper series was validated on Universal Manipulation Interface (UMI) foundations (Chi et al., 2024 — UMI-Bench 1.0: An Open and Reproducible Real-World Benchmark for Tabletop Robotic Manipulation with UMI Data) and improved with better tracking camera placement, backlash-free linkages, magnetic encoders, and integrated force sensing.

Three sizes of the handheld variant were prototyped so operators from a wide range of hand sizes can collect over extended sessions without fatigue. Roadmap: standalone capture devices for out-of-lab data collection (RAI’s stated example is picking litter on a beach) and a battery-installation task (bracket seating, nut tightening, connector mating, verification) as the near-term force-mediated benchmark.

Results

The post is a position + platform announcement, not a benchmark paper. The load-bearing empirical claim is that training on a dataset of several hundred demonstrations has yielded successful policies that unlock capabilities outside the realm of parallel-jaw grippers [tweet + §Koala Platform]. No per-task success rates, transfer numbers, or comparisons to teleop or UMI baselines are disclosed in the post.

Qualitative claims: paper-thin precision pinches match parallel-jaw performance; power grasps span pencil-scale to two-liter-bottle-scale objects; caging grasps handle irregular objects the parallel-jaw could not; operators wield tools (ratchets, screwdrivers, hammers) with the coupled double-thumb design that a parallel-jaw gripper cannot support [§The Evolution of Co-Designing Grippers, §Koala Platform].

Why it’s interesting

Koala is a hardware-substrate contribution to the Human-to-Robot Retargeting interface-location debate, in the same “make-collection-tool-embodiment-invariant” lineage as Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1) mimic (wearable exoskeleton for anthropomorphic hand), Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1 (UMI at 100K-hour scale), and HandUMI — Robot-free bimanual data collection retargetable to any parallel-gripper arm (Brik H. Meza launch tweet) HandUMI (open-source robot-free bimanual capture), but with a distinct co-design commitment: the pitch is not just “make the collection tool match a robot” (UMI, HandUMI) or “make the operator’s hand match a robot” (mimic) but “iterate the operator device and the robot device together as one platform.” The three-controllable-DoF morphology + integrated fingertip force sensing sits between parallel-jaw UMI (1 DoF, no force) and mimic’s 15-DoF anthropomorphic M1 hand — targeting the manufacturing/assembly force-mediated task class that both extremes leave underserved.

The force-modality argument (“pressing vs resting produces near-identical images”) complements the Tactile sensing for manipulation cluster’s central thesis quantitatively established in Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks (per-taxel F/T is the default across dexterous tasks) and FACT: Force-Aware Contact-Rich Manipulation via Timestep Modulation — demystifying when and why VLAs fail on contact-rich tasks FACT / ReForce — force-aware retargeting via closed-loop residual force tracker ReForce (force integration matters how not just whether) — but pushes the argument earlier in the pipeline, into the collection device rather than the policy or the sim. Koala’s F/T sensors also directly address the ambiguity the Tactile sensing for manipulation page names as the load-bearing gap.

The “ergonomics is a data quality issue” framing is a novel angle not explicitly filed elsewhere on the wiki — most retargeting work debates the algorithmic interface, not the physical one — and it aligns with ModPack: An Extensible Teleoperation Interface for Bimanual Mobile Manipulation ModPack’s operator-rig-focused position but applies the argument to handheld rather than teleop rigs. The three-size-prototype recipe is a concrete implementation detail worth watching if the co-design thesis holds up empirically.