Being-H0.8: A Latent Tactile World-Action Model at Scale

ListenTouch Without Sensors: Being-H0.8 and Tactile Pretraining at Scale10:52

Show notes

Being-H0.8 from the BeingBeyond team is the first embodied foundation model to make touch a first-class part of large-scale pretraining — and it does it *without* tactile sensors, by inferring contact and proximity from half a million hours of egocentric human video. We unpack the four bets that carry the work: TactoHand's visual-to-tactile pseudo-labels, the TopoHand shared action space across hands and grippers, a tactile-aware prior–posterior latent world model, and a slow–fast action expert for reactive control. Plus what a launch note without benchmark tables can and can't tell us. **Sources** - Being-H0.8: A Latent Tactile World-Action Model at Scale (BeingBeyond Team)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Being-H0.8 (BeingBeyond) is the third generation of the Being-H series and the first embodied foundation model to make touch a first-class part of large-scale pretraining on egocentric human video. Two representational bets carry the paper: TactoHand turns 500,000+ hours of raw human video into dense pseudo-tactile supervision (binary contact + continuous proximity fields on the canonical MANO surface) via a geometry-supervised visual predictor; TopoHand replaces Being-H0.5’s Unified Action Space with a shared kinematic state–action language covering human hands, dexterous robot hands, and parallel grippers in a single 10-dim morphology latent + 20-DoF canonical joint parameterization. On top, a prior–posterior latent world-action formulation (inherited from Being-H0.7) is trained with future tactile evidence in the posterior and deployed prior-only, with a slow–fast action expert that caches world-context per chunk while regenerating short segments at each state/tactile-feedback anchor for reactive control.

Key claims

Visual world models alone under-supervise contact-rich manipulation — contact establishment, load-bearing region, and post-contact adaptation “can determine the success of grasping, insertion, twisting, and handovers” but are subtle or occluded in RGB [§Introduction]. Being-H0.8’s remedy is not adding tactile sensors at scale but inferring touch from vision as a pseudo-label channel.
Hand–surface interaction (not sensor readings) is the unifying tactile representation: TactoHand predicts a binary contact field and a continuous proximity field over the canonical MANO surface from tracked hand clips, using registered hand+object mesh distance/orientation as geometric supervision [§TactoHand]. Where measured tactile signals exist, device-specific anatomical mappings project sparse contact/pressure onto the same canonical hand surface.
UniHand 3.0 = 500,000+ hours of egocentric human video, described as “the first human-centric corpus at this scale for which every sample is traceable to its source” [§Egocentric Human Data]. Every sample flows through a semi-automated pipeline (ingestion, dedup, filtering, standardization, structuring) with human-in-the-loop source validation, plus a quality-control stage that detects >92% of erroneous episodes before pretraining [§Quality Analysis].
The framing shifts explicitly from “how much video can be collected” (Being-H0/H0.5/H0.7) to “which video provides reliable and useful embodied supervision” [§Egocentric Human Data]. Curation, deduplication, and validation are load-bearing: “nominal scale alone does not determine training value.”
TopoHand separates morphology from articulation into a shared state–action space across embodiments: 10-dim morphology latent from a shape VAE + wrist translation/rotation + 20 canonical joint angles [§TopoHand]. Parallel grippers enter the same space via a morphology-normalized opening derived from thumb-to-nearest-fingertip distance, “giving human hands, dexterous robot hands, and grippers a single shared policy interface.”
The prior–posterior latent world formulation from Being-H0.7 is retained but augmented with tactile evidence in the posterior: future visual + tactile observations supervise deployable latent queries at training time; at inference “the posterior and all future observations are removed” [§Prior–Posterior Formulation].
The slow–fast action expert decouples horizon-planning frequency from reactive control: the slow stream caches world context + full-horizon action prediction at chunk start; the fast stream reuses the cached context and regenerates each short segment with the latest proprioceptive + tactile feedback via blockwise causal attention, sharing parameters and targets [§Slow-Fast Action Expert].
A universal tactile encoder organizes heterogeneous tactile channels (global contact labels, sparse fingertip taxels, dense per-vertex contact/proximity/pressure) as a coarse-to-fine pyramid over canonical hand topology; a query-based Perceiver resamples variable-length tactile-token sequences to a fixed count, and a learned missing-touch representation preserves the interface when no tactile signal is available [§Universal Tactile Encoder].
Synthetic human-robot aligned data is used as an intermediate representation between human demos and robot learning: text-to-video for object-action coverage + image-to-video for scene consistency, filtered by an automatic evaluator, then converted to executable robot demonstrations via hand reconstruction, human-body inpainting, finger retargeting, IK, and depth-aware compositing [§Synthetic Human–Robot Aligned Data].
Hardware deployment is bimanual across two platforms: (a) ROKAE AR5 + interchangeable LinkerHand L25 (16 actuated / 21-DoF, 12×6 piezoresistive fingertip pressure arrays) or DexHand 021 (12 actuated / 22-DoF, fingertip visuotactile sensors); (b) RealMan RM65 + Inspire grippers + Daimon DM-TAC W2M tactile sensors [§Hardware Platforms].
Robot-data heterogeneity is handled by a two-stage standardization: platform-level canonical URDFs (arm kinematics, joint conventions, wrist/EE frames) followed by dataset-level calibration; actions are ultimately expressed as wrist poses in the camera coordinate frame [§Standardizing Heterogeneous Robot Data].

Method

Being-H0.8 has four architectural pieces that hang off one information path. UniHand 3.0 ingests 500K+ hours of egocentric video and passes each clip through TactoHand — a temporal visual model trained with geometric supervision (registered mesh distance + orientation) that predicts binary contact and continuous proximity fields on canonical MANO. Where measured tactile signals exist (robot piezoresistive arrays, DM-TAC pressure, human gloves), device-specific mappings project them onto the same MANO surface, and a universal tactile encoder (query-based Perceiver over a coarse-to-fine tactile pyramid) resamples any of these to a fixed tactile-token count.

Actions live in TopoHand — a shared 10-dim morphology latent (shape VAE) + wrist pose + 20 canonical joint angles per hand, with per-embodiment adapters translating in/out. Parallel grippers get one added morphology-normalized opening coordinate. Training uses a prior–posterior formulation: the posterior fills future latent positions with real future visual + tactile observations; the prior sees only current instruction + observations + latent queries; the two are aligned so the prior learns to anticipate action-relevant latent state. At deployment the posterior and all future observations are removed.

The slow–fast action expert sits on top of these latents. At each chunk start, the slow stream computes and caches world context + a full-horizon action prediction. Before each shorter segment executes, the fast stream reuses the cached context, ingests the newest proprioceptive + tactile feedback, and regenerates the next segment under a blockwise causal attention mask that limits each fast segment to feedback available by its execution anchor. Slow and fast share parameters and targets; only the next segment is committed.

Results

The blog is a launch note without numbered benchmark tables — the only quantitative claim is that the QC stage “detects over 92% of erroneous episodes before pretraining” [§Egocentric Human Data]. Real-world evaluation is described qualitatively: bimanual demonstrations “probe continuous force modulation, deformable-object handling, contact coordination, rapid adaptation, and precise tool use,” each shown from a single focused view so contact sequences are visible [§Real-World Contact-Rich Manipulation]. No head-to-head against Being-H0.7 or external VLAs is reported at filing time.

Why it’s interesting

Being-H0.8 is the first filed entry that combines the four recipes the wiki has been tracking separately: tactile pretraining without tactile sensors (visual → contact/proximity pseudo-labels — extending FELT: Generating Tactile Signals from Vision for Visuo-Tactile Manipulation and PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images from per-task policies to foundation-model pretraining), a shared kinematic state–action space across humans + dexterous hands + grippers (a distinct interface position from the 6+ already catalogued on Human-to-Robot Retargeting — closest to Cross-Embodiment Robot Manipulation via a Unified Hand Action Space‘s geometric-sphere but with morphology + articulation split at the shape-VAE rather than IK level), synthetic human-robot aligned video as a robot-data multiplier (Synthetic Training Data), and a latent world-action posterior formulation (the World Foundation Models recipe that LingBot-VA 2.0: Native Video-Action Pretraining for Generalizable Robot Control and Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models have been articulating). The load-bearing bet is that visual tactile pseudo-labels transfer at 500K-hour scale — which if true would deprecate the taxel-placement discussion in Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks for large-scale pretraining substrates, though not for deployment-time contact reactivity.