VOID: Video Object and Interaction Deletion

TL;DR

VOID (Netflix / INSAIT) is a video object removal model that goes beyond inpainting shadows and reflections to alter downstream physical interactions — when a person holding a guitar is removed, the guitar falls; when middle dominos in a chain are deleted, the trailing dominos no longer topple. Built on CogVideoX-Fun-V1.5-5B-InP and fine-tuned in two sequential passes, the model is conditioned on a 4-value quadmask that encodes (primary object to remove, overlap, affected regions, background). Training data is a paired counterfactual dataset generated from Kubric (rigid-body simulation) and HUMOTO (human-object motion capture re-rendered in Blender) where removing the object requires the physics engine to roll forward to a different outcome. At inference, a vision-language model identifies the affected regions to populate the quadmask. The framing — “video editing as high-level causal reasoning” — directly connects the layered-decomposition / video-removal line to physics-aware world modeling.

Key claims

Existing video object removal methods inpaint content behind the object and correct appearance-level artifacts (shadows, reflections) but fail when the removed object has significant physical interactions like collisions, leaving implausible debris behavior [§1, Fig. 1].
A paired counterfactual dataset is generated from Kubric (synthetic rigid-body scenes) and HUMOTO (human-object motion capture re-rendered in Blender) where re-running the simulator without the target object yields a ground-truth “what would have happened” video — supplying supervision for object + interaction removal that real video cannot provide [§3, dataset pipeline].
A quadmask (4-value mask: primary object to remove, overlap regions, affected regions e.g. falling objects, background to keep) is the conditioning interface that lets the model distinguish “where to inpaint cleanly” from “where to re-simulate physics” [§Method; project page].
VOID is built on CogVideoX-Fun-V1.5-5B-InP (a video-inpainting backbone) and fine-tuned in two sequential passes: Pass 1 trains object + interaction removal from scratch; Pass 2 continues from the Pass 1 checkpoint with optical-flow-warped latent initialization for improved temporal consistency on longer videos [GitHub README].
At inference, a vision-language model identifies the regions of the scene affected by the removed object, which populates the “affected regions” channel of the quadmask before the diffusion sampler is invoked [§Method, Inference].
On both synthetic and real videos, VOID better preserves consistent scene dynamics after object removal than prior video object removal methods [§Experiments, Abstract].
The framing — “make video editing models better simulators of the world through high-level causal reasoning” — explicitly positions counterfactual object removal as a probe of a generative model’s grasp of physical causality [§Conclusion].

Method

VOID treats object removal as a counterfactual generation problem: given a video and a mask of the object to remove, produce the video that would have occurred had that object never been present, including all altered physical interactions. The model is a video diffusion transformer initialized from CogVideoX-Fun-V1.5-5B-InP, an inpainting-fine-tuned variant of CogVideoX. Conditioning is provided via a per-frame, per-pixel quadmask with four values encoding: (0) background to keep, (1) primary object to remove, (2) overlap regions where the object touches/occludes another scene element, (3) affected regions — pixels that will change as a downstream consequence (e.g. a guitar that will now fall, dominos that will no longer topple).

Training data comes from two pipelines. The Kubric pipeline samples random rigid-body scenes and renders two variants per scene: the full scene and the same scene with one object removed before simulation (so physics rolls forward differently). The HUMOTO pipeline takes human-object interaction motion-capture clips (Adobe Research’s HUMOTO dataset, characters Remy/Sophie), re-renders them in Blender with the human present, then re-renders without the human while letting Blender’s physics engine drop the held objects naturally. Each training sample is a paired triplet: (input video with object, ground-truth video without object, quadmask).

Training runs in two sequential passes. Pass 1 fine-tunes the transformer from the CogVideoX-Fun checkpoint on the counterfactual paired data, learning object removal + interaction prediction. Pass 2 continues from Pass 1 with an optical-flow-warped latent initialization (latents from neighboring frames are warped into the current frame’s coordinates before being added to the noise) — this targets long-horizon temporal consistency. Inference can chain both passes for higher quality or run Pass 1 alone for speed. A separate VLM step identifies which regions of the scene are physically affected by the removed object and writes those regions into channel (3) of the quadmask before sampling. Inference requires ≥40 GB VRAM (e.g. A100).

Results

Quantitative metrics live in the PDF; surfaced claims:

On both synthetic and real-world test data, VOID better preserves consistent scene dynamics after object removal than prior video object removal methods [§Experiments, Abstract].
Qualitative results on Figure 1: (a) removing the middle three blocks of a domino chain → the yellow trailing block correctly never falls; (b) removing the hands manipulating spinning tops → the tops continue spinning uninterrupted [Fig. 1].
A Gradio demo on HuggingFace Spaces (sam-motamed/VOID) exposes the interactive removal pipeline; the model checkpoint is published at netflix/void-model on HuggingFace; training code is open-source at Netflix/void-model on GitHub.

Specific FID / VBench / per-baseline ablation numbers are not extractable from the project page text and require the PDF.

Why it’s interesting

VOID is the first paper filed on the wiki that explicitly targets physical-interaction-aware video object removal — not just inpainting “behind” the object or alpha-compositing shadows, but rolling the scene forward as if the object had never existed. This is structurally distinct from the existing wiki cluster on Layered Image/Video Decomposition — both OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models and EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing attribute appearance-level effects (shadows, reflections, wake) to the removed object, but neither alters downstream physics. VOID adds a third axis to the concept: removal as counterfactual world rollout rather than as pixel attribution. Its training-data recipe — render two physics simulations per scene, with and without the target — is the natural sibling of EffectErase’s Blender + real-capture hybrid VOR dataset, but pushes the synthetic side from appearance counterfactuals to dynamics counterfactuals. It is also the training counterpart to PICABench: How Far Are We from Physically Realistic Image Editing?, which built a 900-sample benchmark for physical-realism evaluation of editing — VOID generates training pairs along nearly the same physical-realism axes (Mechanics: gravity, collisions, contact deformation) that PICABench measures. For Luma’s world-model work, the explicit framing — “video editing as causal reasoning, with object removal as the probe” — pulls layered video editing into the same conceptual frame as world-foundation modeling.