Skip to content
Bud's Wiki

Kaolin — A PyTorch Library for Accelerating 3D Deep Learning Research

TL;DR

Section titled “TL;DR”

Kaolin is NVIDIA’s Apache-2.0 PyTorch library of GPU-optimized building blocks distilled from 3D deep-learning research. It packages representation-agnostic physics (Simplicits, plus experimental FreeForm/RKPM and Newton coupling), first-class 3D Gaussian splat containers with PLY/USD I/O and gsplat interop, differentiable rendering (DIB-R, nvdiffrast, easy_render PBR), a GPU octree acceleration structure (Structured Point Clouds), fast conversions between representations (meshes ↔ voxel grids ↔ point clouds ↔ gaussians, FlexiCubes, DMTet), USD I/O with a custom physics schema, and quaternion / camera math. Version 0.18.0 is the current release; a web_framework_prerelease branch adds a Dash-based client-server framework for browser-side interactive prototyping over AI+3D research.

Key claims

Section titled “Key claims”
  • Kaolin exposes representation-agnostic physics (Simplicits + experimental FreeForm) that can simulate meshes, point clouds, and Gaussian splats through a common occupancy interface, with collisions enabled since v0.18.0 [README News § v0.18.0].
  • FreeForm / RKPM (CVPR 2026, ~40× faster than Simplicits) has landed on master post-v0.18.0 and is integrated into the Simplicits API [README News § Unreleased].
  • The library provides a GaussianSplatModel container plus PLY/USD I/O with feature preservation and gsplat-compatible camera converters, making 3DGS a first-class citizen alongside meshes [README Features].
  • SPC (Structured Point Clouds) provides a GPU octree acceleration structure with ray tracing and feature grids for neural-field applications [README Features].
  • USD I/O includes a custom Kaolin physics schema for materials, skinned physics, and subset features — so simulation-ready assets can round-trip through USD [README News § Unreleased].
  • Experimental Newton coupling lets Simplicits soft bodies interact with rigid bodies, MPM, and articulated robots (including a Franka arm example) [README Features].
  • Version 0.12.0+ ships as pre-built pip wheels against pinned PyTorch × CUDA combinations [README Installation].
  • FlexiCubes (gradient-based mesh extraction, SIGGRAPH 2023) was relicensed to Apache 2.0 in v0.18.0 [README News § v0.18.0].

Method

Section titled “Method”

Kaolin is a library, not a paper — its design bet is that reusable, GPU-optimized 3D primitives (physics, rendering, conversions, splat containers, USD I/O) should live in one PyTorch package rather than being reimplemented per project. The physics module implements Simplicits as its production backend and now integrates FreeForm/RKPM for reduced-order elastodynamics on meshes and 3DGS. The GaussianSplatModel container and PLY/USD Gaussian I/O make splats a first-class representation. Differentiable rendering ships DIB-R, nvdiffrast bindings, and an easy_render PBR API with spherical-harmonics and spherical-gaussians lighting. Structured Point Clouds provides a GPU octree with ray tracing for neural fields. The USD physics schema is the interoperability story — assets with material and skinned-physics metadata round-trip between Kaolin and Omniverse. The experimental Newton coupling extends the physics scope to rigid-body / MPM / articulated-robot interaction.

Results

Section titled “Results”

Current release is v0.18.0 (Apache 2.0), with a broader ecosystem of research projects built atop it: FreeForm (CVPR 2026), VoMP (ICLR 2026), ArtisanGS (interactive 3DGS segmentation), TRON (single-step neural renderer for relightable 3DGS), 3DGRUT (ray-tracing/rasterization of Gaussian particles), 3DGUT (joint mesh+splat rendering, CVPR 2025 oral), Diffusion Texture Painting (SIGGRAPH 2024), and the Kaolin Wisp neural-fields engine. Two SIGGRAPH 2026 sessions are scheduled: a talk on the web client-server framework and a hands-on lab on capture-to-simulation pipelines for 3DGS.

Why it’s interesting

Section titled “Why it’s interesting”

For Luma this is the reference stack for shipping 3D research as reusable PyTorch modules — the physics + splats + USD + differentiable-rendering combination is unusually complete relative to ad-hoc per-paper code. FreeForm (FreeForm: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes) and Simplicits (Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation) both ship here as production physics modules, making Kaolin the practical entry point for anyone trying to add deformable simulation to a 3DGS asset pipeline. Complementary to Genesis World — Simulation platform for general-purpose robotics & embodied AI learning (Genesis targets full-stack robotics simulation with broader multi-physics), where Kaolin is the 3D deep-learning primitives layer that ML pipelines compose with. The custom USD physics schema is also a bet worth watching — a lot of the friction in “simulate this scanned scene” pipelines is USD interoperability, and having a schema for skinned physics baked in makes that friction lower.

See also

Section titled “See also”