Nyx — GPU path-traced renderer plugin for Genesis World

TL;DR

Nyx is the GPU path-traced renderer plugin for Genesis World — Simulation platform for general-purpose robotics & embodied AI learning — a purpose-built robotics renderer that targets path-traced accuracy as the baseline with rasterization shortcuts only where they don’t compromise downstream policy learning. It plugs into Genesis as a camera sensor, supports PBR materials, HDRI + analytic lighting, 3D Gaussian splat assets, attached/multi-camera setups, multi-environment rendering, and per-pixel object picking. The repo contains the runnable examples and Sphinx documentation; the actual renderer ships as a separate gs-nyx-plugin PyPI wheel for x86-64 Linux (manylinux 2.34+, validated on Ubuntu 22.04) and Windows 11, Python 3.10–3.13, requiring an NVIDIA GPU with CUDA 12.9+ and driver 575+. Apache 2.0.

Key claims

Plugs into Genesis as a camera sensor rather than a separate render server — driven by batched physics, allowing thousands of parallel rollouts each with its own scenario, lighting, and camera trajectory through a single unified pipeline [README + parent blog §Tight Genesis integration].
Path-traced baseline with multi-bounce lighting, soft shadows, indirect illumination, and a physically grounded camera model — so “what the policy sees matches what a real sensor captures” [parent blog §Nyx].
HDRI pipeline lights scenes with measured radiance, and 3D Gaussian splat assets are first-class — the team highlights that the harder problem is reconciling image-based lighting with splat-based geometry so captured assets participate correctly in path-traced light transport [parent blog §Nyx].
Throughput target: render noise-free 1080p frames in ≤4 ms on a high-end consumer GPU, with no baking and no ghosting. Implementation uses a visibility buffer, bindless GPU-driven architecture, MSAA, hardware ray tracing, hardware matrix cores, and video compression, all tuned for GPU occupancy [parent blog §Nyx].
The seven runnable examples cover the API surface: hello-nyx, attached camera, PBR materials, light types, 3D Gaussian splat, object picking, and multi-camera multi-environment [README §What’s in this repository].
Distribution split: this repo holds documentation + runnable examples; the actual renderer ships as a precompiled wheel (pip install gs-nyx-plugin) rather than buildable source [README §Running an example].
Platform support is narrower than the rest of the Genesis stack: Linux x86-64 (manylinux 2.34+, Ubuntu 22.04 validated) and Windows 11 only, NVIDIA-only, CUDA 12.9+ and driver 575+ — no macOS/Metal, no AMD/ROCm, no ARM [README — Prebuilt wheels].
License: Apache 2.0, with issues/PRs welcomed [README §Contributing].

Method

The repo is documentation-and-examples only — the renderer itself ships as a closed-source-but-Apache-licensed pip wheel (gs-nyx-plugin). Users pip install gs-nyx-plugin and then run the example scripts in examples/ directly; the Sphinx documentation in docs/ embeds those same scripts via literalinclude so what’s on the docs site is exactly what runs locally. The API reference is auto-generated from docstrings and .pyi stubs shipped inside the installed wheel — no source checkout of the renderer is required.

The seven examples form a tutorial sequence: 01_hello_nyx.py (basic camera + scene), 02_attached_camera.py (camera mounted on a robot link), 03_materials.py (PBR material parameters), 04_light_types.py (HDRI, area, point, directional), 05_gaussian_splat.py (loading 3DGS assets into a path-traced scene), 06_object_picking.py (per-pixel object ID readback), and 07_multi_camera_multi_env.py (batched parallel rendering across environments).

The technical detail of how the renderer achieves its claimed ≤4 ms / 1080p / noise-free target is documented in The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward rather than in this repo: visibility buffer, bindless GPU-driven architecture, MSAA, hardware ray tracing, hardware matrix cores (presumably tensor cores for denoising or BVH traversal), and video compression, all tuned for GPU occupancy. The renderer reconciles image-based lighting with splat-based geometry — a known hard problem where naive integration produces incorrect light transport when 3DGS assets sit alongside meshes.

Results

This is a software release; no benchmark numbers in the repo itself. Performance claims come from the parent blog: ≤4 ms noise-free 1080p frames on a high-end consumer GPU (specific model not disclosed), no baking, no ghosting. The parent blog reports a downstream system-level claim — FID-measured reality gap 45% smaller than the next-best alternative simulator on the Genesis dataset — but does not isolate how much of that gap reduction is attributable to Nyx vs the physics/control/asset pipelines. No published comparison against off-the-shelf renderers (Omniverse, Blender Cycles, PBRT) on shared scenes.

Why it’s interesting

Nyx is a sharp contrast point against the “neural rendering everywhere” trend in World Foundation Models — most filed WFM entries (Genie 3, RELIC, LingBot-World, Cosmos Predict) are learned video generators acting as renderers. Nyx is the explicit hand-engineered renderer built specifically so a policy’s pixels match a real camera’s pixels, with reality-gap correctness as the success metric rather than perceptual quality. The choice to make 3D Gaussian splats first-class — and to take “reconcile IBL with splat geometry under path tracing” as the hard problem worth solving — is the most concrete recent statement of how captured-asset pipelines should integrate with physics-grade rendering, in contrast to the MirageLSD: The First Live-Stream Diffusion AI Video Model / Introducing Runway Gen-4.5 approach of rendering through a learned diffusion model.

The ≤4 ms / 1080p / no-ghosting target also makes Nyx a benchmark for what “real-time path-traced” means in robotics specifically — distinct from game-engine real-time targets (which permit baking and temporal denoising artifacts that obscure the visual signals policies learn from) and offline path-tracing targets (which permit minutes per frame). Within Synthetic Training Data, Nyx is the rendering counterpart to the data-generation pipelines that need millions of frames that survive zero-shot transfer to real cameras.