Generative World Renderer

TL;DR

A data + recipe paper that addresses the bottleneck for generative bidirectional rendering: scaling inverse rendering (image → geometry/materials/lighting) and G-buffer-guided forward rendering to in-the-wild video by harvesting AAA games. The authors intercept rendering-API calls in Cyberpunk 2077 and Black Myth: Wukong via ReShade, dual-screen-stitch a mosaic of RGB + 5 G-buffer channels (depth, normals, albedo, metallic, roughness) at 720p/30fps, and capture 4M continuous frames over diverse weather, indoor/outdoor and motion conditions. Fine-tuning DiffusionRenderer on the resulting corpus improves cross-dataset inverse-rendering metrics on a held-out Black-Myth: Wukong benchmark and on MPI-Sintel; a Wan 2.1-T2V-1.3B-based forward renderer learns G-buffer-conditioned video synthesis with text-controlled environment editing. A VLM-based ranking protocol is proposed for material-quality evaluation on real video where ground truth doesn’t exist, validated against human pairwise preference (75–85% agreement on metallic/roughness).

Key claims

A non-intrusive ReShade-based pipeline can intercept and export the rendering pipeline’s G-buffer attachments at the graphics-API level without engine modification, decompilation, or asset extraction, by hooking per-frame render-target bindings and selecting attachments on stable format/extent/binding-pattern invariants [§3.1].
Camera-space normals are reconstructed at runtime from depth via inverse projection plus finite differences in the ReShade effects stage, because the view matrix is not accessible — making the captured normals compatible with prior inverse-rendering work [§3.1, Eq. 1].
Metallic and roughness packed into different channels of the same G-buffer target are decoupled and rendered to spatially distinct screen regions before screen-capture encoding, to prevent inter-channel bleeding from video compression [§3.1].
A dual-screen “mosaic” composition (two 2K monitors, OBS near-lossless capture, center-cropped tiling) preserves 720p per channel while keeping all six channels temporally synchronized in a single recording stream — avoiding per-frame disk I/O stalls [§3.2].
Distribution analysis shows Cyberpunk 2077 has more high-metallic pixels (metal-rich urban) and Black Myth: Wukong has more high-roughness pixels (natural outdoor) — the two titles together give complementary coverage of common real-world material appearances [§3.4, Fig. 3].
An offline motion-blurred RGB variant is provided via RIFE sub-frame interpolation + linear-domain averaging, so a single capture supports both sharp-RGB and motion-blurred-RGB training [§3.5, Eq. 2].
Fine-tuning DiffusionRenderer (Cosmos-based checkpoint) on Cyberpunk 2077 and evaluating on a 39-clip held-out Black Myth: Wukong benchmark improves depth (RMSE 0.430 vs 0.723 for DR), normals (Acc@11.25° 0.150 vs 0.110), scale-invariant albedo (si-PSNR 21.44 vs 19.90), and material RMSE (metallic 0.104 vs 0.230; roughness 0.266 vs 0.281) over the released DiffusionRenderer baseline [Table 1].
The same fine-tuned model improves cross-dataset transfer to MPI-Sintel (depth RMSE-log 0.745 vs 0.911 for DiffusionRenderer; si-PSNR 17.80 vs 17.46), suggesting the dataset bridges the synthetic-to-real domain gap rather than only fitting AAA-game visuals [Table 2].
A longer-clip training variant (113 frames vs 57) substantially improves long-video inference quality [§5.1, Fig. 2].
A VLM-based pairwise-ranking protocol on metallic/roughness — chosen because these channels have strong semantic+appearance priors a VLM can read — agrees with human pairwise preferences at 75–85% on the dataset’s evaluation set (preference rates for the fine-tuned model: 85% metallic / 75% roughness against the no-fine-tune baseline; 70% / 61% against DiffusionRenderer) [§4, Table 4].
A forward-rendering application: a Wan 2.1-T2V-1.3B fully fine-tuned on Black Myth: Wukong G-buffers + clip captions (lighting/environment-focused) supports text-prompt scene editing of game environments and generalizes cross-dataset to Cyberpunk 2077 without retraining [§5.1].

Method

The contribution is split across data capture, inverse-rendering fine-tuning, and forward-rendering adaptation, plus an evaluation protocol. Data capture: ReShade add-ons hook the rendering API, identify candidate render targets offline with RenderDoc (formats, extents, binding signatures), then at runtime maintain a pool of candidate attachments and GPU-copy only those that satisfy stable invariants. The selected target is rebound as an input texture into the ReShade effect runtime, which shades it to screen. Camera-space normals are derived from depth (Eq. 1) because the view matrix isn’t accessible. Metallic/roughness are unpacked into spatially distinct regions of the screen mosaic to prevent inter-channel video-compression bleed. Two 2K monitors are stitched into a single virtual surface that the game renders into; OBS captures the whole mosaic at near-lossless bitrate, yielding 720p per channel. Six synchronized channels (RGB + depth + normals + albedo + metallic + roughness) come out of one video file. For Cyberpunk 2077, a semi-automated driving setup with long-range waypoints, plus street walking and indoor visits, generates 40 hours of long-clip gameplay; for Black Myth: Wukong, exploration sequences from completed save files (combat avoided to limit camera motion). Qwen3-VL-235B-A22B-Instruct (vLLM-served) labels each clip with texture / weather / scene / motion categorical attributes from 5 uniformly-sampled timestamped frames.

Inverse-rendering fine-tuning fully fine-tunes the official DiffusionRenderer Cosmos checkpoint on 57-frame (and 113-frame) Cyberpunk 2077 clips at 480p, using the motion-augmented variant. Forward-rendering adapts Wan 2.1-T2V-1.3B by treating the G-buffers as conditional inputs; Qwen3-VL captions each clip with lighting/environment-focused descriptions (G-buffers already supply geometry and materials, so the text channel doesn’t need to). Trained on Black Myth: Wukong at 480p, 16 fps, 81 frames. The VLM evaluation protocol asks an MLLM to pairwise-rank metallic/roughness predictions on metrics for semantic correctness, spatial fidelity, and temporal consistency; correlation with human pairwise preference is reported per channel.

Results

On the 39-clip Black Myth: Wukong held-out benchmark (in-domain), the fine-tuned model leads on depth (AbsRel 0.023, RMSE-log 0.430, δ₁ 0.609 vs DiffusionRenderer’s 0.030 / 0.723 / 0.267), normals (Acc@11.25° 0.150 vs 0.110), scale-invariant albedo (si-PSNR 21.44, si-LPIPS 0.635 vs 19.90 / 0.646), and material RMSEs (metallic 0.104 vs 0.230; roughness 0.266 vs 0.281) [Table 1]. On MPI-Sintel (out-of-domain real-motion synthetic), depth RMSE-log 0.745 vs 0.911, albedo si-PSNR 17.80 vs 17.46 [Table 2]. The VLM judge on real-scene tests rates “Ours” higher than DiffusionRenderer on semantic / appearance / temporal axes (e.g. roughness Sem 1.78 vs 2.45 — lower is better since it’s average rank) [Table 3]. VLM-vs-human agreement on real videos: 85% (metallic) / 75% (roughness) when comparing to “Ours” baseline, 70% / 61% when comparing to DiffusionRenderer [Table 4]. Ablating motion blur shows the motion-augmented variant wins on Sintel (RMSE-log 0.745 vs 0.773; si-PSNR 17.80 vs 17.37) [Table 5].

Why it’s interesting

This is the strongest filed instance of training-data harvesting from commercial games as a substitute for hand-built synthetic datasets — the upstream extension of the synthetic-training-data thesis (Synthetic Training Data). Where Action100M: A Large-scale Video Action Dataset industrializes pseudo-labeling of real video and NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos synthesizes degraded conditions inside the training loop, this paper extracts photorealistic paired RGB↔G-buffer data that no labeling pipeline can produce — the ground truth comes from the rendering API itself. It complements the WFM thesis (World Foundation Models) from an angle the existing entries don’t cover: rather than treating the WFM as a latent-predictive backbone (Action100M: A Large-scale Video Action Dataset) or a closed-flagship interactive rollout (Project Genie: Experimenting with infinite, interactive worlds), it argues the WFM should be a bidirectional renderer with an explicit G-buffer intermediate — directly in tension with The flavor of the bitter lesson for computer vision, which would predict explicit material/geometry intermediates dissolve under the Bitter Lesson. On the evaluator side it adds a fifth shape to VLM-as-Evaluator: a pairwise material-quality ranker on real video, with human-pairwise agreement reported (75–85%), the gap on this concept page being that prior VLM-judge entries score still images, scalar rewards, or benchmark-time text questions — not pairwise material-channel decisions.