HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer

TL;DR

HiDream-O1-Image is a pixel-space Diffusion Transformer that natively unifies text-to-image generation, instruction-based editing, and subject-driven personalization in one model — with no separate VAE and no disjoint text encoder. Raw image pixels (patchified + learnable patch embeddings), text (the backbone’s native LLM tokenizer), and task condition images (SigLip-2 features + learnable projection) are all mapped into a single shared token space and processed by a decoder-only Transformer initialized from Qwen3-VL-8B-Instruct, using a hybrid attention mask (causal for text/condition tokens, full for generation tokens). HiDream ships open weights for the 8B variant (MIT license) and reports an internal 200B+ “Pro” variant; the 8B reaches GenEval 0.90 and DPG-Bench 89.83, beating Qwen-Image (27B) and matching or exceeding Nano Banana 2.0 and GPT Image 2 on the headline tables.

Key claims

A single decoder-only Transformer can host raw-pixel image generation, instruction-based editing, and multi-reference subject-driven personalization by mapping all three of (text tokens, condition images via SigLip-2 + learnable projection, target noisy patches via learnable patch embedding) into one shared token space [§3.2, Fig. 7].
Eliminating both the VAE and the disjoint pretrained text encoder (“Pixel-level Unified Transformer”, Figure 5c) is the architectural distinction from prior pixel-space DiTs (Figure 5b, which keep a frozen text encoder) and from latent DiTs (Figure 5a) [§1, Fig. 5].
A hybrid attention mask reconciles AR-LLM conventions with DiT conventions in one stack: text and condition tokens use causal masking and attend only to preceding tokens, while generation (noisy target) tokens use full self-attention over the entire sequence [§3.3].
Initializing the 8B backbone from a pretrained multimodal-understanding LLM (Qwen3-VL-8B-Instruct) — rather than from scratch — is the load-bearing choice for the unified pixel-space recipe, with RMSNorm + SwiGLU + RoPE preserved and only patch embeddings + the timestep “register” token added [§3.3].
The training objective combines flow-matching MSE in pixel space with two perceptual losses on the predicted clean image — LPIPS and a perceptual DINO loss — to recover semantic coherence that pixel-space MSE alone fails to capture [§3.4].
A separate “Reasoning-Driven Prompt Agent” built on Gemma sits before the generator and rewrites raw user instructions into a structurally aligned descriptive prompt via an explicit chain-of-thought (“thinking” mode) over layout, attributes, physics, and text rendering [§3.1].
Three-stage progressive pretraining at 512² → 1024² → 2048² resolution, with Stage I jointly optimizing T2I + language modeling + multimodal understanding on text+image+text-only data, Stage II adding editing + subject personalization tasks, and Stage III restricted to >2048² fine-detail refinement [§4.1].
Post-training adds SFT followed by RLHF via GRPO, with the SFT stage replacing logit-normal timestep sampling with uniform sampling to put more training emphasis on late-stage denoising [§4.2].
Fast inference is enabled via Adversarial Diffusion Distillation, producing a 28-step “Dev” variant that complements the 50-step “full” model [§5; README Models table].
At 8B parameters, HiDream-O1-Image matches or beats 27B Qwen-Image and ≥56B FLUX.2 [Dev] on GenEval (0.90 vs 0.87 / 0.87), DPG-Bench (89.83 vs 88.32 / 87.57), HPSv3 (10.37 vs 9.94 / 9.28), CVTG-2K complex visual text (0.9128 avg / 0.8076 CLIP), and LongText-Bench EN/ZH (0.979 / 0.978) [§6, README evaluation tables].
A scaled-up HiDream-O1-Image-Pro at 200B+ parameters extends the lead — GenEval 0.92, DPG-Bench 90.30, HPSv3 10.47, CVTG-2K 0.9222 avg / 0.8349 CLIP, LongText-Bench-ZH 0.980 — demonstrating that the natively-unified pixel-space recipe scales [§6, README tables].
HiDream-O1-Image-Dev-2604 debuts at #8 on the Artificial Analysis Text-to-Image Arena, positioned as the new leading open-weights T2I model [README project updates, leaderboard figure].

Method

The architecture decomposes the input stream into three token kinds — text tokens (Qwen3-VL tokenizer), condition tokens (SigLip-2 features over reference/source images projected to the shared space), and generation tokens (patchified noisy target $x_t = t \cdot x + (1-t)\varepsilon$ with learnable patch embeddings) — all interleaved into a single sequence with the diffusion timestep as an extra register token. The decoder-only Transformer backbone (8B initialized from Qwen3-VL-8B-Instruct; 200B+ Pro variant trained at scale) processes the sequence under a hybrid attention mask: text and condition tokens use causal masking, generation tokens use full attention. A linear prediction head maps each output token back to a clean image patch, which is unpatchified into the target image at up to 2048×2048.

Training data is built by a dedicated pipeline (Figure 6): source data collection across T2I, editing, subject IP, layout, style, and multi-panel; SSCD + Faiss two-stage deduplication (cluster-then-search, ~20% removal); quality filtering with NSFW + aesthetic + watermark + Top-IQ + bytes-per-pixel + VLM-based task-consistency checks; and Qwen3-VL-based prompt construction emphasizing factuality, spatial specificity, and varied prompt granularity. Progressive pretraining runs at 512² (T2I+LM+MMU joint), 1024² (add in-context editing + personalization), and 2048² (fine-detail refinement); post-training is SFT followed by GRPO RLHF. Fast inference uses Adversarial Diffusion Distillation to produce a 28-step Dev variant alongside the 50-step full model.

Results

On GenEval (compositional generation, 8B): Single-Object 1.00, Two-Object 0.99, Counting 0.79, Color 0.89, Position 0.93, Attribute 0.78, Overall 0.90 — beating Qwen-Image (27B, 0.87), FLUX.2 [Dev] (56B, 0.87), Nano Banana 2.0 (0.83), and GPT Image 1 [High] (0.84); the 200B+ Pro reaches 0.92 [Table, README]. On DPG-Bench (dense prompt alignment): 8B at 89.83 vs Qwen-Image 88.32, FLUX.2 [Dev] 87.57, Seedream-4.0 88.63, Nano Banana 2.0 86.90; Pro at 90.30. On HPSv3 (human preference across 12 categories): 8B at 10.37 (best in 9 of 12 sub-categories), GPT Image 2 at 10.21, Qwen-Image at 9.94. On CVTG-2K complex visual text generation (2–5 regions): 8B average 0.9128 / NED 0.9561 / CLIP 0.8076 vs Qwen-Image 0.8288 / 0.9116 / 0.8017 and FLUX.2 [Dev] 0.8926 / 0.9475 / 0.8104; Pro is best overall at 0.9222 / 0.9628 / 0.8349. On LongText-Bench: 8B at 0.979 EN / 0.978 ZH, Pro at 0.982 / 0.980 — best in the table.

The 8B model and a distilled Dev variant (28 inference steps, flash scheduler) are open-sourced on Hugging Face and ModelScope under MIT license, alongside the Reasoning-Driven Prompt Agent (which uses Gemma-4-31B-it via either a local backend or any OpenAI-compatible API). HiDream-O1-Image-Dev-2604 (May 14 update) was released with a dedicated Prompt-Refine model and debuted at #8 on the Artificial Analysis Text-to-Image Arena.

Why it’s interesting

HiDream sharpens the Pixel-space diffusion debate by going further than PixelGen (PixelGen: Improving Pixel Diffusion with Perceptual Supervision) and CAFM (Continuous Adversarial Flow Models): those papers kept a disjoint text encoder and just dropped the VAE; HiDream drops both and reuses an LLM’s tokenizer + native vocabulary as the text path. It is also a direct test of whether the Unified Multimodal Models thesis (one model for understanding + generation) extends past discrete-token AR (Janus-Pro) and AR+Diffusion (DuoGen — DuoGen: Towards General Purpose Interleaved Multimodal Generation) into pixel-space diffusion: the answer in this paper is “yes, and the LLM-init recipe is what makes 8B competitive with 27B Qwen-Image.” Open question this raises for the wiki: HiDream reports parity-or-better numbers on GenEval/DPG/HPSv3 against Qwen-Image and FLUX.2 [Dev] but does not run UniG2U-Bench (UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?) — so whether removing the VAE preserves the understanding side of the unified-model bargain remains untested.