Krea 2 Technical Report

TL;DR

Krea 2 is a series of MMDiT text-to-image foundation models from Krea positioned for creative exploration — broad stylistic coverage and steerability rather than a single polished default aesthetic. The technical report documents the full pipeline: an in-house data-curation stack (hash + phash+colorhash dedup, hierarchical k-means clustering, Wikipedia PageRank for entity coverage, SigLIP-2 sparse autoencoders for unsupervised tag-based filtering, deliberate exclusion of AI-generated images), an architecture ablation table that ends up at a simple single-stream MMDiT (SwiGLU, GQA + sigmoid-gated attention, RMSNorm, 3D axial RoPE, Qwen 3 VL text encoder with multilayer feature aggregation, FLUX 2 / Qwen Image VAE, per-block tunable bias replacing AdaLN MLPs), and a five-stage training pipeline (256→512→1024 px pretraining with iREPA and 8-bit training, midtraining, SFT, a DPO variant called STPO that fixes policy divergence, and multi-reward GRPO). The model sits in the top 10 of the Artificial Analysis text-to-image leaderboard and second among independent-lab models.

Key claims

Krea filters its pretraining mix to remove only duplicates, over-represented concepts, VLM-uncaptionable samples, bias/artifact inducers, high-complexity-at-low-resolution samples, and all AI-generated images — rejecting conventional aesthetic-/IQA-model-based filtering as introducing implicit bias (e.g. motion blur classified as “low quality”) [§Data Curation Principles].
Even a small proportion of AI-generated images in pretraining “imposes an upper bound on model quality” because synthetic images are easier to learn; Krea trains an in-house classifier to exclude them [§Data Curation Principles].
Default 8×8 phash has a high false-positive deduplication rate because it ignores color; Krea combines a 12×12 phash with colorhash for the low-resolution stages [§Pretraining Data].
A sparse autoencoder trained on SigLIP-2 embeddings of the pretraining corpus, with each feature VLM-annotated by its top-k activating samples, is used as an unsupervised tagging system to filter visual artifacts without training an explicit per-artifact classifier [§Pretraining Data].
Hierarchical k-means clustering (FAISS) plus PageRank over English Wikipedia (Danker, top 90% retained, ~5M Wikidata-filtered representable concepts) is used to ensure long-tail entity coverage at midtraining; sampling is biased toward captions referencing rare concepts [§Midtraining Data].
Architecture ablations land on: GQA + sigmoid-gated attention, SwiGLU MLP at 4× expansion, standard residuals, single-stream blocks, zero-centered RMSNorm with QKNorm, 3D axial RoPE (text tokens at index 0), Qwen 3 VL text encoder with multilayer feature aggregation, Qwen Image VAE / FLUX 2 VAE [§Architecture table].
Replacing per-block AdaLN modulation MLPs (20–30% of parameter count) with a per-block tunable bias term preserves quality and frees parameters for attention and MLPs [§Timestep conditioning].
In-context timestep conditioning via timestep tokens (no AdaLN) works at 256 px with 4–16 tokens but underperforms AdaLN at 512 and 1024 px with diminishing returns from adding more timestep tokens [§Timestep conditioning].
Hybrid-stream blocks (dual for first third, single for last two-thirds) slightly outperform pure single- or dual-stream, but the report adopts single-stream for simplicity [§Transformer block].
DC-AE’s 32× spatial compression imposes a hard upper limit on fine-detail generation owing to reconstruction error; Qwen Image VAE and FLUX 2 VAE converge much faster with better reconstruction, so Krea uses the Qwen Image VAE early and FLUX 2 VAE for larger models [§Autoencoder].
Partial RoPE (rotate only the first half of head dims) generalizes better zero-shot from 256→512 px without duplication artifacts, but underperforms full RoPE once high-resolution training continues [§Positional encoding].
iREPA is applied only for the first epoch of the 256 px stage and then removed, encouraging the MMDiT to learn its own representations while substantially accelerating initial convergence [§Pretraining].
8-bit training delivers 15–20% wall-clock speedup vs bf16 at 256/512 px with minimal degradation — tensorwise scaling at 256 px, finer-grained rowwise scaling at 512 px, bf16 from 1024 px onward through RL [§Pretraining].
PMA (post-merging averaging across pretraining checkpoints) achieves quality comparable to EMA without EMA’s memory overhead [§Pretraining].
A two-stage post-training stack: large-scale synthetic preference-pair generation (delta-learning style, with ≥1 on-policy sample per pair) followed by in-house human calibration [§Preference optimization].
The report identifies policy divergence during DPO: under many preference mixtures the model satisfies the DPO margin objective by reducing the likelihood of both winning and losing samples at unequal rates, drifting away from the pretraining distribution and producing high-frequency artifacts; Krea introduces STPO, a DPO variant with an auxiliary loss + formulation tweak that reduces this divergence [§Preference optimization].
Final RL stage uses a multi-reward GRPO-style method with separate reward models for general aesthetics, prompt following, text rendering, and one further category truncated from the fetched excerpt [§Reinforcement learning].
Muon (Dion implementation, Moonlight RMS-matched HP transfer) initially converges faster than AdamW but underperforms over longer horizons unless the first and last linear layers are excluded; after exclusion + Nesterov momentum, Muon consistently beats AdamW at both low and high resolution, though Krea did not adopt it for the final run owing to time constraints [§Optimization].

Method

The Krea 2 stack is organized as five sequential stages — pretraining (256 → 512 → 1024 px), midtraining, supervised finetuning, preference optimization (STPO), and multi-reward GRPO RL — sitting on top of a single-stream MMDiT (SwiGLU MLPs at 4× expansion, GQA with sigmoid-gated attention, RMSNorm + QKNorm, 3D axial RoPE, per-block tunable bias replacing AdaLN MLPs, Qwen 3 VL text encoder with a Unifusion-style multilayer feature-aggregation attention head plus bidirectional refinement layers, and either Qwen Image VAE or FLUX 2 VAE as the autoencoder). Pretraining uses standard rectified flow with v-parameterization, a warmup-stable-decay LR schedule, PMA model merging, and resolution-dependent timeshift sweeps; 8-bit training is used at 256/512 px (tensorwise then rowwise scaling) and bf16 from 1024 px onward; iREPA is enabled only for the first epoch of the 256 px stage.

The data pipeline is the load-bearing component the report spends most space on. Pretraining is bottom-up: massive corpora filtered with cheap CPU heuristics (broken file, resolution, aspect, Laplacian, RGB entropy, white/black pixel ratios), hash dedup (md5 + 12×12 phash + colorhash), and small (<1B) DINOv3/SigLIP-2 classifiers trained on VLM-pseudo-labeled data to remove specific artifact patterns (e.g. flat-color borders). A SigLIP-2 sparse autoencoder is trained on the corpus, each feature is VLM-annotated by top-k activations, and the resulting unsupervised tag system is used as an additional filter. Quality / aesthetic scores are used only to drop the worst, never to oversample.

Midtraining is top-down: hand-picked sources are clustered hierarchically with FAISS k-means, VLM-named, human-reviewed for cluster drops, semantically deduplicated with SigLIP, and rare-entity coverage is enforced by Wikipedia PageRank (Danker) over Wikidata-filtered representable concepts. SFT is a small hand-curated aesthetic set per visual domain, with domain-specific checkpoints merged into a generalist.

Two systems sit between the base model and end users: a prompt expander (two-stage SFT → RL on top of an open-source LLM, rewarded for both quality and creative variation) that maps short user prompts into rich captions matching the model’s training-time conditioning distribution; and a style-reference system that injects style/mood from one or more reference images with minimal content leakage and supports weighted multi-reference style mixing.

Results

Krea 2 lands in the top 10 of the Artificial Analysis text-to-image leaderboard and 2nd among models from independent labs [§Introduction]. The report is largely a methodology document and does not publish per-axis benchmark tables in the fetched excerpt; the headline-number context is the leaderboard placement plus the targeted axes (broad stylistic coverage, controllable exploration, text rendering, prompt following) the architecture and post-training stack were designed for. Quantitative ablation deltas for individual components (8-bit speedup 15–20% vs bf16; Muon faster-then-slower-than-AdamW curve; in-context timestep conditioning matches AdaLN at 256 px but degrades at 1024 px; partial RoPE wins zero-shot 256→512 then loses at longer training; DC-AE upper-bounds fine detail; Qwen Image / FLUX 2 VAEs converge fastest) are reported qualitatively in the ablation text.

Why it’s interesting

This is the most complete public account of a production T2I pretraining + post-training pipeline filed on the wiki to date, and it lands directly on top of several active debates. On data, it is the strongest filed counter-position to the “synthetic data scales pretraining” thread (Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a), A Bitter Lesson for Data Filtering) — Krea explicitly classifies and excludes AI-generated images, arguing they cap model quality — while sharing the DataRater: Meta-Learned Dataset Curation / The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data “filtering is the load-bearing axis” position. On architecture, the per-block AdaLN-MLP → tunable-bias swap, the gated sigmoid attention adoption (Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free), the single-stream choice over hybrid-stream (cf. Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model (daVinci-MagiHuman)) and the iREPA-then-off recipe (What Matters for Representation Alignment: Global Information or Spatial Structure?) read as a deliberate convergence with the late-2025 / early-2026 T2I-design consensus papers like i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models and Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer — and they share Krea’s diagnosis that AdaLN modulation parameters are wasted. On post-training, STPO (a DPO variant that names and fixes policy divergence by adding an auxiliary loss to keep both winning and losing likelihoods anchored to the pretraining distribution) is a concrete new entry in the DPO-fix lineage that includes Divergence Minimization Preference Optimization for Diffusion Model Alignment and DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models.