ERNIE 4.5 Technical Report

TL;DR

ERNIE 4.5 is Baidu’s family of 10 open-weights (Apache 2.0) foundation models — fine-grained MoE in two sizes (300B-A47B text-only and 424B-A47B multimodal, plus a 21B-A3B / 28B-A3B pair and a 0.3B dense checkpoint) — built around a heterogeneous MoE where text and vision tokens are routed to disjoint expert pools (vision experts at 1/3 the intermediate dim) while sharing the self-attention parameters and a set of shared experts. The report pairs this architectural choice with a modality-isolated router, a router-orthogonalization auxiliary loss, a token-balanced multimodal LBL, FP8 mixed-precision pretraining at 47% MFU on 2,016 H800s, and a W4A8 / 2-bit inference stack that fits the 424B model on 4× 80GB or 1× 141GB GPUs.

Key claims

The MoE backbone uses modality-isolated routing — text tokens go to text experts, vision tokens go to vision experts, both pass through shared experts and shared self-attention — to avoid the routing collapse observed when extending a text-only MoE into multimodal training [§2.1].
Vision experts use an intermediate dimension one-third the size of text experts, giving ~66% fewer FFN FLOPs per vision token, and visual experts are excluded from the final transformer layer because their weights do not affect the next-token CE loss [§2.1].
A heatmap of per-layer expert activations shows text experts have concentrated activations while vision experts are more dispersed, used by the authors to motivate modality-isolated routing [§2.1, Fig. 2].
Two new training-objective ingredients: a router orthogonalization loss that pushes router weight vectors for different experts toward orthogonality, and a token-balanced loss for multimodal data so that gradient mass is not dominated by the longer modality [§3.4.1, §3.4.2].
Three-stage pretraining curriculum: Stage I text-only, Stage II vision-only (with the language backbone frozen / partially frozen), Stage III joint multimodal — vision experts are introduced late to “avoid routing collapse” and reduce overall compute, on the rationale that visual understanding builds on prior textual knowledge [§3.3, §2.1].
47% Model FLOPs Utilization is achieved when pre-training the largest ERNIE 4.5 language model on 2,016 NVIDIA H800 GPUs, using a heterogeneous hybrid-parallelism plus hierarchical load-balance recipe tailored to multimodal training [Abstract, §5.1, §5.2].
The framework drops Tensor Parallelism at the MoE FFN in favor of intra-node expert parallelism, pairs it with a memory-efficient pipeline schedule, FP8 mixed-precision linears, FlashMask for flexible attention masks and long-context training, and fine-grained recomputation tuned for compute-vs-memory tradeoffs [§5.2.1, §5.2.2, §5.3, §5.4].
The 424B-total VL model is deployable on 4× 80GB GPUs in 4-bit (W4A8) or 1× 141GB GPU in 2-bit, using a convolutional-code quantization algorithm for near-lossless 4-/2-bit weights and a separate attention + KV-cache quantization path [Abstract, §6.1.1, §6.1.2, §6.1.3].
With Prefill-Decode disaggregation and expert parallelism, the largest language model reaches 56k input TPS and 18k output TPS per H800 node [Abstract, §6.3].
The vision encoder is an adaptive-resolution ViT — height and width independently resized to multiples of the patch size to preserve aspect ratio — with 2D RoPE and image-packing; videos use an adaptive frame-resolution sampling strategy that trades number of frames against per-frame resolution against the sequence-length budget, plus timestamp rendering (absolute time overlaid on each frame) for explicit temporal cues [§2.2].
3D RoPE encodes temporal, height, and width axes with separate frequency bands inside a single unified embedding space; lower frequencies are allocated to the temporal axis “which varies most slowly” [§2.4].
A bitwise-deterministic data manager, REEAO (“Record Everything Everywhere All at Once”), supports reproducible and non-redundant data access across training runs [§3.2].
Post-training uses SFT + DPO or a modified RL recipe UPO (Unified Preference Optimization) for the LLMs; the VLMs use SFT + Reinforcement Learning with Verifiable Rewards and support both thinking and non-thinking modes [§4.1, §4.2, Table 1].

Method

ERNIE 4.5 is a fine-grained MoE transformer that accepts text, image, and video inputs and emits text. The MoE FFN block is partitioned into three expert groups — text experts, vision experts, and a small set of shared experts — and a modality-isolated router sends each token to the experts of its own modality (plus the shared experts, in non-routed fashion). Self-attention parameters are shared across modalities. Vision experts have 1/3 the intermediate dim of text experts and are absent in the final transformer layer.

The vision path is an adaptive-resolution ViT with 2D RoPE feeding an adapter that does both spatial (2×2 pixel-shuffle) and temporal (×2) token compression before projecting into the shared embedding space. Static images are duplicated into “synthetic two-frame videos” so the adapter has a single temporal pipeline. Position embedding inside the shared transformer is 3D RoPE for visual tokens and 1D RoPE for text tokens, allocated from a single frequency budget.

Pretraining is three-stage (text-only → vision-only → joint), with two loss-side innovations on top of the standard next-token CE: a router-orthogonality loss that decorrelates expert-selection directions, and a token-balanced loss that compensates for the imbalance between text and visual token counts inside each batch. The training system uses heterogeneous hybrid parallelism (different strategies for the visual front-end and the MoE backbone), intra-node expert parallelism, a memory-efficient pipeline schedule, FP8 GEMMs, FlashMask, and fine-grained recomputation; the largest language model hits 47% MFU on 2,016 H800s.

Inference relies on a W4A8 quantization stack (with attention + KV-cache quantization separately), a 2-bit variant via convolutional-code quantization, hardware-optimized W4A8 kernels, and Prefill-Decode disaggregation with dynamic role switching.

Results

The 424B-total VL model fits on 4× 80GB GPUs (4-bit) or 1× 141GB GPU (2-bit), with throughput of 56k input / 18k output TPS per H800 node at the largest scale [Abstract, §6.3].
47% MFU when pretraining the largest language model on 2,016 NVIDIA H800 GPUs [Abstract, §5].
The release covers 10 variants spanning MoE (47B and 3B active) and a 0.3B dense baseline; six are post-trained (DPO/UPO for LLMs; SFT+RLVR for VLMs), and the VL variants support both thinking and non-thinking modes [Table 1].
The report claims top scores on multiple text and multimodal benchmarks, with the strongest gains reported in instruction following, world knowledge memorization, visual understanding, and multimodal reasoning — full benchmark tables are in §8 (Evaluation of Language Models, Evaluation of Multimodal Models) but specific headline numbers were truncated from the fetched body.
All weights and the ERNIEKit / FastDeploy toolkits are released under Apache 2.0; code on GitHub (PaddlePaddle/ERNIE) and weights on Hugging Face (baidu) [Abstract].

Why it’s interesting

ERNIE 4.5 is one of the cleanest filed datapoints for modality-isolated fine-grained MoE: it argues, by hold-out activation heatmaps, that text and vision experts naturally specialize and that letting a single router score both modalities causes collapse — a position that sits directly on the axis MoE Routing Design tracks. It complements Path-Constrained Mixture-of-Experts (which constrains router weight sharing across layers) and Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models (which rescopes the LBL across the batch) with a third lever — modality-conditioned routing — and adds two new loss ingredients (router-orthogonalization and token-balanced LBL) to the cluster’s vocabulary. The 47% MFU at 2k-H800 scale on a multimodal MoE is also a useful peer to MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production (ByteDance’s 1.88× MFU-over-Megatron production system) and Scalable Training of Mixture-of-Experts Models with Megatron Core (Megatron-Core MoE), with the ERNIE recipe sitting closer to MegaScale-MoE in dropping TP at the FFN. Finally, it backfills the Baidu ERNIE-4.5 backbone behind PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model (whose 0.9B PaddleOCR-VL system uses the ERNIE-4.5-0.3B dense variant as its language model) — the dense 0.3B referenced there is now a filed artifact.