Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation

TL;DR

Hyper-Bagel is a divide-and-conquer acceleration recipe for BAGEL, the unified understanding+generation model. The next-token-prediction branch is sped up >2× with an EAGLE-3-style speculative decoder modified for interleaved multimodal sequences (meta-query target feature aggregation, zero-init residual FCs, and a relaxed cross-entropy + forward-KL loss). The diffusion-denoising branch is distilled in three stages — CFG distillation (text + image guidance scales baked into the timestep input), 3-segment adversarial TSCD with a multi-head discriminator, and a new DMD variant (DMDO) that keeps the teacher’s Euler ODE sampler instead of switching to the DMD-default SDE consistency sampler — yielding a lossless 6-NFE generation model that beats the 100-NFE BAGEL on GenEval overall (0.8647 vs 0.8640) and beats the 132-NFE BAGEL on GEdit-Bench (6.612 vs 6.602 EN). A further ADP + ReFL stage produces a 1-NFE model that approaches near-real-time editing.

Key claims

A unified acceleration framework can attack both branches of a UMM independently: speculative decoding for AR understanding, multi-stage distillation for diffusion generation, with the distilled generation weights swappable at inference (prefill from the original model, denoise with the distilled one) so the understanding branch’s quality is unaffected [§1, §4.1].
Vanilla EAGLE-3 transfers poorly to BAGEL because its multimodal context interleaves text tokens, ViT image tokens, clean diffusion latents (prefilled), and noisy latents; the limited-capacity draft model can’t model the embedding-space variation [§1, §3.1].
A meta-query intermediate layer that cross-attends learnable queries to all target-model layers (rather than EAGLE-3’s last-few-layer concatenation) gives the draft model richer aggregated features [§3.1.1, Fig. 3].
Zero-init on fc1, fc2, and the last projection of both the cross-attention and decoder blocks, combined with EAGLE-3-style low/mid/high skip connections, lets pretrained target weights guide the gradient signal from the start of training — removing this causes the largest measured degradation in the ablation [§3.1.2, Table 1].
A mixed forward-KL + cross-entropy (one-hot hard label) loss with α=0.1 outperforms vanilla EAGLE-3’s forward-KL only; the strict KL constraint is too tight for the limited-capacity draft model [§3.1.3, Eq. 2, Table 1].
The full Hyper-Bagel speculative recipe lifts TPS from 98.3 to ~212.4 on a single A100 in SGLang chain decoding, a 2.16× speedup with chain-mode decoding; average acceptance length 3.7709 vs vanilla EAGLE-3’s 3.6184 [§1, Table 1].
CFG distillation can fold both text and image guidance scales into a single forward pass by injecting each as an additional timestep-style embedding into every DiT block (FLUX-style), randomly sampling text scale ∈ [1, 5] and image scale ∈ [1.0, 2.5] during training; a timestep shift ≥3.0 is necessary for high structural integrity [§3.2.1].
Stage-2 (TSCD) drops MSE entirely, uses pure adversarial loss with a multi-head discriminator (scaled features from a trainable backbone), and runs 3 segments in a single stage rather than Hyper-SD-style progressive 8→4→2→1 distillation — the intuition being to enhance structural integrity here and defer fidelity to stage 3 [§3.2.2].
DMDO replaces DMD’s SDE consistency sampler with the teacher’s ODE Euler sampler. The fake-model update samples a full 6-NFE student trajectory and supervises the fake model on its velocity field at linearly-interpolated points along the trajectory; the generator-update step reuses these trajectories with gradients enabled. This eliminates DMD/DMD2’s need for GAN or ODE regularizers to compensate for the SDE-induced blur [§3.2.3, Fig. 4].
The 6-NFE Hyper-BAGEL is lossless on GenEval (overall 0.8647 vs 100-NFE baseline 0.8640) — a 16.67× speedup — and on GEdit-Bench (G_O 6.612 EN / 6.671 CN vs 132-NFE baseline 6.602 EN / 6.610 CN) — a 22× speedup [§4.2, Tables 2, 3].
ADP (adversarial diffusion pre-training on 6-NFE-sampled ODE trajectories with both latent- and pixel-space discriminators) plus ReFL using a single VLM-based reward model (HPSv3, built on Qwen2-VL) — rather than Hyper-SD’s multiple aesthetic rewards — yields a 1-NFE model at GenEval 0.7962 (comparable to Janus-Pro-7B / MetaQuery-XL at 0.80) and GEdit-Bench G_O 5.975 EN / 5.966 CN (above OmniGen’s 5.06) [§3.2.4, §3.2.5, §4.2, Tables 2, 3].

Method

The framework splits BAGEL acceleration into two largely independent branches that can be deployed and verified separately at inference. On the understanding side, an EAGLE-3-style draft model proposes multiple tokens that the target model verifies in parallel; the modifications are concentrated in the intermediate layer that bridges target and draft. Learnable meta queries cross-attend to all target-layer features per token, then a target-architecture decoder block further aggregates them. The last projection of both modules is zero-initialized, with EAGLE-3-style low/mid/high skip connections from the target’s forward pass. The two critical FCs (fc1 dim-reduction, fc2 token-embedding fusion at each iterative decoding step) are also zero-init; pretrained weights from the target’s last layers + LM head seed the draft. A mixed cross-entropy + forward-KL loss (α=0.1 CE) relaxes the distribution-matching constraint that EAGLE-3 imposes via pure forward KL.

On the generation side, distillation runs in three stages. Stage 1 (CFG distillation) injects two additional timestep-style embeddings — text guidance scale and image guidance scale — into every DiT block, with scales randomly sampled per training step; diffusion timestep shift is set to ≥3. Stage 2 (TSCD) does 3-segment consistency distillation in a single stage with pure adversarial loss and a multi-head discriminator over a trainable backbone, no MSE term, no progressive segment reduction. Stage 3 (DMDO) keeps the teacher’s Euler ODE sampler instead of switching to DMD’s SDE consistency sampler: the student samples a full 6-NFE ODE trajectory; the fake model is supervised on the velocity field at linear interpolations of consecutive trajectory points; the generator update reuses those trajectories with gradients enabled. A further 1-NFE finetune (ADP + ReFL) uses 6-NFE-sampled ODE trajectories as adversarial targets in both latent and pixel space, then a single VLM-based reward (HPSv3 on Qwen2-VL) with Hyper-SD’s ReLU(reward − 6.0) ReFL loss.

Results

Speculative decoding (Table 1): Average acceptance length 3.7709 / 10-step acceptance rate 0.7452 — best vs vanilla EAGLE-3 (3.6184 / 0.7327). Removing zero-init drops to 2.8273 / 0.6494 — the dominant component. Removing CE drops to 3.6642 / 0.7365. Removing both drops to 3.4832 / 0.7207. End-to-end: ~2.16× TPS speedup (98.3 → ~212.4) on a single A100 with SGLang chain decoding [§1, §4.2].
Image generation (Table 2): 6-NFE Hyper-BAGEL GenEval overall 0.8647 vs 100-NFE BAGEL reproduction 0.8640 — lossless at 16.67× speedup. 1-NFE Hyper-BAGEL 0.7962, comparable to Janus-Pro-7B and MetaQuery-XL at 0.80.
Image editing (Table 3): 6-NFE Hyper-BAGEL G_O 6.612 EN / 6.671 CN — beats 132-NFE BAGEL reproduction (6.602 / 6.610) at ~22× speedup. 1-NFE Hyper-BAGEL G_O 5.975 EN / 5.966 CN — surpasses OmniGen (5.06), Step1X-Edit (6.70), and the Instruct-Pix2Pix / MagicBrush / AnyEdit family by wide margins, trailing only the unaccelerated baselines and the 6-NFE student.

Why it’s interesting

This is the first wiki entry that combines the two main efficiency levers tracked separately on Diffusion Distillation (multi-stage score distillation, here with DMDO’s ODE-preserving twist) and LLM Inference Efficiency (EAGLE-3 speculative decoding adapted for interleaved multimodal sequences) on the same model. The DMDO contribution sits cleanly alongside Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis‘s DP-DMD and Cross-Resolution Distribution Matching for Diffusion Distillation‘s RMD as variations of “DMD with the sampler/architecture/objective swapped to better match the teacher” — DMDO’s specific lever is keeping the teacher’s Euler ODE rather than DMD’s SDE consistency sampler. The CFG-distillation stage also contributes a concrete two-axis (text + image scale) variant to Classifier-Free Guidance variants. Practically: Jiaming’s read in the Slack thread — “do step distillation, skip spec decoding this time” — tracks the paper’s own ablation, which shows the diffusion side is where the order-of-magnitude (16×–22×) speedups live; the speculative decoding contribution is 2.16× and is the harder of the two to transfer cleanly to a different multimodal architecture because of the EAGLE-3 retraining cost (cf. TorchSpec: Speculative Decoding Training at Scale).