C3Net: Compound Conditioned ControlNet for Multimodal Content Generation

TL;DR

C3Net is a tri-modal (image + text + audio) diffusion architecture that takes a compound multimodal condition — any subset of the three modalities provided simultaneously — and synthesizes outputs in all three. It adapts ControlNet’s frozen-backbone + trainable-copy recipe: modality-specific encoders project conditions into a shared CLIP latent space, a per-modality “Control C3-UNet” injects each condition’s information through zero-conv skip connections into a single C3-UNet backbone borrowed from CoDi. The headline change over CoDi is that conditions are learned-combined through ControlNet-style additive control rather than linearly interpolated in latent space, and the alignment encoders are unimodally pre-trained (MAE / SSAST audio encoder) before contrastive fine-tuning so they survive scarce paired data.

Key claims

Adapting ControlNet’s zero-conv injection per modality lets a single trainable copy of the diffusion backbone serve as the multimodal combiner, because all conditions already live in the shared CLIP latent — one Control C3-UNet, not one per modality pair [§3.3].
The Control C3-UNet’s contribution is scaled by a small constant (empirically 0.1) and set to zero when only one condition is provided, so the model gracefully falls back to standard CoDi-style single-condition generation [§3.3].
Unimodal pre-training of alignment encoders (MAE-style for audio, using SSAST-style weights from [8]) outperforms CoDi’s trained-from-scratch alignment on the AudioCap ablation where only the alignment stage differs (CIDEr-D 0.0704 vs 0.0654 on text from audio; Inception 1.7732 vs 1.7730 and CLIP 23.325 vs 23.192 on image from audio) [Tables 1–2, §4.3.1].
On a curated 2,000-tuple tri-modal test set (audio + image + text from AudioCap, image generated with Stable Diffusion and filtered by CLIP score), C3Net beats CoDi on image FID (10.97 vs 11.39) and CLIP (25.29 vs 25.17) [Table 3, §4.3.2].
C3Net also beats CoDi on text-from-compound metrics — BLEU-1, ROUGE-L, CIDEr-D, SPIDEr — across the same tri-modal set [Table 4].
On audio synthesis C3Net beats CoDi on the human-judgement-style OVL/REL metrics but is slightly worse on FAD (11.7 vs 11.4), suggesting the ControlNet injection improves perceived text-relevance but mildly trades off raw distribution match [Table 5, §4.3.2].
Training the Control C3-UNet only requires unimodal supervision per modality (image–text for image/text generation, audio–text for audio generation); no all-pairs multimodal dataset is needed [§3.3].

Method

The pipeline has two stages. Alignment: each modality has a dedicated encoder that projects the condition into the CLIP latent space. Audio and image encoders are unimodally pre-trained (audio uses an SSAST-style MAE pretrained on unlabeled audio per [8]) and then contrastively fine-tuned against a frozen CLIP text encoder, so audio-from-text-caption alignment is inherited rather than learned from scarce paired data.

Diffusion: the backbone is a U-Net taken directly from CoDi (12 encoder blocks across 4 resolutions, 1 middle block, 12 skip-connected decoder blocks). For each input modality the system instantiates a Control C3-UNet — a trainable copy of the 12 encoder + 1 middle blocks. The frozen C3-UNet receives a linear interpolation of the aligned condition latents (the CoDi recipe); each Control C3-UNet receives a single modality’s latent and feeds its features back into the C3-UNet through 13 zero-init convs (one per skip / middle), scaled by a small constant (≈0.1). When only one condition is provided the constant is set to 0, recovering CoDi behavior. Training is per modality on unimodal pairs (image–text from COCO + LAION-400M slice, audio–text from AudioCap + AudioSet + Epidemic-Sound). The objective is standard noise-prediction MSE conditioned on the aligned latent tuple with 50% random text-mask augmentation.

Results

AudioCap ablation (alignment-only diff vs CoDi, audio→text and audio→image): C3Net wins on every reported metric, but margins are small — Inception 1.7732 vs 1.7730, CLIP 23.325 vs 23.192 [Tables 1–2].
Tri-modal compound conditioning (curated 2,000-tuple set, audio+image+text→ each output modality):
- Image: FID 10.97 vs CoDi 11.39; CLIP 25.29 vs 25.17 [Table 3].
- Text: BLEU-1 0.1104 vs 0.1059; CIDEr-D 0.0713 vs 0.0651 [Table 4].
- Audio: OVL 63.25 vs 62.91; REL 59.83 vs 59.01; FAD slightly worse, 11.7 vs 11.4 [Table 5].
ESC-50 synthesized-audio classification (audio synthesized from audio+text, then classified by [3]‘s model): 23.25% vs CoDi 21.05% [Table 6, §4.3.3].

Gains over CoDi are consistent but small in absolute terms — this is a 2023 architecture paper proposing a recipe more than a scaling result.

Why it’s interesting

The paper is a concrete instance of using a ControlNet-style injection — frozen backbone + per-condition zero-conv trainable copy — to handle multiple conditioning streams at once, with the trick that aligning all streams into one latent space lets a single trainable copy serve every modality. That recipe shows up again, in spirit, in Exploring MLLM-Diffusion Information Transfer with MetaCanvas‘s “canvas connector” (a ControlNet-style zero-init connector with AdaLN-on-timestep that injects MLLM spatial features into a frozen DiT) — the family of “use a small trainable side-network to thread extra signal into a frozen large diffusion backbone” is durable. Compared to the more recent LTX-2: Efficient Joint Audio-Visual Foundation Model (joint audio+video generation via a dual-stream MMDiT), C3Net is a much earlier and simpler take on the joint-modality problem: it doesn’t try to allocate capacity asymmetrically per modality, and it only generates one modality at a time conditioned on the rest. It’s useful prior art for any discussion of how to combine multimodal conditioning streams without training an all-pairs model.