Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning

TL;DR

PE-AV (Perception Encoder Audiovisual) is a Meta family of audio + video encoders that extends the existing PE image-encoder line into audio and joint audio-video representations via scaled contrastive learning. The headline contribution is a data engine that synthesizes high-quality captions for O(100M) audio-video pairs, supplying consistent supervision across audio, video, and text. Ten pairwise contrastive objectives are trained jointly, and scaling the cross-modality and caption-type pair count is reported to strengthen alignment and zero-shot transfer. Audio coverage spans speech, music, and general sound effects rather than the single-domain corpora common in prior audiovisual representation work, and the resulting encoders set new state-of-the-art on standard audio and video benchmarks while unlocking new tasks like speech retrieval through unified cross-modal embeddings [Abstract].

Key claims

A single encoder family supports joint embeddings across audio-video, audio-text, and video-text simultaneously, built on PE as the visual backbone [Abstract].
A synthetic-caption data engine generates high-quality captions for O(100M) audio-video pairs to supply cross-modally consistent supervision at scale [Abstract].
Audio coverage is deliberately broad — speech, music, and general sound effects — avoiding the single-domain restriction common in prior audiovisual encoder work [Abstract].
Ten pairwise contrastive objectives are trained together; scaling the number of cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance [Abstract].
Unified cross-modal embeddings unlock new task formulations such as speech retrieval through the shared embedding space [Abstract].
PE-AV sets a new state of the art across standard audio and video benchmarks (specific numbers not in the abstract) [Abstract].

Method

PE-AV builds on Meta’s Perception Encoder (PE) image-encoder line, extending it to audio and to joint audio-video by adding modality-specific encoders for audio and video and training them jointly with text via contrastive objectives. The training corpus is O(100M) audio-video pairs, each paired with synthetic captions produced by an audiovisual data engine that the paper introduces as a core contribution; the captioner is conditioned on both the audio and the video so the resulting text supervision is consistent across modalities. Ten pairwise contrastive losses (over the cross-product of {audio, video, text} embeddings × the cross-product of caption types) are summed during training. The audio data deliberately spans speech, music, and general sound effects so a single encoder covers domains usually addressed by separate models [Abstract].

Results

The abstract reports new top scores across standard audio and video benchmarks and demonstrates novel cross-modal tasks (e.g. speech retrieval) made possible by the unified embedding space; specific numbers are not given in the abstract and would need a body read to extract [Abstract].

Why it’s interesting

PE-AV is the audiovisual-encoder analogue of the MobileCLIP2: Improving Multi-Modal Reinforced Training thesis — the architectural backbone is held roughly fixed (PE) and the contribution lives in the data and caption engine that supplies cross-modally consistent supervision at scale. That makes it a clean fit for Synthetic Training Data: O(100M) captioned pairs are not human-collected but synthesized by a model-driven pipeline, and the ten-pair contrastive setup is essentially a verification-via-redundancy structure (each modality is pulled into alignment with the others through multiple paths). Contrast with Introducing SAM Audio: The First Unified Multimodal Model for Audio Separation, which unifies audio separation across speech / music / sound effects in one model on the output side; PE-AV does the same domain unification on the representation side. And against DINOv3: Self-supervised learning for vision at unprecedented scale — also a Meta encoder-at-scale paper, but purely self-supervised on vision — PE-AV is the contrastive, multi-modally-supervised counterpart with text and audio added.