Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models (AHA)

TL;DR

AHA is a 6.6K-QA evaluation suite (AHA-Eval) plus a 120K-QA mitigation dataset (AHA-Guard) for probing whether Large Audio Language Models (LALMs) actually ground responses in the audio input or hallucinate from language priors. The attack surface spans three axes — query structure (explicit vs implicit phrasing), language priors (random vs adversarially correlated sound prompts), and audio manipulation (injecting synthetic speech describing nonexistent events) — and reaches Attack Success Rates of 95.35% on Audio Flamingo 3 and 79.65% on Gemini 3 Pro, gaps invisible in standard MMAU/MMAR-style benchmarks. Chain-of-Thought prompting fails on implicit attacks; DPO fine-tuning on AHA-Guard cuts hallucination rates by up to 49% on Qwen2.5-Omni. The dataset (project page hosted by UMD + Adobe Research authors) is positioned as both a benchmark and a post-alignment corpus.

Key claims

LALMs frequently bypass the grounding step (verifying a sound exists in the audio) when the query presumes the sound’s existence; the same model that correctly answers “Is the sound of seagulls audible?” hallucinates a distance estimate when asked “How far away does the sound of the seagulls seem to be coming from?” [Introduction].
AHA-Eval contains 6.6K QA pairs over AudioCaps and MusicCaps spanning two attack surfaces — query-based (explicit / implicit phrasing × random / adversarial target sounds) and audio-based (injecting a synthetic-speech utterance like “I can hear a toilet flushing” into the audio stream) [Methodology, Main Contributions].
Audio Flamingo 3 reaches 95.35% ASR and Gemini 3 Pro reaches 79.65% ASR under the strongest attack combination (audio-injection + implicit + adversarial) [Table 1, Main Contributions].
Audio-based attacks substantially outperform text-only attacks: for Audio Flamingo 3, random-explicit ASR jumps from 1.90% (text only) to 53.40% (with audio injection) [Table 1, Result narrative].
Implicit queries consistently raise ASR over explicit queries across all evaluated models — e.g. Gemini 3 Pro random-sound ASR 10.88% → 59.67% just by switching from explicit to implicit phrasing (a 48.79-point gap) [Table 1].
Mean attention to audio tokens is lower under implicit queries than explicit ones (measured on Qwen2.5-Omni at each generation step), giving a mechanistic correlate for the implicit-attack vulnerability [Investigating the cause of hallucinations].
Adversarially correlated sound prompts (sounds plausibly co-occurring with the actual audio scene) induce high model confidence — “yes” log-probability on hallucinated responses is high for ~75-80% of adversarial events vs ~20-25% of random ones [Investigating the cause of hallucinations].
Chain-of-Thought prompting reduces ASR on explicit attacks but is ineffective on implicit attacks (and can even increase implicit ASR — Qwen2.5-Omni baseline-CoT implicit-random text-only 68.74% → 82.90%) [Table 2].
DPO fine-tuning on AHA-Guard (120K QA pairs) cuts implicit-random ASR on Qwen2.5-Omni from 68.74% to 39.01% (text-only) and from 79.19% to 40.24% on implicit-adversarial — but the audio-injection + implicit-adversarial setting only drops from 96.71% to 86.63%, i.e. mitigation is incomplete on the strongest attack combination [Table 2].
Newer / closed models are not immune: GPT-4 Audio reaches 96.32% ASR (audio + explicit + adversarial), and Gemini 3.1 Pro at 50.8% on audio + implicit + adversarial — improved over Gemini 3 Pro but still high in absolute terms [Table 1].

Method

AHA constructs each test sample by pairing a real audio clip (from AudioCaps or MusicCaps) with an artificial “premise” that a specific sound is present. The premise is delivered through one of two channels: (a) the text query itself, either explicit (“Is X audible?”) or implicit (“When does X occur?” / “Describe X in detail”); and (b) the audio stream, by prepending a synthetic female-voice utterance like “So, I can hear a toilet flushing” to the real audio. The target sound is drawn either at random from a vocabulary of common AudioCaps tags (random condition) or chosen to be adversarially correlated with the scene — e.g. a hi-hat over a hip-hop track that contains snares and crash cymbals but no hi-hat (adversarial condition). Each (audio, premise, query) triple yields one QA pair; ASR is computed as the fraction of model responses that confirm and elaborate on the absent sound rather than denying it.

For mitigation, AHA-Guard pairs each of 120K queries with two model responses — one grounded (correct denial) and one hallucinated (confident elaboration on the absent sound) — and trains the target LALM (Qwen2.5-Omni) via Direct Preference Optimization with the grounded response as the chosen completion. The post-alignment is compared head-to-head against inference-time Chain-of-Thought.

Results

The headline ASR numbers in Table 1 show that on the strongest combination (audio-injection + implicit + adversarial), every evaluated LALM hallucinates more than half the time: Qwen3-Omni 95.72%, Audio Flamingo 3 99.19%, AudioFlamingo-Next 98.9%, AudioFlamingo-Next (Think) 93.0%, MOSS-Audio-4B-Think 95.2%, GPT-4 Audio 95.93%, Gemini 3 Pro 79.65%, Gemini 3.1 Pro 50.8%. Even Gemini 3.1 Pro, the strongest model evaluated, still confabulates the requested sound on a majority of implicit queries. Mitigation results in Table 2 show DPO on AHA-Guard is the only intervention that reduces implicit-attack ASR (e.g. text-only implicit-random 68.74% → 39.01%, a 29.73-point drop); CoT actively worsens it. The audio-injection + implicit-adversarial cell — the worst case — drops from 96.71% to 86.63% with DPO, indicating the attack surface is not fully closed.

The qualitative examples are striking: Audio Flamingo 3, given a clip of a helicopter engine with a prepended “I can hear a toilet flushing” utterance, returns “The toilet flushing sound occurs from 1.7s to 2.7s in this audio.” Gemini 3 Pro, given a ballad song with “I can hear a piano playing” prepended, returns a fully fabricated 9-second timeline placing the piano at specific timestamps. These are not stochastic single-token slips; the model commits to and elaborates on the false premise.

Why it’s interesting

This is a benchmark/dataset paper rather than a model paper, but it’s the first filed artifact in the wiki that directly stress-tests audio language models — a gap given how many filed papers ship audio-capable unified models without an audio-side robustness story (Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing reports MMSU 56.83 / MMAU 63.30 for Audio-Omni but does not run a hallucination-attack benchmark; Qwen3.5-Omni: A Native Omni-Modal Model with Thinker-Talker MoE Architecture and Ming-flash-omni 2.0 similarly report only standard understanding scores). Inkyu’s note (cc audio-omni team) suggests AHA-Guard is being eyed as a post-alignment dataset for Luma’s own audio-omni line — the 49% reduction on Qwen2.5-Omni is the relevant baseline number. Methodologically the paper is the audio analog of Universal and Transferable Adversarial Attacks on Aligned Language Models but for hallucination rather than jailbreaking: a reproducible, multi-axis attack suite with a paired mitigation corpus, exposing a reliability gap that standard benchmarks hide. The implicit-vs-explicit gap and the audio-attention-drop mechanism also sharpen the broader question on Unified Multimodal Models of whether MLLM-grounded audio generators actually use the audio encoder’s output or fall through to the LLM’s prior.