Approaching an unknown communication system by latent space exploration and causal inference

ListenTalking to Whales With a GAN's Latent Space10:45

Show notes

This week we dig into **"Approaching an unknown communication system by latent space exploration and causal inference"** by Beguš, Leban, and Gero. The team trains an information-theoretic GAN on raw sperm-whale coda audio, then treats the generator as a causal experiment — pushing individual latent bits to extreme values and using ATE/ICE estimators to read out which observable properties each bit controls. The method (CDEV) recovers the two coda features biologists already use and surfaces two novel acoustic axes later confirmed in real whale data. We close on what this same-noise, swept-treatment primitive could mean for probing conditioning in video-generation models. **Sources** - Approaching an unknown communication system by latent space exploration and causal inference (Beguš, Leban, Gero)

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

The paper trains an information-theoretic GAN (fiwGAN, a WaveGAN + InfoGAN variant) on raw sperm-whale coda audio, then treats the trained generator as a causal experiment: it drives individual latent bits to values outside the training range and applies Neyman–Rubin-style estimators (ATE, incremental causal effect) to detect which observable properties of the generated audio those bits control. The authors call this recipe causal disentanglement with extreme values (CDEV). On a five-bit encoding trained over five common coda types, CDEV recovers the two properties marine biologists already use to classify codas (click count, click regularity) and two novel properties never previously hypothesized as meaningful (coda-level spectral mean, and within-coda spectral regularity). A follow-up paper (Beguš et al. 2023) confirmed the novel properties correspond to formant-like and dipthong-like structure in real whale communication.

Key claims

The fiwGAN latent code partitions into random incompressible noise $z$ and a small featural encoding $c$ trained via mutual-information loss; disentanglement is only loosely enforced during training and typically becomes readable only when $c$ is pushed outside the training range where the primary encoded effect starts to dominate [§2].
Formulating latent-code manipulation as a continuous-treatment Neyman–Rubin experiment ( $z$ = covariate, $c$ = dose, observable = outcome) with the same $z$ replayed across all dose levels eliminates the fundamental problem of causal inference (all potential outcomes are observed per unit), simplifying estimation to sample-mean differences [§2.1].
Two estimators are used: (i) ATE relative to a baseline dose ( $c=-1$ or $c=+1$ ), which reveals disentanglement as a “one bit stabilizes at a large effect, others converge to a common stationary value” pattern outside the training range; (ii) incremental causal effect (ICE) — the expectation of the outcome derivative — which sidesteps baseline choice and reduces to a sample mean of finite differences [§3].
On the number of clicks per coda, bit 1 dominates in both ATE (relative to $c=-1$ and $c=+1$ ) and expected-infinitesimal-shift ICE (0.096 vs at most 0.046 in absolute value for other bits), and simultaneously controls the range (std) of clicks — the same bit encodes count and count-variance [§3, Table 1, Fig. 4].
Bit 1 also monotonically encodes mean inter-click interval and coda regularity (decreasing ICI std with dose), stratified by click count — the generator learns the empirically-known correlation “codas with more clicks are more regular and more closely spaced” without observing high-click codas in the training data (data-quality filter excluded them) [§4, Fig. 5, Tables 2–3].
Two acoustic properties emerge as separately encoded: bit 0 controls the per-coda mean spectral frequency (ATE ≈ +130 Hz above training range vs stationary ≈ -5 to -15 Hz for the other bits), and bit 3 controls within-coda spectral regularity (std of per-click spectral means; ICE ≈ +21 above training range vs -10 to -30 elsewhere) [§5, Tables 4–5, Fig. 6].
Wasserstein distance between the average coda-level spectrum at treatment $c$ and the $c=0$ baseline continues to grow with $c$ for bits 0, 1, and 3 — the three bits with attributed observable effects — while bits 2 and 4 saturate near the training-range limit, i.e. the disentanglement diagnostic agrees across three different estimators [§5, Fig. 6 bottom].
The novel acoustic hypotheses this method surfaced were subsequently verified by an independent analysis of real whale data (Beguš et al. 2023), which identified vowel-like and diphthong-like patterns matching the per-click spectral-mean and within-coda spectral-regularity axes CDEV had flagged [§5, §6].

Method

fiwGAN is used as an out-of-the-box learning mechanism: raw sperm-whale coda audio (2,209 samples restricted to the five most common coda types) is fed to a WaveGAN generator whose latent input is split into 95-D Gaussian noise $z$ and a 5-D Bernoulli-trained featural code $c$ ; an auxiliary Q-network reconstructs $c$ from generated audio, and its cross-entropy loss with the sampled $c$ is backpropagated through the generator to encourage $c$ to carry semantically consistent information.

At inference, the CDEV procedure sweeps $c$ along one axis at a time, holding the other bits at 0, and scans $c \in [-1, c_{\max}]$ with $c_{\max}$ substantially above the training range (chosen so detection-algorithm noise stays bounded — see §2 and Appendix A.4). For each treatment level, $N$ different $z$ vectors are sampled and reused across all treatment levels, so every “unit” $z_i$ is observed at every dose. Observable properties — click count, mean ICI, ICI std, per-click spectral mean, spectral-mean std — are extracted from the generated waveforms by algorithmic detectors (a robust click detector with signal filtering and even-spacing tie-breaking; periodogram averaging for spectral quantities), then fed into ATE and ICE estimators. Disentanglement is diagnosed by the pattern “one bit stabilizes at a large stationary effect above the training range while the others converge to a common baseline.”

Results

Click count (§3): bit 1 sign-score dominates all others; ATE relative to $c=+1$ shows bit 1 continues to grow through $c \approx 5$ while other bits plateau; the same bit also controls click-count std.
Click timing (§4): bit 1 monotonically shortens mean ICI and reduces ICI std across codas stratified by click count; sign scores are −10 for bit 1 vs +4/−4 for other bits on mean ICI, and −10 vs +6/−2/−8/−2 on ICI std.
Spectral mean (§5): bit 0 ICE = +62 over the full range and +128 above the training range vs at most ±55 for other bits.
Spectral regularity (§5): bit 3 ICE = +7 (full range) and +21 (above training range) vs −20 to −35 for other bits; Wasserstein distance from the $c=0$ average spectrum grows further for bits 0/1/3 than for bits 2/4 as $c$ increases.
External validation: the two novel per-click / per-coda spectral axes CDEV surfaced were later shown in Beguš et al. 2023 to correspond to human-vowel-like (single vs two spectral maxima per click) and human-diphthong-like (rising / falling / level formant trajectories) patterns that occur in real coda dialogues independently of the traditional coda-type taxonomy — direct empirical validation of the disentanglement CDEV claims [§5, §6].

Why it’s interesting

The recipe is a direct methodological cousin of the concept-vector line the wiki has been collecting under Mechanistic Interpretability — Anthropic’s emotion vectors, Universal SAEs, and Natural Language Autoencoders all steer or read out behavior along latent directions that the model wasn’t explicitly told to organize. CDEV pushes on the same disentanglement lever but in a generative rather than discriminative setting, and it does so with the language of continuous-treatment causal inference (ATE / ICE) rather than of steering-vector projection. That framing — same-noise, swept-treatment, sample-mean differences — is a cleaner statistical primitive than “add-and-project” for asking which observable output property does this latent direction control, and could plausibly be re-applied to identify what specific dimensions in a video-generation model’s noise / conditioning encode (motion strength, camera motion, text-alignment strength, etc.). It also contrasts with the Emergent Representation Geometry thread: CDEV finds that a five-bit code trained on raw audio spontaneously aligns with human-classified perceptual axes (click count, timing regularity) and discovers axes that human researchers hadn’t posited (spectral mean, spectral regularity) — Platonic-representation-hypothesis-adjacent evidence that the “right” axes for a communication system are recoverable from a purely imitative loss.