PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models

TL;DR

PersonaPlex is a 7B full-duplex conversational speech model built on the Moshi architecture that breaks the long-standing trade-off between voice/role customization (ASR→LLM→TTS cascades) and natural turn-taking (single-voice full-duplex models like Moshi). It accepts two conditioning inputs — a voice prompt (audio embedding for vocal style) and a text prompt (natural-language role description) — and produces speech that simultaneously preserves Moshi-style backchannels and interruptions while adhering to the prompted persona. Training is a single-stage blend of 1,217 hours of real Fisher English conversations (back-annotated with LLM-generated persona prompts) and ~2,250 hours of synthetic assistant + customer-service dialogues generated by Qwen3-32B / GPT-OSS-120B and synthesized via Chatterbox TTS. Reported to outperform open and commercial baselines on FullDuplexBench plus a new ServiceDuplexBench extension along conversational dynamics, latency, and task adherence axes. Code and weights are released (MIT for code, NVIDIA Open Model License for weights).

Key claims

A single full-duplex model can be made voice- and role-conditional by adding two prompt modalities (audio embedding + natural-language text) to a Moshi-style backbone, without giving up the listening-while-speaking dynamics [Capabilities §Full Duplex, Architecture].
Persona-supervised training data can be created retrospectively: an LLM (GPT-OSS-120B) back-annotates unscripted Fisher English conversations with contextual + personality descriptors per speaker, converting an undirected corpus into a directed one [Training data §Real conversations].
Real and synthetic conversational audio play complementary roles — real Fisher recordings supply the behavioral richness (backchannels, interruptions, prosody) that TTS cannot simulate, synthetic dialogues supply task-following coverage (assistant + 105k customer-service scenarios) — and blending them under a shared prompt format lets the model inherit both [Key findings §Disentangled speech naturalness and task-adherence].
Under 5,000 hours of directed data is sufficient to specialize Moshi’s pretrained weights into a prompt-controlled persona model while retaining its broad conversational competence [Key findings §Efficient specialization from pretrained foundations].
The model generalizes to scenarios outside its training distribution (e.g. an astronaut handling reactor-meltdown vocabulary with appropriate stress tone) even though training data only covers assistant + service roles; the authors attribute this to Helium’s pretraining corpus surviving fine-tuning [Key findings §Emergent generalization beyond training domains].
PersonaPlex reportedly leads open and commercial systems on conversational dynamics, latency, and task adherence on FullDuplexBench + ServiceDuplexBench [Evaluation §Conversation Dynamics / Latency / Task Adherence].

Method

PersonaPlex inherits Moshi’s dual-stream architecture: a Mimi speech encoder (ConvNet + Transformer) tokenizes 24 kHz audio, temporal and depth transformers process the joint conversation stream, and a Mimi decoder (Transformer + ConvNet) generates output audio. The underlying LM is Kyutai’s Helium. The model adds two new conditioning channels — a voice prompt (audio embedding capturing speaker characteristics, style, prosody) and a text prompt (natural-language description of role/background/context) — that are processed jointly to define the agent’s persona.

Training data is a single-stage blend across three sources, all using the shared (voice prompt, text prompt, dialogue) format: (i) 7,303 real Fisher English conversations (1,217 h), back-annotated by GPT-OSS-120B with varying-specificity persona prompts to balance generalization vs. instruction-following; (ii) 39,322 synthetic assistant dialogues (410 h) with a single fixed text prompt and varied user/agent voices and content; (iii) 105,410 synthetic customer-service dialogues (1,840 h) with prompts encoding organization, role name, and scenario-specific information (pricing, hours, rules). Synthetic transcripts come from Qwen3-32B and GPT-OSS-120B; synthetic audio is rendered with Chatterbox TTS using TortoiseTTS voices. Total directed data is under 5,000 hours, starting from Moshi’s pretrained weights.

Results

The page reports headline wins on three axes (numerical values are shown only as bar charts on the live page, not in the fetched text):

Conversation Dynamics (FullDuplexBench): higher is better — PersonaPlex top of the chart.
Latency: lower is better — PersonaPlex top of the chart.
Task Adherence (ServiceDuplexBench, the authors’ new customer-service extension judged by GPT-4o): higher is better — PersonaPlex top of the chart.

ServiceDuplexBench is introduced specifically because FullDuplexBench only evaluates response content on generic QA, so service-role task adherence had no public benchmark. Baselines are unspecified in the fetched text (“other open-source and commercial systems”) but qualitative examples include comparison-style demos against Moshi-class systems.

Release: code under MIT, weights under NVIDIA Open Model License, base Moshi model CC-BY-4.0 from Kyutai. ServiceDuplexBench is announced as “to be released.”

Why it’s interesting

PersonaPlex is the first artifact filed on this wiki to cross into audio/speech, so its main value here is as a new datapoint for Synthetic Training Data: it adds a third regime where the audio itself is synthetic (Chatterbox TTS over LLM-written dialogues) and is explicitly blended with real data because the synthetic audio lacks behavioral richness — which is a different failure mode than the verification-gate concerns dominating the other entries on that concept page. The “back-annotate real data with LLM-generated persona prompts” trick is a clean, cheap recipe that turns undirected corpora into instruction-tuned ones; it should generalize beyond speech (e.g. unscripted video, dialogue text, multi-agent traces). The architectural story is incremental over Moshi — same backbone, two new prompt channels — and the more interesting empirical claim is that <5k h of directed mixing is enough to layer task control onto a pretrained full-duplex model without breaking turn-taking dynamics. The blog leaves the actual numbers behind bar charts, so the eventual paper (cited as Roy et al. 2026) will be needed to evaluate the leaderboard claims.