Interaction Models: A Scalable Approach to Human-AI Collaboration

TL;DR

Thinking Machines Lab announces a research preview of “interaction models” — multimodal models that natively handle full-duplex, real-time human–AI exchange across audio, video, and text rather than emulating it via harnesses (VAD, ASR, TTS, dialog managers). The system splits work between a time-aware interaction model that runs on 200ms micro-turns of continuous I/O, and an asynchronous background model that handles deeper reasoning and tool use, with results streamed back into the conversation. TML-Interaction-Small is a 276B-parameter MoE with 12B active, trained from scratch with encoder-free early fusion (dMel audio embeddings, hMLP image patches, flow-head audio decoder). It is the first model reported to be simultaneously strong on streaming interactivity (FD-bench V1.5 77.8 vs GPT-realtime-2.0 minimal 46.8) and on turn-based intelligence (Audio MultiChallenge APR 43.4 vs 37.6).

Key claims

Treating interactivity as part of the model itself (rather than as a harness around a turn-based LLM) lets it scale with model size and absorb capabilities like proactive interjections, simultaneous speech, and visual-cue reactions as special cases [§The collaboration bottleneck].
A 200ms micro-turn schedule that interleaves input ingestion and output generation eliminates the artificial turn boundaries that VAD-style harnesses must predict, and supports speaking-while-listening modes such as live translation and sports commentary [§The interaction model — Time-aligned micro-turns].
Encoder-free early fusion (dMel for audio, hMLP-encoded 40×40 image patches, flow-head audio decoder) co-trained from scratch with the transformer removes the standalone Whisper-like ASR encoder and TTS decoder that most omnimodal stacks rely on [§The interaction model — Encoder-free early fusion].
A two-model split — an always-on interaction model in constant exchange with the user plus an asynchronous background model for tool use, web search, and longer-horizon reasoning — gives the responsiveness of non-thinking models with the planning depth of reasoning models, with the interaction model interleaving background results into the live conversation [§System overview, §Coordination between interaction and background models].
Inference at 200ms granularity requires “streaming sessions”: the client sends each 200ms chunk as a separate request and the inference server appends to a persistent in-GPU sequence, avoiding repeated memory reallocation and metadata recomputation per-chunk; a version of this has been upstreamed to SGLang (PR #19171) [§Inference optimization].
A gather + gemv MoE kernel strategy is used in place of grouped-GEMM for the small-batch streaming regime, citing the same diagnosis as PyTorch’s mixtral-in-gpt-fast and Cursor’s warp-decode posts [§Inference optimization].
Bitwise trainer–sampler alignment is achieved via batch-invariant kernels with <5% e2e overhead, plus NVLS-based deterministic all-reduce / reduce-scatter on Blackwell and a Split-KV attention kernel that splits SMs consistently between prefill and decode (e.g. 4096-token left-aligned splits) so accumulation order matches [§Trainer-sampler alignment].
TML-Interaction-Small is the first model to simultaneously lead on streaming interactivity and turn-based intelligence — FD-bench V1.5 average 77.8 (vs GPT-realtime-2.0 minimal 46.8 / xhigh 47.8, Gemini-3.1-flash-live minimal 54.3) at 0.40s turn-taking latency, with Audio MultiChallenge APR 43.4 (vs GPT-realtime-2.0 minimal 37.6 and Qwen 3.5 OMNI-plus-realtime not listed) [Table — Streaming + Turn-based].
Existing realtime models fail two new internal benchmarks (TimeSpeak, CueSpeak) and three adapted visual-proactivity benchmarks (RepCount-A, Charades, ProactiveVideoQA) — they remain silent or answer at the wrong time when asked to count reps, narrate, or interject on visual cues [§New dimensions of interactivity].
The current TML-Interaction-Small is the floor of model scale they can serve at this latency budget; larger pretrained models exist but are reported as too slow to deploy interactively today [§Limitations and future work — Scaling model size].

Method

The interaction model processes 200ms streaming chunks of audio + video + text concurrently in both input and output directions. Inputs use minimal pre-processing — audio as dMel (Bai et al. 2024) through a light embedding layer; images as 40×40 patches encoded by an hMLP (Touvron et al. 2022); the audio decoder is a flow head (Lipman et al. 2022). All components are co-trained from scratch with the transformer. The 276B MoE / 12B active backbone is paired with an asynchronous background model that the interaction model delegates to when sustained reasoning is required; the interaction model remains responsive throughout and weaves streaming background results back into dialog at moments appropriate to what the user is currently doing.

Inference uses “streaming sessions”: the client sends each 200ms chunk as an independent request, and the server appends it to a persistent in-GPU sequence — this avoids per-chunk memory reallocation and metadata-computation overhead that existing LLM inference libraries (built for large prefills) incur. A gather + gemv MoE kernel replaces grouped-GEMM for this small-batch regime. Bitwise trainer–sampler alignment is enforced via batch-invariant kernels (extending the lab’s earlier defeating-nondeterminism-in-LLM-inference work) with NVLS-based deterministic all-reduce / reduce-scatter on Blackwell that match between Sequence Parallelism and Tensor Parallelism, and an attention kernel that picks a Split-KV strategy consistent between decode and prefill (e.g. left-aligned 4096-token splits across SMs). Safety training uses TTS-generated speech-native refusal data and an automated multi-turn red-teaming harness for long-horizon robustness.

Results

Streaming interactivity. FD-bench V1 turn-taking latency 0.40s (vs GPT-realtime-2.0 minimal 1.18s, GPT-realtime-1.5 0.59s, Gemini-3.1-flash-live minimal 0.57s). FD-bench V1.5 average 77.8 (vs GPT-realtime-2.0 minimal 46.8 and xhigh 47.8, GPT-realtime-1.5 48.3, Gemini-3.1-flash-live minimal 54.3 / high 45.5, Qwen 3.5 OMNI-plus-realtime 39.0) [Table].
Streaming with tools. FD-bench V3 Response Quality / Pass@1 = 82.8 / 68.0 with background-agent enabled (vs GPT-realtime-2.0 minimal 80.0 / 52.0, GPT-realtime-1.5 77.9 / 55.0, Gemini-3.1-flash-live minimal 68.5 / 48.0, Qwen 3.5 OMNI-plus-realtime 60.0 / 50.0) [Table].
Streaming video+audio QA. QIVD 54.0% (Qwen 3.5 OMNI-plus-realtime 59.0 leads; GPT-realtime-2.0 minimal 57.5, xhigh 58.2) [Table].
Turn-based intelligence. Audio MultiChallenge APR 43.4% (vs GPT-realtime-2.0 minimal 37.6, GPT-realtime-1.5 34.7, Gemini-3.1-flash-live minimal 26.8 / high 36.1) — but the thinking-high GPT-realtime-2.0 xhigh reaches 48.5 [Table].
Instruction following. IFEval VoiceBench accuracy 82.1% (audio) and 89.7% (text); GPT-realtime-2.0 xhigh leads at 83.2 / 95.2 [Table].
BigBench Audio. 75.7% standalone, 96.5% with background agent — matching GPT-realtime-2.0 xhigh (96.6) and Gemini-3.1-flash-live high (96.6) [Table].
Safety. Harmbench refusal rate 99.0% text (multiple baselines tie at 98–100%) [Table].
Novel proactivity. No existing commercial model performs meaningfully on TimeSpeak / CueSpeak / RepCount-A / Charades / ProactiveVideoQA in their streaming framing; baselines stay silent or give wrong answers at wrong times [§New dimensions of interactivity].

Why it’s interesting

This is the closest counterpart to PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models yet filed — both attack the harness-vs-native split for real-time speech, but PersonaPlex specializes a 7B Moshi-architecture backbone to voice + role control over 1,217h Fisher + ~2,250h synthetic dialogues, while TML-Interaction-Small is a 276B/12B-active MoE trained from scratch with vision baked in and an asynchronous background reasoner. The systems work overlaps with Better MoE model inference with warp decode (Cursor’s warp decode is cited in-text alongside PyTorch’s mixtral-in-gpt-fast as the inspiration for the gather + gemv MoE strategy) and with FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling (Blackwell-specific kernel rewrite for low-latency decode); the bitwise trainer-sampler alignment piece extends Thinking Machines’ own earlier “defeating nondeterminism” work into a deployable kernel set with NVLS deterministic collectives. As a closed-cohort comparison baseline, this is the open research counterpart to Gemini 3.1 Flash Live: realtime voice and vision agent model (Google product announcement) (Gemini 3.1 Flash Live) — the latter announces capability but discloses nothing; the TML post discloses an architecture, latency budget, training trick, and benchmark. The new TimeSpeak / CueSpeak / visual-proactivity benchmarks are the most useful artifact for the team: they pin down qualitative gaps (speak-while-listening, time-awareness, visual interjection) that current eval suites don’t measure at all, and that any future “Luma realtime” work would have to ace.