Retrieval and multi-stage responses

This episode is brought to you by the MLflow team. Check out more information at MLflow.org.

What does it actually take to build voice AI at a billion-interaction scale? This episode features an ex-Amazon voice AI engineer who built customer support systems handling 2 billion+ interactions — now working on next-gen voice agent platforms. Anurag digs deep into the real engineering tradeoffs, design patterns, and use cases that separate production-grade voice agents from demos.

Voice Agent Use Cases // MLOps Podcast #374 with Anurag Beniwal, Member of the Technical Staff at ElevenLabs

🎙️ Topics covered:

🔹 Cascaded vs. speech-to-speech — Why cascaded systems still win in production, and how to make them feel natural without sacrificing control

🔹 Latency masking — Foreground/background model architecture and how to buy yourself time while deep retrieval runs

🔹 Constellation of models — Using Haiku for tool calling, fine-tuned smaller models for response generation, and why "one model for everything" breaks at scale

🔹 Turn-taking & ASR challenges — Why voice is harder than chat: accents, noise, silence detection, and domain-specific fine-tuning

🔹 Level 1 vs Level 2 customer support — Why today's agents max out at Level 1 and what it takes to capture Level 2 expert judgment

🔹 Inbound vs. outbound sales agents — Where voice agents are already winning, and why inbound lead qualification beats cold outbound

🔹 Booking, reservations & concierge — The clearest near-term wins for voice agents across hospitality, home services, and SMBs

🔹 Continual learning from natural language feedback — How to build agents that improve from real operator feedback without ML expertise

🔹 Conversational TTS — Why passing full conversation history to your TTS model changes everything for tone consistency

🔹 User tiers for voice platforms — Non-technical business owners vs. developers vs. enterprise: why one interface doesn't fit all.

If you're building production voice agents, evaluating voice AI vendors, or scaling AI-first customer support — this episode is packed with hard-won lessons from someone who's done it at Amazon scale.

🔗 Links & Resources:

MLOps.community: https://mlops.communityGoogle Scholar: https://scholar.google.com/citations?user=g_QB5WgAAAAJ&hl=en&o

Amazon science page: https://www.amazon.science/author/anurag-beniwal

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

MLOps GPU Guide: https://go.mlops.community/gpuguide

⏱️ Timestamps

[00:00] Cascaded Systems Control Challenge

[05:35] Voice vs Chat Complexity

[14:16] MLflow's open source platform

[15:03] AI Model Constellations

[23:00] Model Constellations Use Cases

[31:40] Voice vs Text Context

[33:54] Voice as Thought Capture

[42:11] Cascaded vs Speech-to-Speech Debate

[50:02] Wrap up