Voice Agents

Voice agents are AI systems that converse with people in real time by voice: they understand a spoken request, decide on an action — usually by calling external tools such as a booking or CRM system — and reply in natural speech, typically within about a second of the speaker finishing their sentence^[1].

Background

Phone automation began with interactive voice response (IVR) menus that routed callers by keypad, then progressed to cloud-based voice assistants such as Siri and Alexa, which could handle narrow commands but little genuine dialogue. The generative AI generation arrived in 2024, when OpenAI opened its Realtime API for speech-to-speech models and vendors such as ElevenLabs, Vapi, Retell and Deepgram built agent frameworks on streaming speech models; GPT-4o Realtime reached general availability in August 2025, marking the shift from converting audio to text and back toward audio-native processing^[1].

How voice agents work

A typical voice agent pipeline has four stages:

Voice activity detection and turn-taking — detects when a person starts and stops speaking, handles pauses, and allows interruption (barge-in) so the user can cut the agent off mid-sentence.

Speech recognition (ASR) — streaming transcription of the user's words.

Reasoning with tools — an LLM interprets intent, calls functions (look up an account, book a slot, issue a refund) and applies guardrails and authentication checks.

Speech synthesis (TTS) — streams a spoken reply; audio-native models combine stages 2–4 into a single speech-to-speech step.

The binding constraint is latency: if the full round trip exceeds roughly a second, conversation stops feeling natural, so every stage streams and the model must answer without generating long hidden plans. Design concerns include accent and noise robustness, escalation to human agents, and call recording compliance.

Impact

Industry trackers reported that production voice-AI deployments grew roughly 340% year-over-year during 2025, with voice-AI venture funding rising about eightfold to US$2.1 billion^[2]. Mainstream uses include customer service call lines, appointment booking and reminders, collections, outbound sales and surveys, and healthcare triage; Google DeepMind has projected that real-time language translation in voice calls will reach production quality by 2029, letting businesses serve callers in languages they do not staff for^[2].

>See Also

• Speech Recognition
• Text-to-Speech
• AI Agents
• Voice Cloning
• Chatbot