Architecting Real-Time Voice AI Agents: Sub-500ms Latency, Twilio Media Streams, and Speech Interruption
A senior engineer's guide to architecting real-time voice AI agents on live phone calls: bidirectional WebSocket media streams, sub-500ms speech interruption (barge-in), conversation state machines, and fail-safe human escalation.
Building an AI voice agent that can converse on a real phone call is one of the most demanding challenges in software engineering. In text chat, a two-second response delay feels acceptable. On a live telephone connection, an 800ms silence creates awkward interruptions, talk-overs, and immediate caller confusion.
In this guide, we break down the production architecture behind real-time telephony agents—based on our open-source AI Customer Service & Contact Center Platform—covering bidirectional WebSocket media streams, sub-500ms voice barge-in, explicit conversation state machines, and fail-safe human agent escalation.
The Latency Anatomy of a Voice AI Turn#
To achieve a human-like conversational cadence, total end-to-end turnaround latency must remain strictly below 600ms. Every millisecond must be accounted for across five sequential phases:
- Voice Activity Detection (VAD) (50–100ms): Detecting that the user has stopped speaking without triggering premature cutoffs on natural mid-sentence pauses.
- Streaming Speech-to-Text (STT) (100–150ms): Transcribing streaming 8kHz mulaw audio chunks over WebSockets using low-latency transcription models (Deepgram Nova-2, Whisper Live, or Google Cloud Speech).
- LLM Time-to-First-Token (TTFT) (100–150ms): Generating the initial response tokens using optimized models (such as Gemini 1.5 Flash, GPT-4o mini, or self-hosted vLLM Qwen 2.5) with streaming chunk delivery.
- Streaming Text-to-Speech (TTS) (80–120ms): Synthesizing audio buffer chunks (Cartesia, ElevenLabs, or Sherpa-ONNX) as soon as the first clause is emitted, rather than waiting for complete sentences.
- Telephony RTP Packet Transport (30–60ms): Shipping raw 160-byte payload packets over UDP/WebSockets to the carrier or Twilio gateway.
Why Synchronous HTTP Webhooks Fail for Voice#
Legacy telephony architectures rely on synchronous HTTP request-response cycles: Twilio executes a `<Gather>` directive, waits for speech to terminate, sends an HTTP POST webhook to your backend, and waits for a static TwiML response. This creates an unacceptable 2,500ms to 4,000ms latency floor.
Modern telephony architectures replace this with bidirectional WebSocket Media Streams. When a call arrives, your gateway returns an initial TwiML stream connector:
<Response>
<Connect>
<Stream url="wss://voice.example.com/media/stream">
<Parameter name="tenantId" value="acme-corp" />
</Stream>
</Connect>
</Response>Once the WebSocket handshake is established, raw 8000Hz mulaw audio frames flow continuously in both directions. The server receives user speech as raw base64 payloads and pushes generated audio directly to the caller's ear in real time.
Solving the Speech Interruption (Barge-In) Problem#
Nothing destroys the illusion of an intelligent assistant faster than an agent that continues reading a long paragraph while the human is trying to say 'Stop' or 'Actually, that is not what I meant'.
Achieving sub-500ms voice barge-in requires three coordinated actions across the WebSocket session:
- Continuous Client-Side or Edge VAD: Monitoring incoming audio energy levels while assistant audio is playing.
- Immediate Buffer Flush: The moment user speech energy exceeds background noise thresholds, send an immediate 'clear' frame to the telephony gateway to instantly purge all buffered audio queued on the phone carrier's SIP switch.
- Aborting In-Flight Generation: Issue an immediate AbortController signal to the active LLM token stream and TTS worker, resetting the audio sequencer to zero.
function handleUserSpeechInterruption(session: MediaStreamSession) {
if (session.state === "speaking") {
// 1. Immediately clear audio buffered on Twilio/SIP gateway
session.ws.send(JSON.stringify({ event: "clear", streamSid: session.streamSid }));
// 2. Abort in-flight LLM generation and TTS worker
session.abortController.abort();
session.abortController = new AbortController();
// 3. Transition conversation state
session.state = "listening";
logger.info("Speech interruption triggered: audio buffer purged in 42ms");
}
}Explicit Conversation State Machines & Hallucination Defense#
Conversational voice agents cannot operate as open-ended, unconstrained chat prompts. If an agent hallucinates an incorrect pricing quote or invents a non-existent business policy on a recorded phone call, the business faces real legal and commercial liability.
In our AI Customer Service architecture, conversations are governed by an explicit finite state machine with strict transition guards: active, waiting_agent, transferred, interrupted, closed, and failed.
Furthermore, the LLM is coupled with strict BM25 keyword and semantic chunk retrieval against approved business manuals. When a caller asks a question that cannot be verified against approved context with high confidence, the system is strictly prohibited from guessing. Instead, it offers clarification or initiates immediate human escalation.
Hours-Aware Human Escalation & Fallback Callbacks#
The true measure of a production voice system is how gracefully it fails. When a caller demands to speak with a human or when the issue requires manual authority, the agent must not trap the user in an automated loop.
Our telephony pipeline evaluates business operating hours in real time:
- During Business Hours: Initiates a warm transfer to the configured office or support hunt group via E.164 dial directives.
- Agent Busy / No Answer: If the transferred call rings out for more than 20 seconds without an answer, the system intercepts the call back, apologizes, and offers to capture an urgent callback request.
- After Hours: Informs the caller of operating hours and records caller name, phone number, and issue summary, instantly dispatching an HMAC-signed webhook to create a high-priority ticket in HubSpot, Zendesk, or Salesforce.
The Economics of Self-Hosted Telephony vs. Proprietary SaaS#
Off-the-shelf voice bot platforms charge between $0.15 and $0.45 per minute. For a contact center handling 30,000 minutes of call volume per month, proprietary SaaS fees reach $4,500 to $13,500 per month.
By self-hosting a Docker-based architecture using open-source infrastructure, raw telephony via Twilio or Telnyx costs roughly $0.0085/minute, speech-to-text costs $0.0043/minute, and low-latency LLM inference (e.g. Gemini 1.5 Flash or DeepSeek V3) costs less than $0.002/minute. Total running costs drop to under $0.025 per minute—delivering a 75% to 85% net operational savings.
Production Case Studies & Capabilities
Explore how these engineering patterns are deployed in production systems and available through client engagements.
AI Customer Service Platform
A production-grade, Docker-first AI contact center platform for inbound telephony (Twilio/FreeSWITCH) and web chat. Features real-time bidirectional audio streaming, sub-500ms barge-in, Drizzle ORM on PostgreSQL 16, and BYOK AES-256-GCM encryption.
Autonomous AI Virtual Assistant
A privacy-first autonomous virtual assistant designed to perform executive assistant workflows. Runs local open-weight LLMs via Ollama, integrates Google Calendar v3, enforces AES-256-GCM credential vaults, and provides Telegram mobile approval controls.
AI Engineering & LLM Integration
Many AI initiatives fail in production due to prompt fragility, high token latency, hallucination risks, and lack of structured evaluation. I bridge the gap between experimental LLM prompts and deterministic software engineering.
Software Architecture & System Design
Fast-moving teams frequently accrue hidden architectural liabilities: tangled domain logic, unmaintainable monoliths, or over-engineered microservices that paralyze development.
Frequently Asked Questions
What is voice barge-in in conversational AI systems?
How do you achieve sub-500ms total latency on phone calls?
Why are bidirectional WebSockets preferred over synchronous HTTP webhooks for voice AI?
How does human agent escalation work when the voice AI is uncertain?
What are the benefits of self-hosting a voice AI stack versus proprietary SaaS platforms?
Related Technical Articles
Hardening Production RAG Systems: Context Poisoning, Vector Boundaries, and Enterprise Security
Defend enterprise Retrieval-Augmented Generation (RAG) pipelines against context poisoning, indirect prompt injection, and multi-tenant vector data leaks.
Evaluating Structured LLM Outputs and Detecting Regressions with LLM Eval Toolkit
A developer guide to testing structured LLM application outputs against JSON schemas, enforcing field types, and catching prompt regressions with LLM Eval Toolkit.