Adam BlansettSenior Full-Stack & AI Engineer
AI Engineering
11 min read
Adam Blansett

Architecting Real-Time Voice AI Agents: Sub-500ms Latency, Twilio Media Streams, and Speech Interruption

A senior engineer's guide to architecting real-time voice AI agents on live phone calls: bidirectional WebSocket media streams, sub-500ms speech interruption (barge-in), conversation state machines, and fail-safe human escalation.

Voice AITwilioWebSocketsReal-Time AITelephonyAI AgentsSystem Architecture

Building an AI voice agent that can converse on a real phone call is one of the most demanding challenges in software engineering. In text chat, a two-second response delay feels acceptable. On a live telephone connection, an 800ms silence creates awkward interruptions, talk-overs, and immediate caller confusion.

In this guide, we break down the production architecture behind real-time telephony agents—based on our open-source AI Customer Service & Contact Center Platform—covering bidirectional WebSocket media streams, sub-500ms voice barge-in, explicit conversation state machines, and fail-safe human agent escalation.

The Latency Anatomy of a Voice AI Turn#

To achieve a human-like conversational cadence, total end-to-end turnaround latency must remain strictly below 600ms. Every millisecond must be accounted for across five sequential phases:

  • Voice Activity Detection (VAD) (50–100ms): Detecting that the user has stopped speaking without triggering premature cutoffs on natural mid-sentence pauses.
  • Streaming Speech-to-Text (STT) (100–150ms): Transcribing streaming 8kHz mulaw audio chunks over WebSockets using low-latency transcription models (Deepgram Nova-2, Whisper Live, or Google Cloud Speech).
  • LLM Time-to-First-Token (TTFT) (100–150ms): Generating the initial response tokens using optimized models (such as Gemini 1.5 Flash, GPT-4o mini, or self-hosted vLLM Qwen 2.5) with streaming chunk delivery.
  • Streaming Text-to-Speech (TTS) (80–120ms): Synthesizing audio buffer chunks (Cartesia, ElevenLabs, or Sherpa-ONNX) as soon as the first clause is emitted, rather than waiting for complete sentences.
  • Telephony RTP Packet Transport (30–60ms): Shipping raw 160-byte payload packets over UDP/WebSockets to the carrier or Twilio gateway.
Pipelining is essential: do not wait for the LLM to complete its response before starting audio synthesis. As soon as a punctuation mark (comma, period) is received in the streaming token generator, pipe that phrase into the TTS stream immediately.

Why Synchronous HTTP Webhooks Fail for Voice#

Legacy telephony architectures rely on synchronous HTTP request-response cycles: Twilio executes a `<Gather>` directive, waits for speech to terminate, sends an HTTP POST webhook to your backend, and waits for a static TwiML response. This creates an unacceptable 2,500ms to 4,000ms latency floor.

Modern telephony architectures replace this with bidirectional WebSocket Media Streams. When a call arrives, your gateway returns an initial TwiML stream connector:

inbound-voice.twiml
<Response>
  <Connect>
    <Stream url="wss://voice.example.com/media/stream">
      <Parameter name="tenantId" value="acme-corp" />
    </Stream>
  </Connect>
</Response>

Once the WebSocket handshake is established, raw 8000Hz mulaw audio frames flow continuously in both directions. The server receives user speech as raw base64 payloads and pushes generated audio directly to the caller's ear in real time.

Solving the Speech Interruption (Barge-In) Problem#

Nothing destroys the illusion of an intelligent assistant faster than an agent that continues reading a long paragraph while the human is trying to say 'Stop' or 'Actually, that is not what I meant'.

Achieving sub-500ms voice barge-in requires three coordinated actions across the WebSocket session:

  • Continuous Client-Side or Edge VAD: Monitoring incoming audio energy levels while assistant audio is playing.
  • Immediate Buffer Flush: The moment user speech energy exceeds background noise thresholds, send an immediate 'clear' frame to the telephony gateway to instantly purge all buffered audio queued on the phone carrier's SIP switch.
  • Aborting In-Flight Generation: Issue an immediate AbortController signal to the active LLM token stream and TTS worker, resetting the audio sequencer to zero.
media-stream-session.ts
function handleUserSpeechInterruption(session: MediaStreamSession) {
  if (session.state === "speaking") {
    // 1. Immediately clear audio buffered on Twilio/SIP gateway
    session.ws.send(JSON.stringify({ event: "clear", streamSid: session.streamSid }));
    
    // 2. Abort in-flight LLM generation and TTS worker
    session.abortController.abort();
    session.abortController = new AbortController();
    
    // 3. Transition conversation state
    session.state = "listening";
    logger.info("Speech interruption triggered: audio buffer purged in 42ms");
  }
}

Explicit Conversation State Machines & Hallucination Defense#

Conversational voice agents cannot operate as open-ended, unconstrained chat prompts. If an agent hallucinates an incorrect pricing quote or invents a non-existent business policy on a recorded phone call, the business faces real legal and commercial liability.

In our AI Customer Service architecture, conversations are governed by an explicit finite state machine with strict transition guards: active, waiting_agent, transferred, interrupted, closed, and failed.

Furthermore, the LLM is coupled with strict BM25 keyword and semantic chunk retrieval against approved business manuals. When a caller asks a question that cannot be verified against approved context with high confidence, the system is strictly prohibited from guessing. Instead, it offers clarification or initiates immediate human escalation.

Hours-Aware Human Escalation & Fallback Callbacks#

The true measure of a production voice system is how gracefully it fails. When a caller demands to speak with a human or when the issue requires manual authority, the agent must not trap the user in an automated loop.

Our telephony pipeline evaluates business operating hours in real time:

  • During Business Hours: Initiates a warm transfer to the configured office or support hunt group via E.164 dial directives.
  • Agent Busy / No Answer: If the transferred call rings out for more than 20 seconds without an answer, the system intercepts the call back, apologizes, and offers to capture an urgent callback request.
  • After Hours: Informs the caller of operating hours and records caller name, phone number, and issue summary, instantly dispatching an HMAC-signed webhook to create a high-priority ticket in HubSpot, Zendesk, or Salesforce.

The Economics of Self-Hosted Telephony vs. Proprietary SaaS#

Off-the-shelf voice bot platforms charge between $0.15 and $0.45 per minute. For a contact center handling 30,000 minutes of call volume per month, proprietary SaaS fees reach $4,500 to $13,500 per month.

By self-hosting a Docker-based architecture using open-source infrastructure, raw telephony via Twilio or Telnyx costs roughly $0.0085/minute, speech-to-text costs $0.0043/minute, and low-latency LLM inference (e.g. Gemini 1.5 Flash or DeepSeek V3) costs less than $0.002/minute. Total running costs drop to under $0.025 per minute—delivering a 75% to 85% net operational savings.

Explore the complete open-source codebase, architecture diagrams, and telephony integration guides in the official repository: github.com/adamblansett/ai-customer-service.
Applied Architecture

Production Case Studies & Capabilities

Explore how these engineering patterns are deployed in production systems and available through client engagements.

Case Study

AI Customer Service Platform

A production-grade, Docker-first AI contact center platform for inbound telephony (Twilio/FreeSWITCH) and web chat. Features real-time bidirectional audio streaming, sub-500ms barge-in, Drizzle ORM on PostgreSQL 16, and BYOK AES-256-GCM encryption.

Read Case Study
Case Study

Autonomous AI Virtual Assistant

A privacy-first autonomous virtual assistant designed to perform executive assistant workflows. Runs local open-weight LLMs via Ollama, integrates Google Calendar v3, enforces AES-256-GCM credential vaults, and provides Telegram mobile approval controls.

Read Case Study
Related Engineering Service

AI Engineering & LLM Integration

Many AI initiatives fail in production due to prompt fragility, high token latency, hallucination risks, and lack of structured evaluation. I bridge the gap between experimental LLM prompts and deterministic software engineering.

Related Engineering Service

Software Architecture & System Design

Fast-moving teams frequently accrue hidden architectural liabilities: tangled domain logic, unmaintainable monoliths, or over-engineered microservices that paralyze development.

Frequently Asked Questions

What is voice barge-in in conversational AI systems?
Barge-in (or speech interruption) is the capability of a voice AI agent to immediately stop talking the moment the caller begins speaking. In a real-time telephony stream, this requires sub-500ms voice activity detection (VAD) and sending an immediate media cancellation frame to clear the audio buffer on the telephony provider's SIP gateway.
How do you achieve sub-500ms total latency on phone calls?
Total latency requires optimizing five discrete phases: streaming Voice Activity Detection (VAD) (~50-100ms), streaming Speech-to-Text (STT) transcription (~100-150ms), LLM Time-to-First-Token (TTFT) using low-latency models (~100-150ms), streaming Text-to-Speech (TTS) chunk synthesis (~80-120ms), and raw telephony RTP packet transport. By pipelining token generation directly into streaming TTS over WebSockets, the caller hears audio before the full sentence is finished generating.
Why are bidirectional WebSockets preferred over synchronous HTTP webhooks for voice AI?
Synchronous HTTP webhooks (such as traditional Twilio TwiML <Gather> loops) require waiting for the caller to finish speaking, sending an HTTP POST, waiting for LLM generation and full audio synthesis, and returning a static audio URL. This introduces 2,000ms to 4,000ms delays. Bidirectional WebSockets stream raw audio packets (mulaw/PCM) in real time with continuous streaming transcription and immediate interruption support.
How does human agent escalation work when the voice AI is uncertain?
When caller queries fall outside the semantic bounds of the verified knowledge base or when callers express frustration, the conversation state machine transitions from 'active' to 'waiting_agent'. The system evaluates business operating hours and either initiates a warm SIP/E.164 telephony transfer to an on-duty human representative or automatically captures caller intent, phone number, and preferred callback window into a CRM ticket.
What are the benefits of self-hosting a voice AI stack versus proprietary SaaS platforms?
Proprietary voice AI platforms charge heavy markups (often $0.15 to $0.40 per minute) on top of raw telephony costs and lock call recordings in closed third-party clouds. A self-hosted Docker-first stack connects directly to your own Twilio or FreeSWITCH SIP trunks, uses Bring-Your-Own-Key (BYOK) AES-256-GCM encryption, keeps customer audio private, and reduces per-minute operating costs by 70% to 85%.
Share this technical guide

Need Help Solving This in Your Application?

I partner with founders, executives, and engineering leaders to design architectures, resolve production bottlenecks, and deliver resilient software.

Related Technical Articles