AI Customer Service Platform
Open-source, self-hosted AI contact center with sub-500ms voice barge-in and multi-tenant security.
01. Problem & Context
Traditional call center SaaS platforms charge exorbitant per-minute telephony markups, lock customer voice data into closed silos, and struggle with high latency during live phone conversations. Businesses need self-hosted voice AI that answers inbound calls, identifies itself transparently, and cleanly escalates to human agents.
02. Engineering Solution
Architected an open-source, multi-tenant contact center platform combining Twilio/SIP telephony adapters, real-time bidirectional WebSocket media streaming, explicit conversation state machines, and hours-aware human escalation with callback scheduling.
03. System Architecture
The system decouples real-time audio transport from LLM inference using a MediaStreamGateway. A Next.js 14 web console manages multi-tenant configuration, knowledge bases, and live agent takeovers, while an Express API handles telephony webhooks, speech barge-in cancellation, and CRM dispatches.
Core Architectural Components
Provider-Neutral Telephony Adapter
Standardizes inbound voice handling across Twilio webhooks and custom FreeSWITCH/Asterisk SIP trunks with compliant TwiML generation.
Sub-500ms Voice Barge-In Engine
Bidirectional WebSocket audio streaming detects caller speech interruptions and immediately sends audio cancellation frames to silence assistant speech.
Multi-Tenant BYOK Credential Vault
AES-256-GCM authenticated encryption at rest ensures tenant API keys and telephony credentials remain zero-knowledge in PostgreSQL.
Hours-Aware Escalation & Callback Engine
Evaluates timezone-specific operating hours, routes to live human agents, and captures structured callback tickets when lines are busy or closed.
External CRM & Helpdesk Webhook Dispatcher
Dispatches signed HMAC-SHA256 webhook payloads to HubSpot Tickets, Zendesk, Salesforce, and Zapier workflows.
04. Notable Engineering Decisions
Sub-500ms Audio Interruption over WebSockets
Conversations feel unnatural when callers cannot interrupt. Bidirectional streaming enables instant cancellation frames rather than waiting for HTTP audio chunks to finish playing.
Drizzle ORM with PGlite Embedded Testing
Type-safe schema definitions across all 70 automated tests run against embedded in-memory WebAssembly PostgreSQL without requiring external Docker dependencies for CI.
Apache 2.0 Permissive Licensing
Enables enterprise deployment with proprietary PBXs and on-premise hardware without triggering viral copyleft restrictions.
05. Engineering Challenges & Solutions
Challenge: Handling network jitter and telephony packet drops during live audio streaming.
Engineering Solution: Implemented stateful stream sessions with chunk buffering, jitter compensation, and graceful fallback to synchronous TwiML dialogs.
Challenge: Preventing AI hallucinations from making inaccurate commitments to customers.
Engineering Solution: Engineered strict BM25 semantic knowledge retrieval with confidence thresholding, triggering automatic agent escalation when answers are not grounded in approved docs.
06. Implementation Highlights
- Inbound voice answering with Twilio Voice webhooks and FreeSWITCH SIP integration
- Real-time bidirectional WebSocket media gateway with sub-500ms speech barge-in
- Full Next.js 14 App Router administration console and live conversation monitor
- Drizzle ORM schema with 70 automated tests running on embedded PGlite
- BYOK AES-256-GCM credential vaulting and tenant boundary isolation
- Complete Docker Compose deployment stack with multi-stage unprivileged containers