Real-time voice feels broken the moment there's a pause longer than a heartbeat. At 3.1 seconds, users hang up. At 320ms, they think they're talking to a human.
This is the story of how we got from one to the other.
The Problem
We were building a conversational AI pipeline for a VoIP platform — inbound calls routed to an AI handling lead qualification, customer support, and appointment scheduling.
The stack:
- Twilio for PSTN telephony
- NestJS as the WebSocket gateway
- Ollama hosting Llama 3 on an EC2 GPU instance
- ElevenLabs for text-to-speech
Initial end-to-end latency: 3.1 seconds from end-of-speech to first audio byte. For a phone call, this is catastrophic. Users fill silence with confusion, then hang up.
Diagnosing the Bottleneck
The naive pipeline:
[Twilio] → [STT] → [LLM full response] → [TTS] → [Twilio]The key phrase: full response. We waited for the LLM to finish before sending anything to ElevenLabs. A 50-token reply at ~30 tokens/sec means ~1.7s of LLM time alone. Add STT, TTS, and network hops — 3.1s is obvious in retrospect.
Fix 1: Stream the LLM, Flush on Sentence Boundaries
Ollama supports streaming. We piped tokens as they arrived — but flushed to TTS on sentence boundaries, not individual tokens:
const stream
