Why Voice AI Is More Than Latency: AssemblyAI on Building Production Speech Systems

Luka on why reliability matters more than any single metric—and why he'd rather know where models fail than be told they're state-of-the-art in every dimension.

Hermes welcomes Luka Chkhetiani and asks for background on AssemblyAI's speech AI models for transcription, diarization, and conversational intelligence.

Luka traces his path from deep learning into voice AI—drawn by verbal communication's information density and speech's wow factor for non-technical users.

Luka recounts Dylan's founding vision to replace CD-ROM speech kits with APIs any developer could use without becoming a voice AI specialist.

Luka argues great models and great accessibility must both be true—one without the other does not scale, a lesson reinforced at AssemblyAI's hackathon.

Early customers obsessed over latency, but 150 ms emission latency still felt slow—turn-to-turn responsiveness mattered more, and both sides were still learning what production speech needs.

Luka says robustness—not marginal accuracy gains—is the biggest challenge: building systems that perform deterministically across messy real-world conditions.

Focusing on a single latency metric hides subsystem trade-offs; rigorous multi-dimensional evaluation matters more than optimizing one number in isolation.

Luka describes starting with a solid solution to a clear problem, then compressing it—large models cannot always be commercialized as-is.

Among early customer builds, putting AssemblyAI models inside prestigious humanoid robots felt like science fiction coming alive.

High-volume call centers and outreach qualify leads fast—over 80% of bank calls can automate once conversational paths are mapped.

Developers optimize popular online metrics like time-to-first-token while ignoring turn latency, accuracy, and interruption rates working together.

Luka defaults to skepticism—ask which dataset, how normalization was done, and whether the organization has a track record of rigorous evaluation.

Beyond pricing and latency, the moat is product quality and how much downstream value companies create from your speech stack.

Luka previews Universal-3 Pro's next generation—a speech-LM focused on more intelligent decision-making for voice agents and conversational intelligence.

Feeding the agent side of the conversation halves key error metrics in voice-agent situations where traditional ASR only hears one party.

The model uses adaptive conversation history without over-biasing the past, and speaker revision reruns diarization clustering post-call for async-level accuracy.

AssemblyAI's speech LM accepts natural-language instructions to bias transcription—a focused trade-off over generic tool-calling for task-specific ASR value.

Cloud dominates today for certifications and ops simplicity, but on-device and on-prem demand is rising as edge hardware improves.

Closest-server routing helps reduce hop latency for SMBs; large enterprises often choose on-prem for HIPAA patient-doctor conversations over latency alone.

Real-time streaming is complex to maintain at scale; the new sync endpoint returns sub-300 ms results on up to two minutes of audio for teams with their own VAD.

Duplex speech models are young and not industry-proven yet, but Luka expects an inflection point as reliability catches up—similar to generative image quality years ago.

AssemblyAI's north star is minimizing developer annoyance—making speech AI integrable in under five minutes for technical and non-technical users alike.

Instead of dozens of VAD and threshold knobs, presets bundle the painful tuning work while still exposing advanced controls for experts.

Hermes and Luka agree beginners need fewer choices to ship; experts can peel back the knobs once they know what to optimize.

AssemblyAI integrates research and engineering into customer calls so one technical fix can resolve five different-sounding complaints.

With agents in production, economic value is directly measurable through customer interactions—and the needle is moving fast across the industry.

Enterprise wins are elegant but hard to feel; B2C products used by hundreds of millions powered by AssemblyAI transcription hit differently.

Static voice agents that never change tone when a caller gets angry remain deeply frustrating—humans adapt; most agents do not.

Luka bets cascading wins for the next six to nine months, but specialized multimodal models will eventually cover larger blast radii.

A few hundred milliseconds of reasoning feels natural in conversation—AssemblyAI is pursuing real-time reasoning to improve turn detection and messy environments like drive-thrus.