Relatability Over Perfection in Voice AI with Rime's Lily Clifford

Preview clips featuring key insights from the episode about relatability over perfection in voice AI, including why 'bored' voices convert better than polished ones.

Hermes welcomes Lily Clifford from Rime AI and introduces the episode topic.

Lily shares the origin story of Rime Labs, from Stanford PhD research in linguistics to building a recording studio in 2020. Discussion covers the importance of speech data, underrepresented groups in datasets, and the early vision for collecting conversational data.

The moment of discovery when Rime trained their first text-to-speech model on conversational data and realized it sounded like a friend, not a voice actor - a breakthrough in natural-sounding speech synthesis.

How Lily's background in computational linguistics influences Rime's product choices, including detailed annotation of ums, ahs, false starts, and other natural speech patterns that make voices more relatable.

The concept of relatability in voice AI - going beyond realistic to relatable. Discussion on pronunciation challenges, predictability, controllability, and building models that provide confidence scores and allow immediate fixes without retraining.

Exploring the trade-offs between expressiveness and controllability in voice models, multimodal approaches using LLMs trained on text, and the future of phonetic representations combined with orthographic text.

Surprising findings from real-world deployments: the best-performing voices are often the 'bored' ones that sound genuinely human, not overly polished. Discussion on customer preferences and the importance of relatability over perfection.

Rime's approach to making voice AI easier to ship, including tools that allow pronunciation fixes without knowing the International Phonetic Alphabet. The challenge of building complex voice applications with multiple LLMs and guardrails.

Overwhelming demand for multilingual support, especially in India. Challenges include data collection, linguistic awareness, code-switching (like Hinglish), and building models for specific dialects like Castilian Spanish and Saudi Arabic.

How Rime balances R&D ambition with revenue goals. In a fast-growing market, R&D often means meeting customer demand, with partners helping identify where demand exists.

Discussion on speech-to-speech models - overhyped in timeline but underestimated in long-term impact. The analogy to databases in the 70s-80s: there will be many specialized models for different use cases, not one monolithic solution.

What it will take for truly conversational voice agents: the ability to interrupt users naturally, move beyond turn-based interactions, and maintain emotional context throughout conversations.

The biggest technical bottleneck: lack of state management in emotional character. How the bot responds should depend on what happened three turns ago, maintaining emotional frame of reference throughout the conversation.

Future of multimodal models: training LLMs to predict both text tokens and speech simultaneously, creating reasoning models that are drop-in replacements for current LLMs while also generating speech.

Shoutouts to researchers pushing the envelope: Google Tacotron team (RJ Skerry-Ryan, Julian Salazar), Andrew at OpenAI, and Shivam Mehta at Netflix.

The challenge of uptalk in American English - sentences with periods delivered with rising intonation, and questions with falling intonation. The tension between what real people do and what voice experiences need for control.

Final thoughts on the future of voice interfaces and closing remarks.