How to Catch AI failures

Faraz previews digital humans that interact with your product and cut manual voice-agent testing from hours to minutes.

Hermes welcomes Faraz Siddiqi, co-founder and CTO of Blue Jay, and asks how the company evolved from an early meetup idea to today.

Faraz recounts walking Third Ave to sell restaurant voice agents and realizing every agent had to handle 150–200 menu items—a problem that pointed toward something bigger.

Manually testing menu combinations meant 10–12 hours per agent, which led to version zero of Blue Jay—an automated caller—and the pivot into Y Combinator.

Hermes asks how Blue Jay builds trust in testing; Faraz explains code-coverage-style goals, digital humans with accents and background noise, and parallel simulations that cut testing to five minutes.

Digital humans have phone numbers, emotional inflection, and can get angry—simulating realistic production scenarios before real customers ever call.

Parallel load tests surface autoscaling gaps and latency spikes, with Slack and email alerts when average conversation latency crosses thresholds like 4,000 milliseconds.

Blue Jay tracks utterance and punctuation latency across every simulation—and a 1.5-second pause makes users realize they're talking to AI and shorten their intent.

Tool call adherence verifies your agent invokes the right backend tools with the right inputs—one of the few deterministic ways to test non-deterministic voice agents.

Heartbeat simulations, production observability, and dashboards catch degradation early—including a transcription bug that booked Dr. Fan instead of Dr. Pham.

Blue Jay's rewritten docs include copy-paste AI integration prompts for every API endpoint so Cursor or Claude Code can wire the platform in one shot.

Faraz argues docs should serve three audiences, voice will become the default product interface, and LLMs make stream-of-consciousness input workable.

Customers run heartbeat tests as often as every 10 minutes, block CI/CD deploys on simulation pass rates, and use replay to turn production failures into regression cases.

At 10 calls a week, Faraz advises listening to every conversation yourself first—automate only once you've learned what you need from each flow.

Replay isolates the customer utterances that caused a failure and recreates them as a digital human, growing your test suite from 10 cases to 25 overnight.

Sales teams run hundreds of simulations before handoffs and generate PDF test reports as third-party proof that an agent is Blue Jay approved.

Faraz shares a load-test latency scare on their own infrastructure, teases the Blue Jay World rebrand and a skydiving stunt in Blue Jay suits, and Hermes closes the show.