Connect: Voice AI Sales Role-Play

Problem

Franchise offices had been paying for outside role-play tools or role-playing with each other, and the feedback depended on who happened to be coaching. Representatives needed consistent practice for conversations of up to fifteen minutes, from a client calling in to a referral source at a hospital, including regulated topics such as VA benefits, and office owners needed to see where to focus their staff's coaching. The platform is built to raise conversion by giving every representative realistic practice. For a practice call to feel real, the buyer is built to answer within about a second, while scoring the finished call carefully takes a reasoning model about a minute.

Approach

I built it alone, as its product owner, designer, architect and engineer; two sales-and-coaching partners supplied the coaching expertise, data colleagues built its dashboards, and the corporate coaches introduced it to franchise owners. The AI plays the buyer and a separate grader scores the finished call, so grading never slows the conversation. Connect first ran on LiveKit, which carried the live audio while the realtime voice models matured. I ran its franchise pilots through the summer of 2026, sitting with representatives as they used it. In August 2026 I moved its live calls onto the product's own WebRTC connection to Azure OpenAI's realtime calling, so the company owns its scaling and pays no voice-platform fees, and in September, once the new path had proved itself on production traffic, I retired LiveKit and its servers. That September I also moved Connect inside the franchise portal, behind the portal's single sign-on. The rollout across the franchise network goes out in waves, pilot offices first and then office by office as each signs its agreement, and I lead its onboarding, adoption and support.

Architecture

Connect is two halves joined only by the finished transcript. In the live half, the representative's browser holds a WebRTC audio connection straight to Azure OpenAI's GPT realtime model, which plays the buyer in a voice chosen per persona. The product's own server sets every call up: it admits the call only when there is capacity for it, mints a short-lived session carrying the persona's instructions, and relays the WebRTC handshake, so the browser never holds a key or sees the instructions. Server-side software, the observer, joins each call over a separate channel, captures the transcript that gets graded, and re-asserts the persona if a browser tampers with the session. When the call ends, the transcript lands in Blob Storage, and a Service Bus queue feeds the grading worker on Azure Functions. A reasoning model judges the call under a strict output schema against the rubric for its call type (out of 90 for inside sales, 100 for outside), code recomputes every total, and about two minutes after the call ends, the representative gets an email with the score and the transcript lines behind it. No call audio is kept: speech becomes text as the representative talks, and only the text is graded.

More Results

• By the time the pilots closed in August 2026, nearly every major suggestion had shipped, was being fixed before release or had a documented product decision
• Before the pilots, tested the grader against the official sales rubrics, found half the outside-sales criteria missing (ten of the twenty), and rebuilt the scoring once the rubric owners approved the corrections; 101 tests now pin the scoring arithmetic
• Practice is honest by construction: the buyer's instructions never reach the browser, and the transcript that gets graded is captured on the server
• In the first design, fixed a buyer persona's voice drifting partway through a session by having the model write the words and a speech engine speak them in one fixed voice per persona; the August move retired that split, and the realtime model now speaks in a voice chosen per persona

Lessons Learned

A live conversation and careful scoring need different time budgets, so I split the live buyer from the grader, and that split outlived the rebuild of the voice path. A voice platform was the right start while the realtime models matured; once they had, the voice platform was the only layer left between the browser and the model, so connecting the browser directly to Azure OpenAI over WebRTC removed that vendor and its fees. A grader is a product claim too, so I audited it against the rubric it follows, then made every change to it prove itself before it merges.