First written in March 2026 and updated in October 2026 to cover the franchise pilots, the August 2026 move of the live voice path to a direct WebRTC connection, the relaunch inside the franchise portal and the rollout across the franchise network.
A practice call has to answer within about a second, and grading a call carefully takes a reasoning model about a minute, so one pipeline can't do both. I built two halves that never call each other.
The Problem
At Right at Home, a 480+ location in-home senior-care franchise network, franchise offices had been paying for outside role-play tools or role-playing with each other, and the feedback depended on who happened to be coaching. I built Connect alone from its first commit in January 2026: a voice AI sales-coaching platform where franchise sales representatives rehearse calls against AI buyer personas. Representatives practice conversations of up to fifteen minutes, from a client calling in to a referral source at a hospital, including regulated topics such as VA benefits, and office owners see where to focus their staff's coaching.
The AI plays the buyer and nothing else. A separate grader scores the finished call against the company's sales rubrics, and the coaching lives in that report.
Two Halves, Joined Only by the Transcript
The two-halves design was decided before the build began, and it is still the shape of the product.
The live half (the buyer) holds the conversation. The representative's browser has a WebRTC audio connection straight to Azure OpenAI's GPT realtime model, which plays the buyer. The product's own server sets every call up: it mints a short-lived session that carries the persona's instructions and relays the WebRTC handshake, so the browser never holds a key and never sees the instructions. Server-side software, the observer, joins each call over a separate channel and captures the transcript that gets graded.
The grading half (the grader) runs after the call. When a call ends, its transcript lands in Blob Storage, and that event starts grading. A Service Bus queue with duplicate detection and a dead-letter queue feeds a grading worker on Azure Functions, which scores the transcript with a reasoning model under Structured Outputs, against the rubric for the call type. Code then recomputes every total instead of trusting the model's arithmetic. About two minutes after hanging up, the representative gets an email with the score, strengths, gaps, one suggested focus and the transcript lines behind it.
Storage is the only link between the two halves, so grading can never slow a reply.
flowchart TD
accTitle: Connect's two halves
accDescr: The representative's browser holds a WebRTC audio connection to Azure OpenAI's realtime model, which plays the buyer. Connect's server mints a short-lived session carrying the persona's instructions and relays the handshake, and a server-side observer captures the transcript. At hang-up the transcript lands in Blob Storage, which starts the grading half, where a Service Bus queue feeds a grader that scores it with Structured Outputs against the rubric for the call type. Code recomputes the totals, and a coaching report is emailed.
rep(["Sales representative"])
subgraph live["Live half: the buyer"]
direction TB
server["Connect server: short-lived session, handshake relayed"]
model["Azure OpenAI GPT realtime model"]
observer["Observer: captures the transcript"]
end
rep -->|"start a call"| server
rep <-->|"WebRTC audio"| model
server -->|"persona's instructions"| model
observer <-->|"side channel"| model
observer -->|"transcript at hang-up"| saved[("Blob Storage")]
subgraph grading["Grading half: after the call"]
direction TB
queue["Service Bus queue"] --> grader["Grader: Structured Outputs, rubric for the call type"] --> calc["Totals recomputed in code"]
end
saved -->|"starts grading"| grading
grading --> email["Emailed coaching report"]Dynamic Buyer Personas
Each training scenario is defined by a persona that sets the buyer's voice, personality, opening line and call direction. Call direction matters because chat models are trained to behave as helpful assistants, and a buyer who has just picked up a sales call should sound like that buyer, not like an assistant. On inside-sales calls, practice is blind: the representative does not know in advance who is calling.
Holding that identity took several reinforcing techniques rather than one, and together they held the persona in testing. They are prompt design, not a guarantee. Connect started on Azure OpenAI's GPT Realtime model in January 2026, moved to GPT Realtime 1.5 that February and moved its live calls to GPT Realtime 2.1 in August. Each time the model changed, instruction-following changed with it, and the prompt design had to be re-tested.
Keeping Practice Honest
A practice tool is only useful if its scores mean something, so honesty is built into the live half rather than asked of the people using it. The buyer's identity and instructions travel only from the server to Azure, so they never reach the browser. If a browser tries to change the session, the observer notices and re-asserts the persona from the server. The transcript that gets graded is captured on the server: a transcript posted by a browser could be forged, so the product never accepts one.
No call audio is kept. Speech becomes text as the representative talks, only the text is graded, nobody listens in, and there is no scoreboard or ranking.
Grading by Call Type
Inside sales (incoming calls) and outside sales (outbound calls) are evaluated on different rubrics, scored out of 90 and 100 respectively. Rather than forcing both into a single schema, I implemented separate prompts and Structured Output schemas per call type. The persona's call type flows through the transcript into the grader, which selects the matching rubric.
The inside-sales rubric was written for human mystery shoppers, who can check things a transcript cannot, such as whether the phone was answered within three rings. Rather than let the AI grader guess at that and return a complete-looking score that was partly invented, I carried those points on what a transcript does show, the greeting and the use of the caller's name, and kept the unmeasurable item in the report as a coaching note. Every scored point traces to something a person can read.
Hard calls are scored against their own goal. A hard persona is written to decline, so the grader judges closing skill against that persona's expected outcome rather than whether the AI agreed, and it never curves the total. Goal-relative scoring went live only after a calibration showed eight of nine personas within model noise.
Auditing the Grader, Then Gating It
A grader is a product claim: it says a score reflects the rubric. So I tested it against the official sales rubrics, and it did not hold up. Half the outside-sales criteria were missing (ten of the twenty), and inside-sales empathy was weighted at double what the rubric allowed. I took the findings to the rubric owners and rebuilt the scoring once they approved the corrections.
One audit was not enough. Later I made every change to the grader prove itself before it merges: a required check replays fourteen reference transcripts, three times each, against the live grader in staging and blocks the merge if scores drift beyond tolerance, and nine red-team cases, among them prompt injection and score manipulation, must hold. The same check runs weekly to catch the model drifting on its own, and 101 tests pin the scoring arithmetic that the grader's results are recomputed with.
Voice Consistency in the First Design
The first design had a problem that showed up in development testing and live demos: the realtime model kept changing a buyer persona's voice partway through a session, so the person a representative was pitching stopped sounding like the same person. That breaks the exercise.
Telemetry showed the configured voice stayed fixed throughout each session. The change was coming from inside the model's own audio synthesis, downstream of every setting the application controlled, so no prompt-side fix could touch it.
The fix was to split the pipeline. The realtime model emitted text only, and a dedicated speech engine (Azure AI Speech) spoke it in one fixed voice per persona. A per-session check flagged any session that used more than one voice as a failure. Before merge, all six persona sessions checked in staging held one voice, and in a seven-session smoke test the speech engine began speaking in about a quarter of a second on average and never took longer than about a third of a second.
That text-and-speech split did not survive the move described next. Today the realtime model speaks in its own built-in voice, chosen per persona when the session is minted, and a dashboard flags any session where the voice setting the service reports changed. It checks that setting, not the sound of the audio; in the first design, that setting stayed fixed while the voice changed.
Moving Off the Voice Platform
Connect first ran on LiveKit: LiveKit Cloud carried the live audio to an agent worker that drove Azure OpenAI's realtime model. That was the right start while the realtime voice models matured. It let me build what mattered to the business (the personas, the grader and the pilots) without first solving real-time media transport.
By mid-July 2026, three things had changed. Azure offered browser-direct WebRTC for its realtime model as a generally available service, GPT Realtime 2.1 had reached Azure OpenAI, and a network launch would have required a higher voice-platform tier. The remaining dependency was the media transport, not the model, so I built the product's own WebRTC connection to Azure OpenAI's realtime calling. Connecting to OpenAI directly was rejected: it would have taken the product outside the Azure compliance boundary it already runs inside.
The new path shipped dark behind a flag and was compared with the old one in staging, with both scored by the same grader. Live calls moved onto it in August 2026, with LiveKit kept as a hot fallback, and after the new path proved itself on production traffic I retired LiveKit and its servers in September. The company now owns its scaling and pays no voice-platform fees. Speed was never the reason: the expected gain was well under a fifth of a second, because model inference dominates a turn.
Inside the Franchise Portal
In September 2026 I moved Connect inside the franchise portal, behind the portal's single sign-on. The new front door stood beside the old one, so the old address kept working until a redirect moved people across.
From Pilots to the Network
The platform went to franchise offices as pilots through the summer of 2026. I sat with representatives as they used it and owned the post-session survey. By the time the pilots closed in August, nearly every major pilot suggestion had shipped, was being fixed before release or had a documented product decision. The pilots grew the buyer personas from six to nine, added difficulty levels, added the full transcript to each coaching report, and sent a copy to the representative's supervisor.
The rollout across the franchise network then began in waves, pilot offices first, with the company's corporate coaches using it before they introduced it to franchise owners, and it continues office by office. Each office is enabled as it signs its agreement, and I lead onboarding, adoption and support. Two sales-and-coaching partners supplied the coaching expertise, and data colleagues built the dashboards.
What I Learned
Keep grading out of the live call. Separating live interaction from careful analysis is the design that satisfies both constraints at once, and it outlived a rebuild of the whole voice path.
Structured Outputs fix the shape of every report, and code does the arithmetic. Without strict JSON Schema enforcement, the grader would occasionally invent new rubric categories. Strict mode fixed the shape of every score report, and recomputing the totals in code means the model's arithmetic never reaches a representative.
Audit your own grader, then gate it. An automated score inherits the trust of the rubric it claims to follow, so it has to be checked against that rubric, and every later change has to prove it did not move the scores.
Start on a platform, and know what you would own later. A voice platform let me ship while the models matured. Once the voice platform was the only layer left between the browser and the model, connecting the browser directly to Azure OpenAI removed a vendor and its fees without leaving the compliance boundary.
Keeping an AI in character takes more than one technique. It took several layered techniques to hold the persona in testing, and every model change meant re-testing them.
Pilot feedback has to ship. The pilots mattered because their feedback turned into product changes before the wider rollout.