How KPMG Got a Healthcare AI Demo Built in 12 Weeks
We built the entire AI layer behind Natalie, a medical avatar for Apple Vision Pro
KPMG needed proof that real-time conversational AI could handle hospital intake, with a result credible enough to show hospitals and stakeholders. Working alongside FS Studio, we delivered a demo-ready prototype in 12 weeks that holds a natural conversation and responds in under 5 seconds, live.
Impact at a glance
Within 3 min
Simulated intake completed, with automatic recap and specialist referral
<5 sec
Response latency after the participant stopped speaking
12 weeks
From kickoff to a working, event-ready prototype
End-to-end
Voice, reasoning, and facial animation synced on Apple Vision Pro
About the client
KPMG is a global organization of independent member firms and one of the Big Four, providing Audit, Tax, and Advisory services in 138 countries and territories through a network of more than 276,000 people. Its client base spans nearly every major industry, and healthcare is one of the sectors where it has built the deepest advisory presence, working directly with hospital systems, payers, and health organizations on operational, financial, and technology transformation.
Within that structure, KPMG's Healthcare Innovation Team is dedicated to evaluating what emerging technologies can realistically offer the healthcare industry, turning exploratory ideas into something concrete enough to bring to its own hospital-sector clients.
Can hospital intake forms be automated without losing the human conversation?
As part of that mandate, KPMG's Healthcare Innovation Team wanted to test what real-time conversational AI could realistically do when applied to a process that touches nearly every hospital: patient intake and triage. This wasn't an upgrade to an existing tool, it was a new product built from the ground up (0 to 1). The goal was a functional demo to bring to hospitals, one that could help validate a business hypothesis: that this kind of technology could meaningfully reduce wait times and operating costs.
The end-user problem
When a patient arrives at a hospital, before being seen they go through an intake/triage process, a series of questions that determines the severity of their case and where to route them. Today that process depends on in-person staff: the patient waits to be assessed, and the hospital spends time and staff running the process.
KPMG's problem
A hospital wasn't Eagerworks' or KPMG's client here, it was the patient. KPMG wanted to evaluate the potential of a conversational alternative to the intake form process. To do that meaningfully, it needed more than a smart form: a system that could hold a real conversation, make a patient feel heard, and deliver a referral they'd trust. Anything less wouldn't demonstrate the value of the approach. That set a specific bar.
The conversation had to:
Feel as natural as possible
Empathetic, polite, able to interrupt and be interrupted like a real dialogue.
Complete a full intake without losing any detail
The patient mentioned, and close within 3 minutes, recap included.
Respond in under 5 seconds
From the moment the patient stopped speaking, the threshold KPMG considered the limit of what still feels human.
Run on Apple Vision Pro specifically
This engagement was scoped to that device, not a broader multi-device experience.
Two more constraints shaped everything else: a 12-week development window with a fixed deadline tied to a health industry event, and a fixed-scope budget with no room to renegotiate scope mid-project.
Why the First-Mover advantage in real-time healthcare AI was on the line
This was a race against time and against competitors. Real-time conversational AI was, and still is, a new space, and no vendor had yet delivered an end-to-end experience good enough to feel like a real conversation.
KPMG's reasoning was that whoever got there first would be positioned as the reference point hospitals and health systems would look to when evaluating this kind of technology, and that being first would carry weight that would be harder for a later entrant to earn back.
That's why KPMG set a fixed event date and didn't move it: shipping late meant shipping after someone else had already shown it first. If the demo had failed in front of an audience because of noticeable latency, lost context, or if a conversation didn't feel human, the risk wouldn't have been just technical. It would have undercut what KPMG was there to show: that this approach was credible enough to justify exploring further with hospitals and stakeholders, not just an idea that looked good on paper.
For KPMG, what was ultimately at stake was twofold: whether this could meaningfully reduce hospital wait times for patients, and whether KPMG could point to it as proof the technology was worth developing further.
A human-sounding AI intake avatar, demo-ready in 12 weeks
1. Make the concept tangible
Turn KPMG's idea into a working experience it could show hospitals and stakeholders to test its commercial potential.
2. Run a full intake on Apple Vision Pro
Build an AI avatar that handles a complete simulated medical intake, end to end.
3. Feel like a real conversation
Empathetic, polite, and able to interrupt and be interrupted.
4. Close within 3 minutes
Complete the full interaction, including a recap and a specialist referral.
5. Be demo-ready in 12 weeks
Deliver a stable prototype in time for a health industry event.
What Eagerworks owned: The entire AI and backend layer behind KPMG's medical avatar
Eagerworks was end-to-end responsible for the entire intelligence layer of the avatar, the "brain" and "voice" of KPMG's medical avatar, Natalie. That meant owning the architecture from the ground up: researching and evaluating alternatives, selecting and integrating the underlying models, optimizing for latency, building and running the infrastructure, and validating that the whole pipeline worked end-to-end. Thus, we made sure the system could:
Understand the participant's spoken audio in real time.
Generate a context-aware response based on what was just said and the full conversation history.
Generate the audio for that response (realistic, real-time voice).
Generate the avatar's facial animations, synced to the generated audio.
FS Studio was responsible for the avatar's visual design and the frontend. The engagement worked as a partnership between two specialists: Eagerworks owning the entire backend and AI layer, FS Studio owning the visual experience and interface on Apple Vision Pro, each team with full ownership over its layer, coordinated at the integration points. This wasn't a generic team-extension role: Eagerworks had technical autonomy over how the intelligence layer worked, including the autonomy to challenge technical proposals from KPMG or FS Studio when they didn't meet the project's requirements.
Biweekly demos and clear ownership
Eagerworks shared progress with KPMG through biweekly demos. FS Studio managed the day-to-day client relationship as KPMG’s main point of contact, while Eagerworks participated directly as the AI and backend team. The demos gave KPMG visibility into progress and an opportunity to review the experience, while Eagerworks retained ownership of the underlying technical decisions.
Need a partner to think through your next product decision?
Talk to our team4 Technical decisions that kept AI response time under 5 seconds
A real-time voice pipeline, not the traditional STT → LLM → TTS chain
Most voice AI converts speech to text (STT), sends that text to an LLM, then turns the reply back into audio (TTS). That pipeline is simpler to build and debug, but every handoff adds latency, and together they push responses past several seconds. For this project, that broke the core requirement: replies had to arrive in under 5 seconds and feel fluid. We chose a real-time language model that takes audio as input and returns text directly, paired with ElevenLabs for real-time voice. Text generation and voice synthesis still happen, but removing the standalone transcription step cut a full stage of latency and made the exchange feel like a real conversation.
Processing background noise in the backend, not the frontend
Initially, background noise reduction happened on the frontend, before audio reached the backend. That approach caused two problems: in some cases the system didn't receive audio it should have processed (if someone spoke in the background, it got over-filtered), and it added extra latency at a point in the pipeline where every millisecond mattered. We recommended moving noise processing out of the frontend and into the backend, having it sent as full, unprocessed audio and handled in a single step. This simplified the pipeline to one audio-processing point instead of two, reduced total latency, and avoided losing relevant information from what the participant said.
WebSockets instead of HTTP calls between frontend and backend
Originally, communication between the frontend (FS Studio) and backend (Eagerworks) used traditional HTTP calls. HTTP is simpler to implement and debug, but each call adds connection overhead, latency that, combined with the rest of the pipeline, put the 5-second response limit at risk. We proposed migrating to WebSockets, keeping a persistent connection between frontend and backend, instead of continuing with the initial HTTP-based approach. This cut communication latency between layers, an infrastructure decision invisible to the end user, but decisive in making the experience feel real-time.
Replacing the proposed facial animation model (NVIDIA) with NeuroSync
KPMG initially proposed an NVIDIA model to generate the avatar's facial animations from the response audio. That model wasn't built to run in real time under the project's conditions, which would have introduced delay between generating the audio and animating the face, breaking the sync that makes the avatar believable. We ran extensive research and testing of alternatives and recommended NeuroSync, a model that did run in real time on the same server infrastructure as the rest of the backend. Without this decision, the avatar would have had a mismatch between voice and facial expression, exactly the kind of detail that breaks the illusion of talking to something human, the central requirement of the entire project.
From:
An unproven concept: could real-time conversational AI handle something as sensitive as hospital intake?
To:
A working prototype that answered that question live: a simulated intake conversation, completed in under 3 minutes, credible enough to bring in front of hospitals and stakeholders.
Tech Stack
- OpenAI Realtime - Audio understanding
- ElevenLabs - Text-to-speech
- FastAPI - Backend
- AWS - Infrastructure
- Python - Language
- WebSockets - Real-time communication
- NeuroSync - Facial animation
A complete AI-led patient intake in under 3 minutes
Product outcome:
The avatar sustains a full real-time conversation: it understands audio, generates a contextual response (considering the entire conversation history, not just the last message), generates the response audio, and animates the face in sync, all within a sub-5-second threshold.
The system can be interrupted at any point by the participant, just like a human conversation, and correctly picks the thread back up.
No detail mentioned during the conversation is lost, even as the session approaches the time limit.
KPMG outcome:
KPMG now has a functional, demo-ready experience that shows how this kind of intake process could realistically be automated, something concrete it can use to evaluate the idea further and pitch it to hospitals, not a system deployed or validated in hospitals.
The demo consistently closes with an automatic recap and a specialist referral, regardless of how the conversation unfolded.
User outcome:
Within the simulated experience, a participant could complete the intake conversationally rather than through a form, receive a recap, and reach a specialist recommendation within three minutes.
From unproven concept to a live healthcare AI demo
Because Natalie was delivered as a demo/prototype rather than a live deployment, KPMG doesn't have hospital-side metrics like wait-time or cost reduction to report yet.
What it does have is a working prototype that proves the concept end-to-end, and that's what carries the real business impact:
A concept turned into something KPMG could actually show:
Before this project, KPMG had an idea it wanted to explore, not something it could put in front of hospitals or stakeholders. The working prototype closed that gap, giving KPMG a tangible experience to bring to the industry event and use in conversations with stakeholders and hospitals evaluating the opportunity.
Technical viability proven under real constraints:
The prototype held a full intake conversation with response latency consistently under 5 seconds, a bar the team had to hit live, in front of an audience, on a fixed date, with no room to adjust scope if it didn't.
A functioning example of a rarely-solved architecture problem:
Running a real-time LLM and a real-time facial animation model in parallel, on shared infrastructure, without breaking the sub-5-second target, gave KPMG a tangible answer to a technical question most vendors in this space haven't solved yet.
Delivered on a fixed, non-negotiable timeline:
The prototype was ready in 12 weeks, meeting the external deadline tied to the industry event, with no slippage to work around.
Taken together, these outcomes gave KPMG something to act on: a working reference to validate the use case internally and open conversations with hospitals, rather than an idea still waiting to be tested.
In healthcare AI, latency is part of the experience
In conversational AI, generating the right response is only part of the challenge. It also needs to arrive fast enough for the interaction to feel natural. This project shows that, in healthcare, latency isn't a technical footnote; it directly shapes how natural, continuous, and credible the experience feels.
For teams working on real-time, voice-driven AI products, this is particularly important anywhere the interaction has to feel human, not just be technically correct. Running a real-time conversational language model alongside a real-time facial animation model, on the same server infrastructure, while keeping response latency under 5 seconds, is an architecture problem with almost no publicly documented precedent. The key learning: a credible conversational demo depends on how the entire system performs together, not just the quality of any individual model in isolation.
Have an AI concept you need to prove on a fixed deadline? We can help you get a working version in front of the people who matter.
LET’S TALKFrequently asked questions
How do you get an AI avatar to respond in under 5 seconds?
The traditional approach, convert speech to text, run that text through an LLM, convert the response back to speech, adds latency at every step, and those delays compound into a response that takes several seconds to arrive. To stay under 5 seconds, we used a real-time language model that takes audio as input and produces text output directly, paired with a real-time text-to-speech model (ElevenLabs) to generate the response audio. We also moved communication between frontend and backend from HTTP calls to a persistent WebSocket connection, since each HTTP call adds its own connection overhead. Every one of these choices was about removing a specific source of delay, not just picking "faster" technology in the abstract.
What's the difference between a traditional STT→LLM→TTS pipeline and a real-time conversational model?
In the traditional pipeline, audio is converted to text (STT), that text is passed to an LLM to generate a text response, and that response is converted back to audio (TTS), three separate conversions, each adding processing time. A real-time conversational model skips the standalone speech-to-text step: it takes audio as input and generates text output directly, which removes one full stage of latency from the chain. Text generation and voice synthesis still happen, this doesn't remove those steps, but cutting out the separate transcription step is what makes a response arrive fast enough to feel like a real conversation instead of a delayed exchange.
Can facial animation be synced in real time to AI-generated audio?
Yes, but not with every model. The facial animation model KPMG initially proposed (built by NVIDIA) wasn't designed to run in real time under this project's conditions, which would have created a visible gap between the generated audio and the avatar's mouth movements, breaking the illusion of a real conversation. After testing alternatives, we replaced it with NeuroSync, a model that could run in real time on the same server infrastructure as the rest of the backend, keeping voice and facial animation in sync without adding perceptible delay. Running a real-time conversational model and a real-time facial animation model in parallel, on shared infrastructure, without either one slowing the other down, is a problem most real-time AI products don't have to solve, and one with very little public precedent to build on.
