A Call Center AI Voice Agent should easily meet Payer AI Governance Requirements

A health plan's voice agent answers a provider's call about claim status or confirms eligibility for a member. So the AI Voice Agent looks up backend systems, conveys what the plan has already set up decided, and ends the call. The AI Voice Agent never makes a decision.  

So when it goes to the Payer governance committee, it gets reviewed against a framework that is built to look for use-cases where AI algorithms are replacing human judgement and decision making – for example auto adjudicate and denial claims in bulk. And hence this is a very important distinction. The call center is where most payers run AI in production first. Routine high volume calls just convey information about decisions already made by humans, and the business case is obvious. So when a member services leader brings a voice agent to the governance committee, it is really important to highlight to the committee that a lot of their concerns that arise from a framework built to evaluate auto claims adjudication and utilization management algorithms are not valid any more.

What a Call Center leader also needs to highlight is that the Payer needs a different framework. These are rules that apply to a voice agent to make sure it provides grounded verifiable information.  

Why the Usual Framework doesn’t fit

Most payer AI governance rules were built for one thing: AI that makes or influences a coverage decision. A voice agent that just reports status was never the one who these rules were written for.

CMS set the tone in a February 2024 FAQ memo on the CMS-4201-F final rule. Medicare Advantage plans may use algorithms to assist coverage determinations, but an algorithm that decides coverage from a larger data set instead of the individual patient’s history would not comply, according to McDermott.

Colorado’s HB 26-1139, effective from January 1, 2027, says AI output alone cannot support a medical-necessity denial or delay without independent approval by a qualified health care professional. Arizona’s HB 2175, effective from July 1, 2026, requires a medical director to review medical-necessity denials individually.

The NAIC Model Bulletin, adopted on December 4, 2023, works the same way. It asks insurers to keep a written AIS Program for AI systems that make or support decisions related to regulated insurance practices. As of April 1, 2026, 25 states and D.C. had adopted it.

A voice agent that tells a provider whether the plan adjudicated a claim does none of that. It reports a determination. It doesn't make one. So start with the four requirements that do apply. Call center performance rules, like the CMS call center measures, are a separate question.

The four requirements, and how a voice agent meets them

1. Disclosure: State rules differ. Utah requires generative AI to say it isn’t human when a consumer clearly asks and requires licensed professionals to disclose at the start of high-risk interactions, which include those involving health data. California requires dialing calls to say when a prerecorded message uses an artificial voice. Maine requires notice when a chatbot could pass for a person. Because triggers and wording differ, build the disclosure into the call flow for every call and log that it played. Utah’s safe harbor protects AI that discloses at the start and throughout.

2. TCPA: The FCC’s February 2024 ruling confirmed that the TCPA’s limits on artificial or prerecorded voice cover AI technologies that generate human voices. For outbound calls, you need prior express consent unless an emergency or exemption applies and artificial or prerecorded messages must identify who is calling. Plans that run outbound outreach, such as Medicaid renewal campaigns, need this control: dial from a list with a documented consent basis or exemption, record it for each number and suppress on request.

3. HIPAA: Four testable controls: verify identity before disclosing PHI, apply the minimum necessary standard to what the volunteers do, keep audit controls over access to electronic PHI and sign a business associate agreement with the vendor. A voice agent can enforce the first two in the call flow instead of relying on each agent’s adherence.

4. Vendor Oversight: The NAIC bulletin makes insurers responsible for vendor diligence and says third-party contracts should allow audit rights and require cooperation with regulatory inquiries. Ask for both. A vendor that refuses audit rights has answered the governance question.

What defensible looks like in practice

An automation rate does not answer a governance question. Evidence on one specific call does.  

• Did the agent deliver the disclosure and did the system log it?  

• Did it verify identity before PHI moved?  

• Can you retrieve what it said on one call on one date?  

• When the call escalated, what triggered the transfer and what context went with it?  

• For outbound, what consent basis covered that number? And did anyone review the call against the plan’s own quality and compliance standards?

Casey records, transcribes and scores 100% of calls for QA, CSAT and compliance. That turns the last question from an attestation into a record.

What the agent must not do

The strongest thing a plan can bring to a governance review is the clarity of what the system will never do, supported by the architecture.  

A voice agent reporting claim status and answering claims questions isn’t making a coverage determination. And an Agent that confirms benefits isn’t deciding program eligibility. It’s a different question with different ownership.  

Also we have designed Casey with strong guardrails. It never hands the dialog control to a language model to improvise. Each interaction runs through deterministic modules with set entry and exit rules and any generation stays inside guardrails for that module, with ongoing monitoring of what the agent says.

A committee can verify this directly, in the system itself, instead of taking the vendor’s word for it. That’s also why a deployment with visible boundaries clears review faster than one that claims to handle everything.

The asymmetry worth raising

According to Verint research, manual QA typically evaluates between 1% and 3% of interactions. Any quality or compliance attestation is built on that sample test on inference.

It doesn’t have to anymore. When you transcribe, score and retrieve every interaction, “we sampled and found no issues” becomes “here is the full population.” That changes the kind of evidence a committee sees.

See how Casey produces call-level evidence. Talk to us.

Common Questions

Do health plans have to tell members they're talking to AI?
Yes, in a number of states. Utah, California and Maine have enacted AI disclosure requirements, and California's AB 1609 would extend them explicitly to customer service. Because triggers and wording differ, build the disclosure into the call flow for every call and log that it played.

Can AI voice agents be HIPAA compliant?
Yes, the vendor operates under a business associate agreement, and the deployment enforces identity verification, minimum-necessary disclosure, and audit controls over PHI access. In our case, Casey is HIPAA compliant and SOC 2 Type 2 certified and can run entirely inside the plan's own VPC so PHI never leaves the plan's network.

Does the TCPA apply to AI voice calls?
Yes. FCC Declaratory Ruling 24-17, released February 8, 2024, classifies AI-generated voices as artificial or prerecorded under the TCPA, so outbound calls require prior express consent absent an exemption, and the message must identify the entity making the call.

Does the NAIC AI Model Bulletin apply to AI voice agents?
The bulletin targets AI used in underwriting, rating, claims, fraud detection and marketing, so a member-service voice agent sits outside its core subject matter. Its written AI Systems Program and third-party vendor oversight expectations still reach the deployment, including contractual audit rights.

Is an AI voice agent making a coverage determination?
No, provided it reports existing determinations rather than producing new ones. CMS guidance on algorithmic coverage decisions applies to systems that decide, not systems that report — but the architecture has to prevent drift, not just the policy.

How do health plans audit an AI voice agent?
Through call-level records: transcripts, disclosure and identity-verification logs, escalation trails, and quality and compliance scoring applied to every interaction rather than a sample. Casey scores 100% of calls, which means the audit population is the call population.

What records should a health plan keep of AI voice interactions?
At minimum the transcript or recording, disclosure and verification events, escalation reasons, and the consent basis for any outbound contact — retained under the plan's existing HIPAA documentation schedule.

Which states require AI disclosure in customer service calls?
Utah, California, Maine, Idaho and Nebraska have enacted disclosure requirements, with scope and triggers varying by state. Check current status before relying on any list, since several bills were still moving through legislatures in 2026.

How do you stop an AI voice agent from making things up?
By not letting the language model control the conversation. Casey divides interactions into modules with strict entry and exit criteria, generating responses inside guardrails within each module with continuous monitoring, so the agent can't answer a question outside the scope it was given.

‍