Bellcourt Review Copilot — agentic RAG for prior authorization
A health-plan administrator was denying requests against policy versions that had already been retired. The fix is not a smarter model: it is code that picks the rule in force, a model that checks each criterion with a quote, and a person who decides.
Client
FDE Academy individual hackathon case — a fictional US third-party administrator (Bellcourt) with synthetic data: 120 QA-audited cases, 30 open cases, 13 of them fax images, and a knowledge base of policies, memos and plan amendments
Role
Solo. Diagnosis, case document, build, evidence and the client presentation.
Timeframe
October 2026 — two-day build, shortlisted and presented to an evaluator
Stack
Python · Agentic RAG · Gemini 2.5 Flash via OpenRouter · JSON vector index · Vercel Functions · Supabase
Demo sign-ins for each role are listed on the sign-in screen. Everything in it is fictional.
01 — Situation
34% of the year's denials cited a policy version that was already retired, and 91% of those were overturned on appeal. QA accuracy sat at 56.7% against a 95% contract standard.
Leadership blamed staffing. The data said otherwise: 66 of 69 late decisions had arrived incomplete, and faxes waited 17 hours before anyone keyed them in.
02 — Constraints
- !The tool may never deny, delay or modify a request — a person decides, and only a physician can deny.
- !Runs beside the existing system, read-only. Protected health information minimised, every action audited.
- !Two of the fax images carried planted instructions. They had to be treated as data, never as commands.
03 — Stakeholders
Nurse and physician reviewers
The right policy version, the criteria, and the evidence — on one screen.
Intake coordinators
Missing fields flagged at receipt, and a queue ordered by deadline.
Provider offices
Knowing what to send, and the status, without phoning.
The evaluator
Numbers that trace to the data pack, and controls that can be proven live.
04 — Architecture
- 01Code, not the model, selects the governing rules: plan document and amendments, the state addendum, and the policy version in force on the date of service. Retired versions and memos are excluded with the reason shown.
- 02The model plans what to read, checks each criterion against the record with an exact quote, and a verifier re-reads every approval sceptically.
- 03Six server-enforced roles, a provider portal with a deterministic coverage check, decision letters generated from the human decision, and a governance page where each of eleven controls is a button that runs the real code path.
05 — What shipped
- ✓Fax, PDF, JSON or form intake with the clock started at receipt.
- ✓Review screen with recommendation, criteria pass/fail, quoted evidence, and one-click source sections.
- ✓Role-gated decisions, PDF letters by email, audit log, status desk, and an evidence page with the confusion matrix.
06 — Outcomes
117 / 120
audited cases right (97.5%), against 68 by the original reviewers
0
wrongful approvals, and denials are impossible by design
29 / 30
open cases matched the answer key, 13 of them read from fax images
41
automated tests, plus 23 of 24 adversarial inputs handled
07 — Retro: what I'd do differently
Putting rule selection in code rather than the prompt was the decision that made the accuracy number possible, and made it explainable.
The governance page earned more trust in the interview than any accuracy figure: evaluators could press a button and watch the server refuse.
Questions about this engagement?
Ask the copilot →