Practice before agents: how a healthcare-data engineering team rolled out headless PR review under PHI constraints

The visible fix versus the actual fix

When a US healthcare-data company moving tens of billions in clinical spend every week wanted AI inside a pipeline that thousands of facilities depend on, the request landed like most enterprise AI requests do: a mandate to build something. The team that took the work did the unusual thing. They started with practice before they touched agents.

The visible outcome, eventually, was a headless agent that reviews every pull request for one to three dollars of model spend, capped at twenty dollars a day. That number is worth pausing on, because it is the kind of figure a board can read in five seconds and an engineering organization can defend in an audit. It is not the interesting part of the story. The interesting part is what the team had to put in place for that number to mean anything.

What the surface looked like

Three problems were running underneath the request, and none of them were about model selection.

Problem one: uneven adoption. Some engineering teams used AI tools daily. Some never touched them. The same licenses were provisioned across both groups. The same access, the same documentation, the same optional training. The outcomes diverged anyway. This is a familiar pattern inside large engineering organizations: a tool is available, a small fraction of teams rebuild their workflow around it, and the rest keep doing what they were doing the week before. The gap is not technical.

Problem two: no shared approach. Around a hundred developers were working in the same monorepo. There were roughly a hundred different workflows. Nobody could tell whether anyone was following a standard, because no standard existed in any artifact the team could read. Code review practices, prompting patterns, what to delegate to a model and what to keep human, how to handle context for long-running investigations — all of it was tribal knowledge held inside individual teams and a handful of senior engineers.

Problem three: no cost or security control. LLM calls were going directly from developer workstations and CI runners to provider APIs. There were no budget caps. There was no audit trail. There was no redaction layer in front of model calls. In a healthcare environment handling protected health information, that last point is the one that would have ended the program at the first incident report. The other two would have been acceptable risks for a quarter. This one was not a risk at all. It was a guarantee of an eventual breach.

What they put in place before any agent ran

The team did not start by building the PR-review agent. They started by writing down the operating practice that had to exist for any agent to be safe inside the organization.

A budget layer went in first. Every model call, from any developer or any CI job, routed through a single proxy. The proxy enforced a per-developer daily cap and a per-repository weekly cap. The proxy also logged every call with the prompt hash, the response hash, the model identifier, and the token counts. That log became the audit trail the security team had been asking for. It also became the dataset the engineering team used to figure out what people were actually doing with the tools.

A redaction layer sat in front of every model call. Anything matching the patterns the security team had flagged — names, dates of birth, medical record numbers, free-text clinical notes — was masked before the prompt left the building. The mask was imperfect, and the team knew it. They treated it as a defense in depth, not a guarantee. The guarantee was that no developer prompt could exfiltrate PHI to a model provider by accident or by design.

A small set of standard workflows got documented and taught. Not fifty. Five. The PR-review pattern. The investigation pattern for unfamiliar legacy modules. The doc-drafting pattern for service READMEs. The bug-triage pattern for incoming issues. The migration pattern for cross-cutting refactors. Each one had a short written playbook, a recommended model, a recommended context shape, and a recommended human checkpoint. They were not mandatory. They were not enforced. They were what good engineers did because good engineers were doing them, and the rest of the organization watched.

Then, and only then, did the team build the headless PR-review agent.

Why this ordering matters

The PR-review agent is the kind of tool that fails quietly. It reads a diff. It suggests a comment. The developer either accepts or dismisses the comment. If the suggestions are bad, the cost is a few seconds of friction. If the suggestions are subtly wrong on a security-sensitive change, the cost is measured in incident reports. The reason the agent could be allowed to run unattended is that the layer underneath it was already doing the work a less mature program would have asked the agent to do.

The budget layer meant a misbehaving prompt could not blow through a monthly allocation. The redaction layer meant a curious developer could not accidentally ship PHI to a third party. The five documented workflows meant the agent was operating inside a context the organization already understood. When something went wrong — and something always goes wrong — the team could tell what had happened and why.

This is the part that most enterprise AI writeups skip. The story gets told as “we built an agent that does X.” The real story is “we built the conditions under which an agent could be trusted to do X without anyone needing to watch it.”

What enterprise teams should take from this

The pattern is portable. A regulated-industry engineering organization of any size can copy it without copying the agent.

First, write the operating practice before you write the prompt. If your organization cannot describe in a page what good AI-assisted engineering looks like for your codebase, no agent will fix that for you. The agent will at best formalize whatever the most senior engineers were already doing. At worst it will produce a thousand parallel versions of whatever the median engineer was doing on a tired Tuesday.

Second, route every model call through a proxy you own. The cost ceiling matters. The audit log matters more. You cannot govern what you cannot see, and direct-to-provider calls are invisible to the rest of the organization by default.

Third, put the redaction layer where the prompt leaves your perimeter, not where it lands at the provider. A redaction layer at the provider is a contractual promise. A redaction layer at your perimeter is something your security team can test.

Fourth, document a small number of workflows well rather than a large number poorly. Five workflows the team actually uses beats fifty workflows nobody has read. The documentation is the surface that future agents will be trained on and the surface that future hires will be evaluated against.

Fifth, measure adoption as a leading indicator. If ninety percent of developers are not using the tools three months in, the failure is not the tools. The failure is the practice layer, and the fix is more writing and more teaching, not a new model.

Closing

The PR-review agent is real and it ships a useful comment most of the time. The interesting engineering was the layer underneath it. For enterprise teams considering a similar rollout, the question is not which model to pick or which agent framework to adopt. The question is whether the operating practice underneath the rollout can survive a quiet failure at two in the morning without anyone noticing. If it can, the agent is safe to run. If it cannot, no agent is safe to run, no matter how good the suggestions look in a demo.

For teams ready to put the practice layer in place before the agent layer goes in, a strategy consultation with Otonomi is the right starting point.