During the first week at our first production clinic, I caused a production regression in our SOAP note generator by changing the OpenRouter API key and forgetting to lock the providers in OpenRouter.

At HapiVet, we're big on open source, so we were using an open-source model through OpenRouter. With the provider lock missing, our requests could reach hosts we hadn't validated, including Open Inference. That changed the output even though we hadn't deployed a new version of the application.

This showed up in the notes our vets were generating. SOAPgen is supposed to follow the clinic's template, but it started dropping labels and parts of that structure. The requests still completed, which meant the vets received drafts they had to fix themselves.

I fixed the issue on October 2: provider pinning finished deploying at 1:55 PM PDT, followed by the validation and regeneration fix at 3:03 PM. Two days later, on October 4, I went back through the usage, latency and editing data to understand how much the regression had affected the vets. That's what the numbers below show.

The impact

The retrospective analysis estimated template-text preservation at around 97–100% over September 21–26. It dropped to about 77% on September 28 and 76% on October 1.

Daily template-text overlap, not a clinical accuracy score. October 2 includes the fix; October 3 has 12 paired completed notes.

I also wanted to see how much the vets had to edit. The first analysis attributed some of the change to a different mix of clinicians working that week. But I'd already dealt with the regression. I wanted to measure its impact, not explain it away as a staffing change.

There was a problem with the metric itself. Our template includes optional lines that vets normally remove when they don't apply. If the AI deletes those lines first, the vet has less to delete. A lower edit rate can make a worse draft look better.

Across the two reporting windows, pooled word edits went from 15.1% to 20.4%, roughly a 35% relative increase.

September 21–27: 121 paired completed notes. September 28–October 3: 108. Different case and clinician mixes; the second window includes the recovery day.

That tells me there was more editing overall. It doesn't tell me whether each change fixed a missing detail or just removed boilerplate. Those need to be measured separately.

Debugging and the fix

During debugging, I replayed the same inputs against different providers. Open Inference kept 69 of 184 template labels. DeepInfra kept 179. These were development tests on our workload, separate from the production metrics above.

Was it quantization

My hypothesis was that Open Inference was using a bad quantization. But I couldn't verify that.

The endpoint metadata captured during the investigation listed both Open Inference and DeepInfra as FP8. They still produced very different results. That doesn't rule out differences in their quantization implementations, but the tests didn't isolate quantization from other parts of their serving setup.

What I could verify was that Open Inference's endpoint performed poorly on our workload. I couldn't verify why. So I fixed the routing and added output checks rather than treating the quantization theory as proven.

I added provider restrictions to both the primary request and the hedge, the parallel request we use when generation is slow.

I also added template validation before saving the output. Review caught a gap where fallback output could bypass that check, so I added validation there too.

Then there was regeneration.

A note that had already lost its structure could keep producing the same broken layout. We were using the current note as the layout. If labels were missing there, regenerating could keep them missing. Even a check against that damaged baseline could pass.

The fix uses the original template when the current layout is damaged, while carrying the existing content forward.

In the development recovery test, the damaged notes had 80 of 184 labels. Regenerating without the fix still produced 80. With the fix, it produced 183.

Development test of template-label preservation.

After the fix

On October 3, estimated template-text preservation was back to 99.7% across 12 paired completed notes. That's a small sample, but the structural recovery is visible.

It doesn't mean every clinical detail was correct or that all editing disappeared. It means the template was being preserved again.

An unintended benefit

Testing different providers also helped me find faster ones, so locking generation to the providers that performed well improved latency too.

Median server generation time fell from 13.9 seconds during September 27–October 1 to 7.2 seconds on October 3, the first full day after the fix. That's a 48% reduction, or roughly half the wait. It was also about 25% lower than our earlier baseline, so the improvement went beyond recovering from the slowdown.

That comparison covers 186 generations during the incident and 31 on October 3. It's one post-fix day with different visits, rather than a controlled benchmark, and the timer excludes browser network and rendering time.

I caused this by changing the key without preserving the provider lock. Checking application deploys alone wouldn't have caught that change. The fix needed provider restrictions, template checks and a way to repair notes that were already affected.

Analysis window: September 21–October 3, 2026, Pacific time. Editing metrics compare 229 completed notes with their latest stored drafts, using approximate word-level changes and ignoring case, punctuation and formatting. Missing stored drafts are excluded. No clinical note text is included here.