Clinical document intelligence with privacy built in

The client builds software for healthcare providers, and a great deal of the value their customers needed was trapped in paper: scanned clinical documents in inconsistent layouts, full of critical structured information that no system could use because it was locked in images of forms. Extracting that data by hand was slow, expensive and error-prone, and it was a bottleneck on everything downstream.

Two constraints made this far harder than a generic document-extraction project. First, the data was clinical and personal, which meant it could never leave the client's controlled environment — sending it to a third-party API was simply not an option. Second, the output had to be accurate enough to feed clinical workflows, where a wrong extraction is not a minor inconvenience. They needed accuracy and privacy at the same time, with no compromise on either.

The challenge

The combination of requirements ruled out the easy paths and demanded a custom approach built around the client's constraints rather than the other way around.

• Documents were scanned, inconsistent in layout, and varied widely in quality — the long tail of real-world paperwork rather than clean templates.
• Patient data could not leave the client's environment under any circumstances, which excluded hosted extraction APIs entirely.
• Extraction had to be accurate enough for clinical downstream use, where errors carry real consequences.
• The system had to be auditable and maintainable by the client's own team after we handed it over — not a black box only we could operate.

Our approach

Because off-the-shelf APIs were off the table and the domain was specialised, this was a genuine custom-model problem — one of the cases where fine-tuning earns its keep. We fine-tuned an open document-understanding model on the client's own labelled corpus, deployed it entirely within the client's cloud, and wrapped it in the data isolation, human review and audit machinery that a clinical workload requires.

A model fine-tuned for the domain

Generic models stumbled on the inconsistent layouts and clinical vocabulary, so we fine-tuned an open document-understanding model on a carefully labelled set of the client's own documents. The hardest and most valuable part of the work was the data: assembling a consistent, representative labelled corpus that taught the model the document types it would actually face. The training run was the easy part; the dataset was where the accuracy came from.

Privacy as the architecture, not a setting

The entire pipeline runs inside the client's own virtual private cloud. Documents are ingested, processed, extracted and stored without ever leaving that boundary, and every extraction is written to an audit log that lives in the same environment. Data isolation was not a configuration we turned on at the end; it was the constraint the whole architecture was designed around, which is the only way to satisfy a serious clinical-privacy requirement.

Human review where confidence is low

Rather than pretend the model would be perfect, we built a confidence check into the pipeline. High-confidence extractions flow straight to the structured store; low-confidence ones are routed to a human review queue, and the verified results both correct the immediate case and provide data to improve the model over time. This kept accuracy high where it mattered without forcing a human to check everything, and it gave the clinical team a controllable quality lever.

The results

Manual data entry was largely eliminated for the supported document types, freeing the client's customers from a slow and costly bottleneck. The system reached an extraction F1 of 0.94 on the held-out, expert-labelled gold set — scored against ground-truth labels created independently before the model was trained, making it a genuine held-out measure rather than self-assessment — while running at 99.9% uptime (measured as monthly availability of the serving endpoint, averaged over the first quarter of production) entirely inside the client's VPC, with a full audit trail for every extraction. Patient data never left the VPC, which let the system clear the privacy bar that had blocked every previous attempt.

We handed over a system the client's own team can operate and maintain: documented, observable, and continuously evaluated against the gold set, with the human-review loop giving them a direct lever on quality. It was not a black box delivered and abandoned; it was infrastructure the client now owns.

What made it work

This engagement is a clear example of when custom model work is the right answer — a specialised domain, a hard privacy constraint, and an accuracy bar that generic tools could not meet — and of how to do it responsibly. The wins came from investing in the labelled data, designing privacy in from the first diagram, keeping a human in the loop where confidence was low, and measuring relentlessly against a clinical gold set. Custom AI done with that discipline is what turns a privacy constraint from a blocker into a system a healthcare organisation can actually deploy.