The trade-off between hand-checked data quality and AI-driven automation and why the right line between the two moves with scale.
The setup
For the past few months I’ve been building the AI Solutions Hub, a catalogue that documents which artificial intelligence insurers actually deploy in the Swiss market. The platform’s promise isn’t completeness at any cost; it’s reliability. Every entry rests on a source, and every source is graded for how much it can be trusted.
The architecture behind it is quickly told: sources (insurer newsrooms, trade press, targeted web searches, manual tips) flow into an automation layer (scheduled crawler jobs and an interactive agent), pass through several AI quality gates, land in a Postgres database, and only then go public. Sounds linear. But the interesting part, and the real lesson isn’t in the architecture. It’s in a single seam: where does the machine stop and the human begin?
The one sentence that governs everything
A catalogue is worth only as much as the reliability of its weakest entry. Two naive positions follow from that sentence, and both are wrong:
- “A human has to check every entry.” – Highest quality, but doesn’t scale.
- “Just let the AI do it.” – Scales, but produces neatly formatted garbage.
The whole craft lives in between. And here’s what I had underestimated: the right dividing line isn’t fixed. It shifts with volume.
Why handwork doesn’t scale
With five entries I read every source carefully, check every link, weigh every phrasing. That’s craft, and it’s good. With fifty it becomes a chore. With five hundred I lose the overview, not through carelessness, but because human attention is a finite resource. The crawler surfaces dozens of signals a week; nobody reads forty sources at 11 p.m. with the same care as at 9 in the morning.
The subtler failure isn’t speed, it’s coverage: a duplicate resubmitted under a different name will almost inevitably slip past me in a list of two hundred entries. The human is the bottleneck and, worse, an unreliable bottleneck whose error rate climbs with fatigue.
Why pure automation isn’t enough either
The counter-test is sobering. Language models are confident and, at times, completely wrong. A concrete example from the project: I had executive-summary PDFs a chatbot had produced cleanly formatted, complete with citations. On inspection, roughly half the cited URLs were simply invented. Plausible, formatted, non-existent. Vendor marketing reads the same way as a real deployment: “AI-powered claims handling” except the text names no insurer that actually uses it. A model left to its own devices publishes exactly that: claims that look like evidence.
The bridge: frameworks and thresholds
Here’s the actual methodological insight. To automate a judgement, you first have to make it explicit.
The Admiralty Code, a two-axis rating system from the intelligence world, in use for the better part of a century and today codified in NATO doctrine does exactly that. It breaks the diffuse question “how much do I trust this?” into two separate, nameable axes: the reliability of the source (A–F) and the credibility of the specific claim (1–6). An insurer press release picked up by independent trade press is a B1. A slick but unattributed landing page is an E4. Once the judgement is structured this way, a machine can propose it — it no longer has to sense it.
The second lever is thresholds. A gut feeling (“this is probably relevant”) can’t be automated; a number can. On the platform, the relevance filter discards anything below a score of 0.75, and that value isn’t guessed, it’s derived from the data: every entry I ever approved scored ≥ 0.8; the junk I clicked away averaged 0.70. The threshold turns my past decisions into something operational.
And third, the feedback loop: every rejection reason I note when discarding an entry flows back into the filter prompts as a negative example. The system gets a little stricter with each human decision. That’s how you raise the automation ceiling without lowering the quality floor.
When the human is non-negotiable
The rule of thumb that crystallised: automate the routine, escalate the exceptional. What stays with the human:
- Defining the criteria themselves. That an entry only counts when a named insurer actively deploys the solution is a human stipulation, no model derives that on its own.
- The top confidence tier. No algorithm awards itself a “verified.” A pipeline may propose a grade; the highest tier is a human-only decision.
- The irreversible step. Publishing makes an entry public. I don’t hand that off without a safety net even at high AI confidence it only runs behind a multi-stage control gate.
- Genuine edge cases with conflicting sources and the cases where the whole framing is wrong.
The gate doesn’t decide the hard cases. It sorts: the clear ones pass through, the doubtful ones land on my desk.
The counterintuitive part: when the model has the better overview
And now the twist I didn’t see coming. “Human = quality, AI = speed” is too simple. For certain tasks, past a certain volume, the human is no longer the better guarantor of quality.
The best example from the project: a duplicate that would have slipped through. Two entries described the same insurer deployment, one called “Claims Voicebot,” the other “AI voicebot for claims reports.” A fuzzy text match rated the name similarity at 0.33, far too low to trigger. And a human skimming a list of two hundred entries wouldn’t have connected the two either, the names are too different. What caught it was a semantic AI judgement that checked the new entry against every existing entry for the same insurer and concluded: this is the same solution.
The point: at volume, consistency and total recall beat human attention. The model holds all five hundred entries in view at once, every time, without fatigue, without an off day. The human’s strength is judgement on the ambiguous single case; the model’s strength is the overview across the mass.
The division of labour
| Better done by a human | Better done by an LLM |
|---|---|
| Defining the criteria (“what qualifies at all?”) | Reading forty sources with steady care — at 11 p.m. as at 9 a.m. |
| The irreversible sign-off (publishing publicly) | Extracting structured fields and filling them in two languages |
| Weighing conflicting edge cases | Applying the same threshold identically — no drift, no off day |
| Spotting when the whole framing is wrong | Catching near-duplicates across languages (Claims ≈ claims reports) |
| Awarding the top “verified” tier | Cross-reading every claim against the live source — on every entry |
The learning nugget
The reflex to frame this as “human or machine” leads you astray. The real work is designing the seam between them. Three moves have proven their worth:
- Make the quality judgement explicit – with a framework like the Admiralty Code that turns “I trust this” into two nameable axes.
- Translate it into a threshold a machine can pass or fail.
- Reserve the human for the rare – definitions, edge cases, the irreversible.
And the punchline that ties it together: the right dividing line shifts with volume. What needs a human at ten entries belongs automated at five hundred – and the human moves up a level: from checking individual rows to designing the system that checks them. Data quality doesn’t scale by checking more. It scales by engineering the checking itself well.