How Do You Trust a Number an LLM Extracted?

Mon, 10 Aug 2026 00:00:00 +0000

A pipeline returns a perfectly-shaped JSON object. Every field present, every type valid, Pydantic green across the board, not a parse error in sight — and the number inside it is wrong. Not malformed. Wrong. The model misread a figure, or invented one, or pinned it to the wrong entity, and nothing in the system can tell, because nothing in the system was ever built to ask. The output was indistinguishable from a correct output, so it shipped as one. Somewhere downstream, a person made a decision on it. If you have shipped an extraction pipeline, this one probably looks familiar — it is the most common way a demo that dazzled becomes a number nobody can defend.

“Schema-valid” and “true” are different words, and most AI pipelines only check the first one. That is the whole problem, compressed. The honest worry underneath “we can’t trust the numbers it extracts enough to act on them” is one the model itself can never answer — because you would be asking the thing that might be wrong to certify that it isn’t, and no amount of prompting fixes that. Trust isn’t a property you prompt into a model; it’s a property you engineer downstream of it, in layers your own code owns and the model never gets to author. The model proposes. Something you build disposes. What follows are four of those layers, climbing from cheap to authoritative — each drawn from a system running in production, and each one you can adopt on its own, in the order your case actually needs.

A well-shaped number can still be wrong

Validating the shape of structured output is table stakes now, and it’s worth having. Pydantic tool-calling gives you real guarantees: the fields you asked for exist, the types match, and the whole free-text-parsing class of bug — the regex that breaks the first time the model adds a stray comma — is simply gone. Keep doing it. Every layer below assumes it.

But shape validation does exactly nothing about a hallucinated value. A number that is confidently, precisely wrong is perfectly well-shaped; it passes every schema check you have, because schema checks were never asking whether the value is true — only whether it’s the right type of thing to be wrong about. The gap between “well-shaped” and “actually correct” is where a confidently-wrong number lives, and it’s the gap most pipelines never close. Closing it is four escalating moves, cheap to expensive. You rarely need all four at once — the useful thing is knowing which one your case calls for, so here they are in order.

Stop letting one pass grade its own work

The cheapest move is to stop letting a single extraction pass certify its own output. A model asked to both find a candidate fact and judge whether it’s true will happily rate its own inventions highly — those are two jobs, and they want two stages.

does exactly that: it turns live broadcast transcripts into broadcast-quality game narratives across nine categories, and every extracted element is handed to an independent judge stage whose only task is suspicion. The independence that matters here is structural — not a second vendor’s model sitting in judgment, just a separate call with a separate prompt whose entire job is to look for reasons to reject, so nothing grades its own homework inside a single pass. The judge checks five things: every fact must trace to a specific source quote; every player name must match the official roster; every number must match confirmed final statistics; nothing may be added that the commentator never said — no invented timing, outcome, or location; and any category-specific requirement must hold. An element that fails any check is dropped, with its reason logged, before it reaches the final set.

That logging is worth more than it looks. It’s the difference between a filter and an audit trail — between “the system removed some things” and “here is exactly what it removed, and why.” The drop rate isn’t a number worth quoting, because it depends on the game — a noisy overtime broadcast sheds more than a clean blowout — and that dependence is the point: it’s a number you get to watch, not one you have to take on faith. If you build only one of these four layers, build this one.

When something else can confirm the number, let it

The second move applies whenever a value can be independently confirmed. When it can, the model is not your system of record, and it doesn’t get a vote.

Broadcast speech-to-text fails in a specific, dangerous way: it mishears numbers. It captures a “45” as “42,” or grabs a partial mid-game count and presents it as a final total. Left alone, a model faithfully extracts the wrong number — and it’s well-shaped wrong, a plausible integer in a field that expects one. The fix isn’t a cleverer prompt; it’s a policy that authoritative stats win, enforced at two points in the same pipeline. At extraction time, the prompt carries an explicit priority rule: confirmed final statistics are authoritative and always override whatever the transcript seems to say. And the same judge from the first move enforces it on the way out — its number-check rejects any figure that doesn’t match the confirmed stats, so a misheard “42” that should be “45” gets dropped rather than shipped.

The move worth carrying into your own architecture is simple to state: walk your extracted values and ask which of them have an authoritative source somewhere in your systems — a ledger, a database of record, a canonical feed. For every value that does, the model is a proposer, never the final authority. Whatever you can confirm, you confirm, and the confirmed source wins. The model only gets to be the system of record for the things nothing else can check.

Check what a value is before it becomes durable state

The third move is the easiest to defer and the most expensive to skip, and it’s the one that turns a correctness problem into a security one. Validate what the values are — not just their type — and do it hardest right before anything becomes durable state, because durable state is what gets read back and acted on later.

Two systems, one principle. In an , the model reads each user’s entire conversation history and distills durable, evolving insights that are persisted and fed back into future prompts. That feedback loop is exactly where an injected instruction would do lasting damage: the conversation text is fully user-controlled, so in a naive design a prompt injection buried in a message could write fabricated vocabulary into someone’s permanent profile and steer every future run. It can’t here, because the model’s output is allow-listed before it’s ever stored — every principle and behavioral dimension it emits is checked against a controlled catalog and dropped if it isn’t a member, and the user’s text is escaped so a message can’t close the transcript wrapper and pose as instructions. The contents are vetted, not just the shape — secure by design, not a filter bolted on after an incident.

For a fintech, the same instinct guards a database. An , and the generated SQL is parsed — every table reference extracted and rejected if it isn’t on an approved allow-list, every write, DDL, and privilege command scanned for and refused. But the application-level check is deliberately not the last line. Underneath it sits a read-only database role with no BYPASSRLS privilege, so PostgreSQL Row-Level Security is enforced in the kernel: even under total application compromise, the model physically cannot read another tenant’s rows. Five independent layers, the innermost one holding when every outer one has failed — no cross-tenant incidents to date, and the part worth trusting isn’t the clean log, it’s the reason for it: a guarantee that lives in the database kernel rather than in hope. Shape validation was never a security boundary. Contents allow-listing, backed by a control the model can’t reach past, is.

Let your code own provenance, and your people own the last call

The fourth move is a refusal: never let the model author facts about itself. Where did this extract come from — which conversation, which user, which moment in time? The model must not be the one to answer, because a misattributed fact is a wrong fact in correct clothes. In that , conversation identity and timestamps are stamped by code, never taken from model output. The model proposes content; the system owns provenance, so an extract can never be quietly reassigned to the wrong person.

The other end of provenance is a person. A — high-stakes documents where a wrong figure outlives any demo. Every finding is forced through a Pydantic contract with required fields — source, location, time_period, category, detail — and a finding missing any of them isn’t a finding. Then it passes five explicit human-in-the-loop checkpoints — category assignment, post-parse review, pre-generation context confirmation, post-generation edit, and iterative regeneration — before a report ships, and every generated section carries provenance back to a source file and page number so a reviewer can check a claim against the document in seconds instead of hunting for it. Those checkpoints aren’t an admission that the AI is weak. They’re a decision about where a human’s judgment is worth more than throughput — which, for a report an engineer signs their name under, is exactly where you want the friction.

The economics: checking is cheap next to being confidently wrong

The reflex objection is that all this sounds expensive — a judge call on every element, a parse on every query, five checkpoints in the loop. It’s worth being concrete, because most of the cost is smaller than the reflex assumes. Allow-listing a value against a controlled vocabulary is an in-memory set lookup. Parsing a query for forbidden commands and unapproved tables is nearly free. Row-Level Security is enforced by the database you’re already paying to run. The judge call is the one real added cost — a second LLM pass per element — and it’s small next to what it stands in front of: a decision that would otherwise ride on an unchecked number.

The human layer is the honest exception, and it’s worth naming rather than glossing. Five checkpoints on a civil-engineering report aren’t cheap. They’re a deliberate spend, placed exactly where an engineer is about to sign their name — and every automated layer around them exists so that expensive human attention lands only where it truly has to.

It helps to name the real cost function. A confidently wrong number that ships is expensive precisely because its cost lands later, downstream, after someone has acted on it. That reframes failing closed from timidity into arithmetic. In the map-reduce pipeline, when a batch’s error rate crosses a threshold, the synthesis stage simply doesn’t run — the system produces nothing that cycle and reprocesses next time, rather than synthesizing an insight from partial coverage and calling it complete. Nothing is a cheap, honest output; a plausible answer built on a corrupted input is the costly one. Staying correct under failure — returning nothing rather than something plausible and unfounded — is how the whole approach stays cost-bounded at scale: the automated checks are small and constant, the human check is deliberately placed, and the disaster they foreclose is the one with no ceiling.

One idea underneath all four: the model proposes, your code disposes

Step back and the four moves are one idea. Verify with an independent judge. Treat a confirmed source as the authority. Allow-list the contents before they persist. Let code own provenance, and people own the last call. In every case the model proposes and something you built — a separate judge, a code-owned gate, a human reviewer — disposes; the model never gets the final say over its own output.

And wherever that disposing layer is pure code, it has a property the model never will: you can test it. In the map-reduce pipeline, the deterministic core — the partitioning, the window and epoch merging, the tally accumulation, the generation-hash preimage — is unit-tested with no model in the loop at all. The part of the system responsible for trust is exactly the part you can prove correct on a laptop, deterministically, before it ever meets a token.

That’s the reframe worth keeping. “Can we trust the number this model extracted” isn’t really a question about the model, and it isn’t one you settle by buying a better one. It’s a question about the architecture around the model — whether someone wrapped it in layers that verify, reconcile, constrain, and attribute, or whether it hands its output straight to a decision and hopes. The first is a production system.

A test you can run before you act on any extracted number

Here’s an honest test you can apply to your own pipeline today. If it extracts numbers that feed a decision, a report, a customer-facing surface, or any state you write down and read back later — and the only thing between the model and that consequence is a schema check — then you’ve validated that the output is well-shaped and validated nothing about whether it’s true. That’s the gap worth closing before more depends on it, and most of it you can close with the first three moves above — none of which needs a better model.

This is one failure mode among several that quietly sink AI projects. It sits underneath the bigger question of why a demo that dazzles on ten documents still — the cost, coverage, and correctness-under-retry failures that share this one’s DNA. And if the security end is what keeps you up — the injection buried in user text, the output that poisons state you’ll re-read tomorrow — that’s the deeper audit, where prompt injection turns out to be a persistence problem more than a prompt one. If that gap is open in a pipeline you own, it’s a better conversation to have before the number ships than after.

AI Security | Marcin Wylot, PhD