Most frontier models read a document beautifully. Ask one to transcribe a private equity capital call notice and it gets the fund, the currency, the amounts, the dates, almost always right. The word “almost” is the whole problem. A probabilistic reader gives you a number that is usually correct and never certified, and in a back office that reconciles other people’s money, “usually” is not a state you can sign off on. So you add a human in the loop, and in a back office that human stays. The point is not to remove the reviewer but to make the review faster and better aimed, concentrated on the documents where a human eye actually changes the outcome. The question this study asks is narrower and more useful than “are the models good”: given that they are good but not certain, how much of that human review can a pipeline of deterministic checks remove, so that what remains is targeted rather than exhaustive? And what is left that no check can see at all?
I built a benchmark to measure it, on documents chosen so the checks have something real to bite on, and I ran nine vision-language models across roughly four thousand extractions. The short answer: hand-written, zero-ground-truth consistency checks plus a hardcoded reference database close identifier errors completely, catch about three quarters of all transcription faults, and almost never cry wolf; but there is a hard floor of errors that are internally coherent and invisible to any amount of arithmetic. That floor, not the catch rate, is the interesting result, and the most useful picture in the study is not a rate at all but a map: field by field, capture tracks exactly how much redundancy holds each field, from one hundred percent where a full equation applies down to zero where nothing does.
The design decision that shapes everything: transcription only
The pipeline asks the model to do one thing: transcribe what is printed. Every derivation lives in code. No currency conversion by the model, no summing line items, no consolidation across documents. This is not a convenience. It moves entire classes of error out of the probabilistic component and into ordinary, unit-tested code. In an earlier version of this work where the model was asked to convert and consolidate, one cheap model wrote a portfolio total one hundred euros short in ten runs out of twelve, silently, with an empty anomalies array. Move the conversion into code and that failure mode cannot occur, because the model never touches the arithmetic. It is not that the error is eliminated in some absolute sense, the conversion code has to be correct and it is tested, but it is removed from the space of things a probabilistic reader can get wrong. What remains for the model is transcription, and what remains for the guards is checking that the transcription is self-consistent and consistent with a reference.
The documents, and why they are what they are
The corpus is fifty synthetic documents from eight fictional PE funds over four quarters: capital calls, distribution notices, quarterly capital account statements. They are modeled on real conventions (the ILPA Capital Call and Distribution template, the ILPA quarterly reporting roll-forward, ISO checksums for identifiers) and deliberately span a gradient of redundancy: some funds print the full ILPA reconciliation (commitment, cumulative called, unfunded, all cross-footing), some print a total and a breakdown, some send a one-page letter with none of that. That gradient is the point. It lets me measure how much the guards can do as a function of how much structure the document itself carries, rather than pretending every document is richly checkable.
The oracle is emitted by the same code that renders the documents, so it cannot drift from what is printed. Errors are not injected: they are whatever the models actually produce, labelled automatically against the oracle and sorted into a fault taxonomy in code. All seven classes are natural; two of them (pure digit slips, order-of-magnitude errors) occur too rarely in this corpus to measure at volume, so their rates are reported as directional only, marked as such wherever they appear.
What the guards catch, class by class
The honest way to report this is per class, because a single aggregate number would be dominated by one class (wrong-picks) and would hide that the classes are wildly different in prevalence and in how catchable they are. Prevalences below are specific to this corpus and, especially for identifier errors, to the scan-degradation recipe I chose; they do not generalize. What generalizes is the shape: which kinds of fault a deterministic stack closes and which it cannot.
The stack is cumulative: schema checks (P0), then within-document invariants (P1), then cross-document consistency (P2), then checks against the reference database (P3). Residual = faults that survive the full stack. Counts below are the raw per-class fault totals, counted as field-level fault instances rather than deduplicated events, so one root misread that breaks several fields of a family is counted once per field; the global figure in the next paragraph additionally includes the pure digit slips the table omits, which is why it is not exactly a weighted average of the rows. The table lists six of the seven classes: pure digit slips are too rare in this corpus (18 raw fault instances) to warrant a row, and order-of-magnitude errors are shown but should be read as directional.
| Fault class | Faults | Survive full stack | Caught | What closes it |
|---|---|---|---|---|
| Identifier (IBAN/LEI/ref) | 340 | 0 | 100.0% | every identifier is checksummed (LEI mod-97), matched against master data (account, LEI, investor ref), checked against the fund’s documented reference convention, or required present |
| Order-of-magnitude (rare/directional) | 96 | 15 | 84.4% | cross-document NAV and roll-forward chains plus commitment-vs-DB catch the “in thousands” co-scaling |
| Wrong-pick (semantic) | 1134 | 280 | 75.3% | the printed-but-wrong amount usually breaks an equation; the ones that do not are the blind spot |
| Omission | 257 | 99 | 61.5% | completeness against the format’s mandatory fields, and the expected-document calendar |
| Format (currency/date/sign) | 60 | 30 | 50.0% | currency- and units-vs-DB close some; a swapped date order partly survives |
| Hallucination | 168 | 118 | 29.8% | P1 to P3 catch a fabrication that breaks a sum or contradicts the reference; the ones that fit every check slip through |
Across all non-anecdotal natural faults from all nine models, the full stack catches 73.5% cumulatively, with a document-clustered bootstrap interval of [67.0%, 79.7%]. I lead with the per-class table rather than that single number on purpose: the aggregate is more than half wrong-picks, so a lone “73%” mostly measures how well one equation catches one class, and it averages a fully closed class (identifiers) with a barely-touched one (hallucinations, 30%). Read the classes, not the mean.
Two things stand out. Identifier errors, which explode on degraded scans, are entirely closed, and closed by the most classical tools imaginable: a checksum, a lookup against known-good master data, and the fund’s own documented reference conventions. This is the anti-fraud story of a real back office, and it is the strongest quantitative result here; it also leans hardest on the registry, so it comes with a caveat I keep in Limits rather than a footnote: one hundred percent measures what full master-data coverage buys, on a registry authored alongside the corpus, not what a typical production registry knows. Order-of-magnitude errors, the classic “the statement is in thousands and the model expanded it,” break the within-document sum only if they hit one line, but hit every line and the sum still balances; what catches them is comparing across documents and against the database, exactly the check a within-document invariant cannot make.
Capture is a map, not a mean
The single most useful artifact the harness produces is the capture rate of amount misreads broken down by field, because it makes the mechanism visible. Fields held by a complete equation close entirely: the transaction ledger, the commitment, the call-to-call cumulatives, the distributed cumulative and the paid-in cumulative all sit at one hundred percent, more than 250 faults among them, zero escapes. Fields held by one partial equation land in the middle: the NAV endpoints at ninety-seven percent (the residual hides at chain edges), contributions and unrealized change near sixty-five, because a misread that co-moves with another term can still balance. And fields held by nothing sit at zero, however many guards the stack has: the printed IRR, which nothing on the page can recompute, and the bare total on a one-page minimal letter, where called capital drops to six percent from eighty-eight on formats that print a breakdown to sum against.
This map is also how the stack got its final shape, and I think that loop is the transferable method here. An earlier version of the stack left the distributed cumulative at zero out of seventy-eight misreads caught, for a reason the map made obvious: the only check that ever touched that field was a presence check, no equation constrained its value. Writing the one missing equation, the distribution twin of the NAV chain, closed all seventy-eight at zero false alarms. The same pass showed which equation must NOT be written: the symmetric paid-in step looks just as natural, but the fund’s own accounting identities make it false (paid-in absorbs capital actions that the period’s printed contributions do not), and it fires on perfectly correct documents. Capture is not a property of the stack, it is a property of the field, and the map tells you, before any model runs, exactly where adding one equation pays and where no equation can exist.
And then the floor. What survives is not a whole fault class but a coherent subset of two, the wrong-picks and hallucinations that pass every check (280 and 118 of the 549 survivors respectively), and it is worth being precise here because it is easy to overclaim. Three in ten hallucinations are caught (50 of 168), and caught for a concrete reason: a fabricated number that also breaks a sum, contradicts a cross-document chain, or fails to match the reference database trips a guard like any other fault. What slips through sits where the guards have nothing to compare it against: a field that enters no equation, repeats on no other document, and matches nothing in the registry, or, less often, a field whose wrong value still satisfies every equation that does apply. No guard fires, and the reason matters, it is not that the value was checked and found consistent, it is that nothing constrains that field in the first place. The number is a real number from somewhere, nothing is out of place, and a wrong-pick survives for exactly the same reason: there was no redundancy to contradict it. Only semantics, or a human, catches a well-formed wrong answer, whether chosen or invented. This is the precise limit of the technique, and it is worth stating plainly rather than burying: deterministic guards close identifiers completely and arithmetic almost completely, and are weakest exactly where the model’s error is a plausible, well-formed, wrong answer.
Why not just run it three times and vote?
The obvious objection to all of this is that you do not need clever checks, you need three runs and a majority vote. The data answers this directly, and it is the result I find most convincing.
When a model slips on a document, and you run the same document again, it makes the same slip 93.5% of the time, 1730 of 1850 comparable cases (95% Wilson interval [92.3%, 94.5%]). The error is a property of the checkpoint and the input, not noise. Voting across reruns therefore fails on almost every slip, because there is nothing to out-vote: all three runs agree on the wrong answer.
But when the same fact is printed on two different documents (a NAV that closes one quarter and opens the next), the model replicates the slip once in 38 comparable cases, 2.6%, with a wide Wilson interval of [0.5%, 13.5%]. Those 38 cases are concentrated (three funds, three models, mostly Claude Haiku and Sonnet), so the effective sample is smaller still, and I report this as a bound rather than an estimate. But even the top of that interval, 13.5%, sits about seven times below the bottom of the same-document interval. Structural redundancy beats statistical redundancy because it compares independent observations of the same fact, not the same observation repeated. That is the case for cross-document checks over voting, and it is the result I trust most in this study.
How far triage narrows the review
This is where that acceleration becomes concrete, and it is targeting the human, not removing them. If you rank documents by how many guard violations they trip (spreading a cross-document violation across the documents it implicates) and hand the most-suspicious ones to a reviewer, reviewing the 20% most-flagged documents surfaces about 80% of the faults, and 32% surfaces 90%. Then the curve flattens hard, sitting near 91% from 45% of documents to 55%: the remaining faults are the invisible ones from the previous section that trip no guard and therefore carry no triage signal, so getting to 95% means reviewing nearly three-quarters of everything. This curve is computed with knowledge of where the faults are, so it is an optimistic bound on what a real triage queue would achieve. The flat tail is the same floor seen from another angle: you can cheaply find most of the errors, and you cannot cheaply find the last few.
Do the guards cry wolf?
A checker that flags every healthy document has not removed the human, it has renamed the work. Two kinds of false alarm matter here, and they are very different.
The structural checks (the equation and database checks) have a false-alarm rate that is essentially zero by construction: a correctly transcribed document from a corpus whose invariants hold cannot violate an invariant. That is a property of the design, not a discovery, and I report it as what it is, an audit of the normalization and comparison code (across roughly four thousand extractions it fired four times, each a genuine coarse-grained catch on a real misread, not a spurious fire). The number that actually measures whether vigilance costs noise is the false-alarm rate of the heuristic checks, the ones that can fire on a correct document: majority voting and cross-model agreement, scored on the same fields the guards cover. Scored that way, they are not drowning in noise. Majority voting across three runs raises 571 disagreements; about a quarter of them are false alarms (27%, cases where the value was actually right), and it catches about half of the faults it is scored against (47%). Cross-model agreement flags more and detects more: 38% of its flags are false, but it catches four in five of the faults it could. That cross-model figure is an agreement between two different budget models, not a majority vote across many. Using two different models instead of reruns of one lifts detection sharply, because different models make less correlated mistakes, but it lifts the false alarms with it. So voting is a real tool, not a useless one. The narrower point still stands, and it is the reason the deterministic checks are worth building: the structural checks fire on a correct document essentially never, by construction, while every heuristic flag, right or wrong, still needs a human or a ground truth to decide which reading is right, which is the human-in-the-loop creeping back in through the side door. More models would not remove that: vision models share failure modes on the hardest inputs, so they agree on the same wrong answer exactly where it would matter most. The structural checks are silent on correct documents but blind to semantic picks; the heuristic checks see the semantic pick but cannot certify it without the ground truth the guards are built to do without.
The one guarantee with no asterisk
Almost everything above is a measured rate with a confidence interval and a corpus caveat. One result is not. Because the reference database declares which documents are expected in each period, a document that never arrives, or a call missing from a cumulative chain, is detected by construction, and this is proven by drop-one tests in CI rather than by any campaign. A “model plus vote” pipeline has no way to notice a document it was never given. There is a practitioner’s lesson under this one result: a business rule, here the reference database’s calendar of which documents each period should contain, sometimes buys far more than any amount of parsing accuracy, since a document that simply never shows up is caught without a single character being read. This is the one place the architecture offers a guarantee rather than a probability, and it is the anchor of the whole approach: not “the model is more accurate,” but “the pipeline knows what it has not seen.”
Limits, stated plainly
These are synthetic documents, and the scans are clean-office degradations, not the hostile photocopies and phone captures of the real world. The prevalences (how many identifier errors, how many wrong-picks) are artifacts of this corpus and of the degradation recipe; only the conditional catch rates and the completeness properties of the guards carry over, and even those should be read as directional given document-clustered confidence intervals. The reference database is authored alongside the documents, so it is independent of any single issuer’s claims but not epistemically independent of me; in production its independence comes from being real master data, which is a stronger claim than I can make here. The identifier row’s one hundred percent leans on this the hardest: it includes a wire-reference format convention and per-fund declarations (LEI, reporting units, mandatory fields) that this registry knows completely and a typical production registry might not. The economic angle (a cheap model plus guards versus an expensive model alone) I deliberately do not headline: the per-extraction API cost of the frontier tier ran about eight to nine times the budget tier here, but the cost that dominates is engineering the guards and the database, and the cost of the errors that still escape, neither of which this study prices. And the whole thing measures the transcription layer, not the messy reality of bad source data and restatements, which is a different and harder problem.
What I think holds up, and would build on: the method transfers to any structured-extraction pipeline, the per-field capture map and the classification of each guard by the strength of guarantee it offers are the parts worth stealing, and the honest limit (deterministic checks cannot see a plausible wrong answer) is a design constraint worth designing around, not a footnote.
The one extension I would build first aims straight at that constraint. A hallucination is, by definition, a value that appears nowhere in the document, so checking that each extracted value is actually present in the source, grounding every field to the text it came from, attacks exactly this class and only this class, since a wrong-pick is a value that is present, just in the wrong place. The difficulty is where the source text comes from. This stays a code-side guard only if presence is verified against a trusted, non-model source, a PDF text layer or a reliable OCR pass; a bounding box returned by the model is the model checking itself, and a reader confident enough to invent a value is confident enough to invent its location, the same correlated-error failure that sinks voting. It is weakest exactly on the hardest scans, where the text layer is gone and OCR is least reliable, and it needs fuzzy, normalized matching to avoid firing on legitimate reformatting, which makes it a heuristic guard with a real false-alarm rate rather than a structural one. It would chip at the hallucination floor. It would not touch the larger wrong-pick floor at all.
Reproducible: the corpus, the constructed oracle, the guards and the scoring harness are all in PittsCraft/doc-extraction-evals. Every catch rate, count and interval here regenerates from the committed model outputs with pnpm report:v2 and pnpm coverage:v2; the API cost figures are logged campaign metadata.