AI document processing you can check: what a source check caught on ten 10-K filings, and what it missed
We had three AI models pull six headline figures from ten companies' annual reports, made each one cite the line it read, and checked every figure twice: against the filing, and against the numbers the company itself filed with the SEC. All three models got every gradable figure right. The one wrong figure came from a model quoting a real line that held the wrong number, and our first version of the check let it through.
~6 min spoken. Keeps playing while you work in another tab.
Extracting figures from documents is one of the jobs AI models now do quickly and cheaply. The risk is not that they fail loudly. It is that they occasionally return a plausible number that is wrong, and a plausible wrong number is the one nobody looks at twice.
The usual defence is to make the model cite its source: give the value, and the line of the document it came from. A citation can be checked by code, with no second model and no answer key, so it is the check that can run on every document a client processes. We wanted to know how much that check is worth, on documents where we could find out.
The test
Public companies' annual reports, their 10-K filings, are a good place to find out, because each figure in them is reported twice. It is printed in the financial statements, and the company also files it as tagged data (XBRL) with the SEC. That tagged data is an answer key the company wrote itself.
We took the latest 10-K of ten large companies: Apple, Microsoft, NVIDIA, Amazon, Walmart, JPMorgan Chase, Johnson & Johnson, Exxon Mobil, Coca-Cola and Costco. Their financial statements run from 63,000 to 400,000 characters. From each we asked for six figures: revenue, operating income, net income, diluted earnings per share, total assets, and cash and cash equivalents.
Each answer had to be a value and a quote: the statement row it was read from, copied character for character. Every answer then went through two checks:
- The source check, the one a client gets in production. The quoted row must occur in the filing and contain the value. A failure goes to a person.
- The answer key, which only exists here. The value, scaled to dollars, is compared with the fact the company filed for the same year.
Fifty-seven of the sixty figures have an answer key. The other three are operating income for JPMorgan, Johnson & Johnson and Exxon Mobil, whose income statements have no such line. The right answer for those is to return nothing.
We ran the same prompt through Gemini 3.8 Flash, Claude Opus 5.5 and Claude Sonnet 5.5, each at its low reasoning setting.
Every gradable figure was right
| Model | Keyed right | Right, flagged | Wrong | Wrong, flagged |
|---|---|---|---|---|
| Gemini 3.8 Flash | 57 of 57 | 0 | 0 | 0 |
| Claude Opus 5.5 | 57 of 57 | 0 | 0 | 0 |
| Claude Sonnet 5.5 | 57 of 57 | 2 | 1 | 1 |
Sources: SEC EDGAR 10-K filings; SEC XBRL Company Facts
All three models got all 57 keyed figures right. That is good news about the models, and awkward news for a demonstration of a checker: with no wrong figure among the 57, there was nothing for the check to catch. We cannot yet say how often it catches a wrong figure, only that it did not raise false alarms on the right ones. That number has to come from harder documents, and we are running them now.
The error we did find, and why the check passed it
Banks do not report operating income, and JPMorgan's income statement has no line for it. Gemini and Opus correctly returned nothing. Sonnet returned $72,595 million, quoting the row "Income before income tax expense". It is a real row, for the right year, and the number is on it. It is the wrong figure: pre-tax income, not operating income. Sonnet made the same substitution when we ran it a second time.
| Model | What it returned | Source check |
|---|---|---|
| Gemini 3.8 Flash | Left blank | Nothing to check |
| Claude Opus 5.5 | Left blank | Nothing to check |
| Claude Sonnet 5.5 | $72,595M, from "Income before income tax expense" | Flagged: row label is not operating income |
Sources: SEC EDGAR 10-K filings; SEC XBRL Company Facts
Our check, as first written, passed it. The quote is in the filing and contains the value, which is all it asked. There was no answer key for this figure, so nothing else would have flagged it either. On a client's invoices or contracts, where there is never an answer key, this is the error that ships.
So a quote check proves that a figure is printed in the document. It does not prove that it is the figure you asked for.
Tightening the check, and what that cost
We added one rule: the quoted row's label must name the figure asked for. Operating income has to come from a row labelled operating income, operating profit or loss, or income from operations. The patterns are deliberately loose, because companies label the same line differently. Johnson & Johnson reports revenue as "Sales to customers", Coca-Cola as "Net Operating Revenues", and JPMorgan's cash line is "Cash and due from banks".
The rule flags Sonnet's JPMorgan figure, as the table above shows. It also has a cost. On the first runs it flagged correct figures whose quote had no label at all, such as a bare "6.70 | 7.84 | 8.89" for Exxon's earnings per share: four for Gemini, three for Sonnet and none for Opus, out of 57. Each of those means a person checking a figure that was right.
All of them had the same cause. In the filing, the label sat on the line above the numbers, often behind a footnote marker such as "Total assets(a)", and the model copied only the number line. One more sentence in the prompt, telling the model to copy from the label through to the value, cut the false alarms to none for Gemini, two for Sonnet and none for Opus. Sonnet's remaining two are JPMorgan's total assets and Coca-Cola's diluted earnings per share, both in that same layout.
What the check still cannot catch
A row that is labelled correctly and still holds the wrong number passes. The clearest case is the prior-year column: each statement row prints two or three years side by side, and a model that takes last year's figure quotes a real row, with the right label, containing the value. A segment subtotal labelled "Operating income" would pass the same way. Against those, the protection is an answer key, a second source, or a person.
What this means if you are processing documents
- Make the model cite the line. It is what makes checking cheap, and it tells a reviewer exactly where to look when something fails.
- Check the label, not just the number. A number found in the document is not the same as the right number.
- Measure the check where you know the answers, before trusting it where you do not. The SEC's tagged data made that possible here. Most organisations have a backlog of documents already processed by hand that can serve the same purpose.
- Choose the model on your own documents. Two of the three models here never made the substitution. One made it twice.
We are now running harder filings, chosen to make errors: insurers and banks, companies with several share classes, companies reporting losses, and smaller filers with less regular layouts. We will update this article with what the check catches there, including if it catches nothing.
If you process documents and want this kind of measured pipeline for your own, tell us about the document type.