Extracting structured data from invoices and documents with LLMs: what accuracy to expect

Pulling names, dates and amounts out of PDFs, invoices and emails is one of the most common business uses of language models, and one of the easiest to measure. Every extracted field has a right answer you can check. This post looks at what a published invoice benchmark reports about accuracy, and how to measure it before you trust the output in a workflow.

What the evidence shows for invoices

A benchmark from Fraunhofer IAIS researchers, posted in August 2025, tested eight multimodal models (GPT-5, Gemini 2.5 and Gemma 3 families) on three open invoice datasets, with zero-shot prompting ^[2]. A field counted as correct only if it matched the ground truth exactly after minimal normalization, such as date format and whitespace ^[2].

Four findings stand out for anyone planning a pipeline ^[2]:

• Messier inputs lower accuracy. The same model scored 96.50%, 92.71% and 87.46% as documents went from clean to scanned.

• Reading the image beat converting to text first. Passing page images directly to the model beat a pipeline that first converted pages to markdown with an open-source tool: for example 92.71% versus 64.03% on scanned invoices.

• Some fields are much harder. Bank account numbers (IBANs) were extracted significantly worse than other fields, with mistakes such as confusing the digit zero with the letter "O".

• Smaller models can fall off a cliff. A 4-billion-parameter open model scored 45.69% on the clean invoices where larger models exceeded 84%.

The authors also note that the datasets are open source and may have been in the models' training data, and that they could not find a suitable dataset of native-text PDFs ^[2]. We did not find a comparable published benchmark for extraction from emails, so we do not quote an accuracy figure for them. One other 2025 benchmark, of legal-domain AI tools, reported that simple single-fact lookups were handled reliably while extraction of many fields across several documents was much harder ^[1].

How to measure it on your documents

Both studies make the same point: an overall percentage hides where the errors are. The practical approach is to measure like they do, on your own material.

1. Build a test set that mirrors real work. Anthropic's evaluation guidance says to design evals that mirror your real-world task distribution and to include edge cases ^[3]. For documents, that means your worst scans, unusual layouts and missing fields, not only clean samples.

2. Score field by field. Define the exact-match rule up front, including how dates, currency and whitespace are normalized, as the invoice benchmark does ^[2]. Report accuracy per field so a weak field like an account number is visible.

3. Automate grading where you can. Anthropic recommends structuring questions for automated grading, and says more questions with slightly lower-signal automated grading beat fewer hand-graded ones ^[3]. Extraction suits exact-match grading well. If you use a model as judge, the Vals report on legal-domain tools is a reminder that an LLM judge approximates, and does not replace, expert review ^[1]; Anthropic advises using a different model to grade than the one that produced the output ^[3].

4. Keep the test set and re-run it. A model or prompt change can shift accuracy on specific fields without moving the average, so the same set should run on every change.

Where Sinthetix fits

Sinthetix's Generative AI & LLM service lists document analysis among its capabilities: extracting structured data, summaries and insights from PDFs, invoices, emails and other unstructured sources. It also lists prompt engineering with evaluation frameworks, and automated evaluation pipelines that catch regressions when models or prompts change. The engagement starts with a data and use case audit that defines measurable quality targets, then builds and evaluates the system against a test set, and deploys with logging and evaluation hooks so performance can be tracked.

Practical implication

Before automating extraction, decide which fields matter, what error rate each can tolerate, and who reviews the rest. Then measure on a sample of your real documents, scans included, instead of relying on a published average. Talk to us: sales@sinthetix.com