OCR for supplier invoices: why it fails and how to do it properly

Keying supplier invoices by hand is the most automatable administrative task there is: high volume, clear rules, zero creative judgement. And yet it's where most projects stall halfway.

They don't stall because of the OCR. Text recognition has been a solved problem for years. They stall on everything else.

What breaks an OCR pipeline in production

Every supplier has their own layout. And changes it without telling you. A system that learns "the tax ID is top right" works until an invoice arrives with it under the logo — and then until that same supplier redesigns their template in January.

Real scans are horrible. The demo uses a native PDF. What arrives is a phone photo, crooked, shadowed, with a received stamp covering half the document number, forwarded through chat and compressed twice.

Tables. Extracting a total is easy. Extracting the line items from an invoice with sixteen rows, where one description wraps onto two lines and there's a subtotal in the middle, is where most of the development time goes.

The things that aren't invoices. The same inbox brings delivery notes, credit notes, purchase orders, and an image of the supplier's logo from their signature block. The system has to classify before it extracts, and getting that wrong contaminates everything downstream.

Duplicates. The same invoice arrives by email, again on paper with the driver, and a third time because the supplier got no reply. Without duplicate detection you end up paying twice — which is precisely the error automation was supposed to prevent.

How to build one that holds up

Classify before extracting

The first step isn't reading, it's deciding what the document is. Invoice, credit note, delivery note, something else. Each type gets its own extractor and its own validation rules. Merging everything into one giant prompt is the fastest route to a system that works 80% of the time and nobody can explain the rest.

Extract with the layout, not just the text

Classic OCR returns flat text and throws away the spatial information — which is exactly what tells you which number belongs to which field. Modern vision models read the page with its geometry: they understand that this number sits in the "Amount" column, in the row for item 3.

That's what lets the system tolerate every supplier having their own design, instead of needing a template per supplier.

Validate against reality, not against the model

This is the part that separates an experiment from a production system. Every extracted field is checked against something you already know:

• Does the tax ID exist, and does it belong to a registered supplier?
• Do the line items add up to the subtotal? Does subtotal plus tax equal the total?
• Is the tax rate one of the allowed ones?
• Have we already processed this document number?
• Is the amount inside this supplier's historical range?

When the arithmetic closes, confidence doesn't depend on the model claiming to be sure — it depends on the numbers reconciling. That's an objective check, and it's what makes automatic approval defensible.

Set the threshold and design the exception queue

No serious system approves 100% unattended. You set a threshold: what validates cleanly goes straight through; what doesn't lands in a queue where a person sees the document beside the extracted data and corrects it in ten seconds.

The realistic target is 80–90% straight-through. The remaining 10–15% isn't a failure, it's the design. Promising 100% is the most reliable sign that whoever is selling to you has never put one of these into production.

Close the loop

Every correction made in the queue has to come back as signal. If one supplier always needs the same fix, that's a new rule — not a recurring annoyance.

What we learned building this

At Fily we process scanned documents in production, and the two lessons that cost us most are counterintuitive.

First: an image's size comes from its rectangle on the page, never from counting pixels. It sounds like a minor technical detail and it's the difference between a rebuilt document that looks like the original and one that looks like a badly assembled puzzle.

Second: when a document fails, the error message almost never tells the truth. You have to look at the pipeline's own result, not the exception that got logged. An OCR system without good observability is a system nobody can fix.

The return

The numbers we typically see in an accounts payable team handling several hundred invoices a month: 60–75% less administrative time, and something less visible but more valuable — the disappearance of typos in amounts and tax IDs, which are the errors that later cost an entire reconciliation.

A typical project is four to six weeks, and the slow part isn't the OCR: it's agreeing what happens to the odd cases.

If you were picturing your own invoice intake while reading this, tell us how it works today and we'll tell you what share is realistically automatable in your case.