by Serguey Shinder
Every month our three regional carriers send us invoices for the deliveries they make on our behalf. Each one runs to hundreds of lines: a consignment, a postcode, a weight, a pallet charge, a fuel surcharge, sometimes a tail lift or a failed delivery fee. For years somebody in our finance team spent most of the first week of the month typing those lines into a spreadsheet, so that each branch could be charged for its own deliveries.
In the spring of 2024 we gave that job to a language model. It read each invoice, picked out every line, worked out which branch it belonged to, and produced a short summary with a total for each branch. It was remarkably good at the reading. It coped with three invoice layouts and scanned pages. We checked the first two months line by line and found almost nothing wrong.
In September our finance controller reconciled the quarter and found that the branch charges added up to eleven hundred and eighty pounds more than the carriers had actually billed us. Not much against the total, but it should have been zero.
The lines were fine. When we went back through them, the model had read nearly every one correctly. The totals were the problem. On a summary of sixty or seventy lines, one branch total would be a few pounds out, or two digits would be swapped, and once a single line had been counted twice. Each total looked exactly like a total should look. Nothing in the output said that one of them had been estimated rather than added.
What I had misunderstood was what the model was doing when it produced a number at the bottom of a column. A language model does not add things up the way a spreadsheet does. It writes the answer that most plausibly follows the text in front of it, and most of the time that is the correct sum. Most of the time is fine for a description of a consignment. It is not fine for a figure we were going to charge to somebody's budget.
So we split the job in two. The model still does the reading, which is the part it is genuinely better at than anything we had. It now returns only the lines, one per row, in a fixed format, and never a total. Ordinary code adds them up. Before anything is posted, the sum of the extracted lines is compared with the total printed on the carrier's own invoice, and if they differ by a penny, the invoice goes to a person with the mismatch highlighted. About one invoice in thirty stops there.
The quarter has balanced to the penny ever since.
Use the model for the work that needs reading. Anything that needs counting belongs to something that cannot guess.
– Serguey Asael Shinder
Leave a Reply