by Serguey Shinder
In the summer of 2025 we built a model to sort the email that arrives at our customer services address. About two thousand five hundred messages a day: orders, delivery queries, returns, invoice disputes, quotes and everything else. Each one has to reach the right team, and for years that had been the first job of whoever opened the inbox at seven.
To check it, two of our team leads labelled six hundred real emails from the previous spring with the team each should have gone to. Over three weeks a developer worked on the instructions we gave the model, ran them against those six hundred, read the mistakes and adjusted. To help with the hard cases, he copied forty of the trickiest emails from the set, with their correct answers, into the instructions as examples. By the end it routed ninety six per cent of the six hundred correctly, and we switched it on.
A month later our returns team lead asked why she was receiving so many invoice disputes. We took five hundred fresh emails from the live inbox and labelled them the same way. The model had routed eighty one per cent of them correctly.
Nothing about the model had changed. What had changed was that it was now being asked questions it had not seen. Forty of the six hundred test emails were sitting in its instructions with the answers attached. The other five hundred and sixty had shaped three weeks of adjustments, each one kept because it fixed a mistake on that particular set. We had not measured how well the model sorted our email. We had measured how well a set of instructions, written while looking at six hundred emails, sorted those same six hundred.
The developer had done nothing careless. The arrangement was what was wrong. The same emails had been used to build the thing and to judge it, and a test you keep fixing against stops being a test and becomes part of what you built.
So we keep three piles now. One set of emails is for writing and adjusting the instructions, anybody can look at it, and examples come only from there. A second set of four hundred is held by one team lead and used only to decide whether a change goes live, and the developer sees the score but never the emails. And every week fifty live emails are labelled by hand and compared with what the model did, which is the only figure that appears on the dashboard.
That figure is now eighty nine per cent, and I believe it, which I could never have said about the ninety six.
A model will always do well on the questions you showed it while you were building it. The only score worth reporting comes from the ones you kept away.
– Serguey Asael Shinder
Leave a Reply