by Serguey Shinder
A parcel worth two hundred and forty pounds went to a flat our customer had moved out of in January, and for two days we believed an agent had mistyped the postcode.
She had not. The audit trail, once somebody read the whole of it rather than the last line, showed the delivery address changed at nine minutes past eleven on the Tuesday, and changed back to the old one at fourteen minutes past, by a different member of the support team who had not touched the address at all. He had added a note about a promised callback and pressed save. That was the entire incident.
Our edit screen loaded a customer, gave you a form with thirty four fields on it, and sent all thirty four back when you saved. Two people had the same record open, four minutes apart. The second save was not wrong about anything it contained. It was a picture of the customer taken before the change, written over a picture taken after. Nothing failed. No error was raised, and both agents saw the green line that says the record has been updated, because it had.
We went looking afterwards with a query that compared consecutive versions of every record and flagged any field that had returned to a value it held before. In a year there were about eleven hundred. Most were harmless. A few dozen were telephone numbers, which means somebody rang a stranger and told them about an order. Four were bank details.
The part I remember most is that the support team already knew. Not in the sense of having reported anything, but as folklore. You had to check your change had stuck. The system was funny like that on busy mornings. They had built an accurate working model of a defect nobody had told them existed, and had never described it to us in words we could act on, because from where they sat it was not a defect. It was weather.
What we changed is ordinary. Every record now carries a version, and a save against a version that has moved is refused, with a screen that shows what changed, who changed it and when, and lets the agent keep both. That refusal happens about forty times a week. Forty times a week, for years, our system had quietly chosen between two people and told neither of them it had.
The distinction I carry is between two things I had treated as one. The absence of an error is not the absence of a conflict. Our system was written by people imagining one person holding a record, and our business puts eleven people on the same account on a busy Tuesday. Last write wins is a decision. Almost nobody makes it deliberately, and every system that has not made a different one has made that one.
– Serguey Asael Shinder
Leave a Reply