by Serguey Shinder
The partner emailed on a Wednesday to ask, politely, why they had received the same eleven orders twice in one morning. They had removed the duplicates themselves, which is the only reason this arrived as a question rather than as an incident, and it was not the first time. Looking back through their own records they found nine mornings that month, always between eight and eleven.
Our export ran hourly. It selected the orders that had not yet been sent, wrote them into a file, delivered the file, and then marked those orders as sent. Four lines to describe, written in 2018, and correct in every respect except one that nobody had ever written down.
The assumption lives inside the word hourly. A job is only hourly if it finishes within the hour, and ours had stopped doing that. Not suddenly. The order table had been growing for four years, the selection did a little more work every month, and somewhere in the spring a busy morning run crossed seventy minutes. From that point the next run began while the previous one was still working, found the same unsent orders, because nothing had been marked yet, and delivered them a second time.
Every signal we watched said the job was healthy. It exited cleanly. It logged no errors. It produced a valid file every time. The one number in which the fault was visible was how long it took, and that was the one number nobody had ever plotted, because it had never been interesting.
There is a second thing in there that I now think is the more general mistake. We claimed the work at the end rather than at the beginning. Between selecting a row and marking it there was a window of up to seventy minutes in which that row looked untouched to anybody else who asked, and a schedule guarantees you will eventually have somebody else asking. Marking rows as claimed the moment they are picked up, with the claim owned by a run identifier, removes the entire class of problem whether or not the runs overlap.
The changes were small. The job takes a lock and declines to start if one is held, loudly. Every scheduled task now records its duration, and anything that exceeds half its own interval raises a warning long before it exceeds the whole one. Rows are claimed at selection.
The habit is the part I would keep. Every recurring job in a system carries a silent promise that it will be finished before it is asked to start again, made by somebody who measured it once, on a quiet afternoon, against a fraction of today's data. Nothing in the system re-checks that promise, and the job will not tell you when it stops keeping it. It will simply start overlapping with itself and go on reporting success.
– Serguey Asael Shinder
Leave a Reply