by Serguey Shinder
The pricing module had no tests. It was six years old, about nine hundred lines, and every change to it followed the same nervous procedure. Read it for an hour. Change three lines. Run the application by hand and stare at the numbers.
So I did the obvious thing. I handed the file to a model and asked for a test suite. It gave me forty-three tests. They were well named, they covered the branches, they were arranged the way our tests are arranged, and every single one passed the first time I ran them. Coverage on that file went from nothing to the high eighties in an afternoon, and I remember writing a cheerful sentence about it in the team channel.
Six weeks later a customer on a particular kind of account was charged the wrong amount. The discount was being applied after tax rather than before, on one path through the module, which for most accounts made no difference and for theirs made about four pounds.
I went to the tests expecting to find a gap. There was no gap. There was a test, clearly named, asserting that on this path the discount is applied after tax, passing contentedly since the afternoon I generated it. The suite had not missed the bug. It had written the bug down and made it official.
Nothing had gone wrong with the tool. I asked it to describe what the code did, and it described what the code did. What I actually wanted, without ever saying so and without possessing it myself, was a description of what the code should do. That lives in a pricing policy, in the head of somebody in finance, in a contract. It does not live in the source, and no amount of reading the source will recover it.
A test written from an implementation is a photograph of that implementation. Photographs are genuinely useful. If I refactor the module now, forty-three tests will tell me the moment behaviour shifts, and that is worth having. What they cannot tell me is whether the behaviour was right, which was the only question the module ever raised.
The worse part was social. The instant the suite went green, all pressure to write real tests evaporated. Nobody argues for testing a file that reports eighty-seven percent. I had not merely failed to write the tests we needed. I had removed the reason anybody would ever ask for them.
The generated ones live in their own directory now under a name that says what they are, which is a change detector. And any assertion about money, eligibility, or who is allowed to see what has to trace back to a person or a document. If I cannot say where a rule came from, I have not tested it. I have only agreed with myself in writing.
– Serguey Asael Shinder
Leave a Reply