by Serguey Shinder
I needed a client library to stop retrying a connection forever and start giving up after a while, so I asked a model how to configure it. The answer was immediate and specific. There was a property, it named it, it told me the default value, it explained how that property interacted with another one I already knew was real, and it warned me about a sensible edge case.
I put the property in our configuration file and deployed. Nothing changed.
What followed was four hours of the most ordinary debugging I have ever done. I checked the spelling. I checked I was editing the file that was actually being read, which took twenty minutes and involved printing the resolved configuration at startup. I checked precedence between environment variables and the file. I rebuilt the image in case something was cached. I read our own wrapper around the client in case we were overriding it. Around the middle of the afternoon I stopped believing my own eyes and asked a colleague to look at the file with me.
Then I searched the library's source code for the property name, and it was not there. Not deprecated, not renamed, not moved to another package. It had never existed.
Two things had to be true for this to cost four hours instead of four minutes. The first is that the invented name was excellent. It was in the exact style of that library's real properties, the same prefix, the same word order, and it sat comfortably next to the genuine settings in my file. Nothing about it looked wrong, because there was nothing wrong with it except that it was fictional.
The second is that the library accepted it in silence. Unknown keys were ignored. That is a design decision somebody made for good reasons about forward compatibility, and it meant the system had no way at all of telling me I had asked for something meaningless. I want to be fair about where the blame sits: the model produced a plausible name, which is what such things do, but the four hours came from our own configuration accepting anything I typed.
Both fixes were small. Our configuration loader now fails at startup on an unrecognised key, which found two other dead settings that had been doing nothing for a year. And any identifier I am given by a model, a config key, a method, a header, a command line flag, gets searched for in the actual source or the actual documentation before it goes anywhere near a file.
That check takes two minutes and it is the cheapest one available, because a name is the one kind of answer I can verify completely and cannot evaluate by reading.
– Serguey Asael Shinder
Leave a Reply