by Serguey Shinder
I had a design I was unsure about, so I did what had become natural. I described it to a model, asked what was wrong with it, got a list, addressed the items on the list, and felt better. Then, because I was still faintly uneasy, I opened a fresh session and asked again in different words, hoping for something like an independent check. It agreed with the revised design. I took that as confirmation and moved on.
The flaw in this took me a long time to see, and it embarrasses me slightly, because it is not subtle. I had not obtained two opinions. I had obtained the same opinion twice, from something with the same training, the same blind spots, and, most importantly, the same description of the problem, written by me, containing exactly the same omissions both times.
The omission was what mattered. I had not mentioned a constraint about how the data actually arrived, because I had internalised it so completely that it no longer registered as information. Both sessions reasoned beautifully inside a world that did not contain that constraint. Both arrived at a conclusion that was correct in that world and wrong in ours.
A human reviewer would have failed differently. My colleague would not have known the constraint either, but he would have asked something adjacent that made me say it out loud. Where does this come from, who calls it, what happens on a retry. Those questions come out of curiosity about a situation, and a model asks fewer of them because it is shaped to be useful with whatever it has been handed.
What I do now is treat repetition as worth nothing. If I want a second opinion, the second one has to differ in a way that actually matters: a framing I did not write, a person who works on the system that calls mine, or an explicit instruction to build the strongest case against the thing I have already decided. Asking the same question twice is a comfort exercise wearing the costume of diligence.
I have also started noticing when agreement arrives too easily. Not because agreement is suspicious in itself, but because the moment I feel relief is the moment I stop testing, and relief is precisely what a second confirmation manufactures. Certainty is the output I should trust least and the one I keep going out to look for.
The model was not wrong that day in any way I could fairly hold against it. It answered the question I asked, twice, using what I gave it. The thing that was missing was inside my own head, and no amount of asking would have drawn it out, because I did not know it was there to give.
– Serguey Asael Shinder
Leave a Reply