by Serguey Shinder
The customer had a screenshot and she was entirely reasonable about it. Our help page assistant had told her, at ten past nine on a Sunday evening, that we offer a full refund within thirty days of renewal, no questions asked. She had cancelled on that basis. Our actual policy is fourteen days and pro rata.
We honoured it, which was correct and cost about two hundred pounds.
The assistant had not invented the policy. That was the part that took me longest to accept, because I had spent months confident that our arrangement was safe precisely because it only answered from our own documents. It had found a marketing post from 2022 describing a promotion that ran for a single quarter, and it had quoted it accurately. The document was real. The page was still on our site, because nobody removes a blog post, and nothing in the text said the offer had ended that December.
Every evaluation I had built asked the wrong question. I checked whether an answer was supported by a source, and this one was. What I had never asked was whether the source was still true, because our corpus was not a curated set of policies. It was everything we had ever published, assembled by pointing a crawler at our own domain, and a company's website is not a statement of what is currently the case. It is a sediment.
Underneath that sat a bigger thing I had not thought about at all. I had been treating the assistant as an information retrieval system, and judging it by whether the information was there. It is not an information retrieval system. It is a channel through which our company makes statements to customers, and a statement made by the company is binding no matter which part of the company produced it, at what hour, or how confident anyone was in the machinery. We had deployed something capable of entering into commitments, unattended, at ten past nine on a Sunday, and the only review it received was mine, before launch, on questions I had chosen.
We went back through a quarter of transcripts. Forty-one conversations contained something I would call a commitment. Most were harmless. Six were not.
Three things changed. The corpus is now a small, owned, dated set of current policy documents rather than our published history. Anything touching money, cancellation, eligibility or a date routes to a person, with a drafted answer attached, which costs a few minutes and is still faster than we used to be. And we sample conversations weekly against what the policy actually is, rather than against what the documents say.
The question was never whether it is accurate. It is what it is permitted to promise on our behalf while nobody is watching.
– Serguey Asael Shinder
Leave a Reply