by Serguey Shinder
We added a weekly summary to our product in January. It read what had happened in your account over seven days and wrote you four paragraphs about it. Each summary cost us four fifths of a penny to produce and the plan those customers were on was seven pounds a month, so at the four summaries a month we had modelled, the feature cost about three pence per customer and nobody in the room thought about it again.
In March we added a small button underneath that said try again, because the first summary was often flat and the second one was usually better. It was an obviously good change. Nobody asked what it did to the model, and I mean the spreadsheet rather than the language model, because we did not have a spreadsheet.
By June the median customer was generating three point one summaries for every summary they read. The heaviest five per cent were generating sixty a month. One customer, who I suspect had the page open on a second screen, generated four hundred and ten in a single week. The bill for that feature went from about nine hundred pounds in the first quarter to eleven thousand four hundred in the second, on a customer base that had grown by eleven per cent.
The thing that took me longest to understand was not the number. It was that every one of those extra generations was a customer telling us the last one had been no good. The retry rate was our accuracy measurement, sitting in the billing data the whole time, and we had been reading it as usage. A feature that answers poorly costs more to run than a feature that answers well, which is not true of anything else we have ever built. A slow page is a slow page. It does not send us an invoice.
What we changed was the affordance. The button now asks what was wrong in three words and edits the existing summary rather than starting again, which is cheaper, better, and tells us something. Cost per active customer sits on the same dashboard as latency and error rate, and it is reviewed in the same meeting, by the same people who wrote the feature.
The distinction I now carry is that for twenty years the marginal cost of one more click was zero, and every instinct I have about interface design was formed inside that fact. Add the button. Let them try it again. It costs nothing. That has stopped being true for a whole category of feature, and the people making the decision are designers and engineers rather than the finance team, who see it a month later as a number with no story attached. Unit economics has quietly become a design discipline and almost nobody has been trained for it, including me.
– Serguey Asael Shinder
Leave a Reply