Research
Your Customers Are Not the Average
At Google with Plug and Play, Liam Bolling shared why strange customer behavior is not noise, why generic AI answers collapse toward the mean, and what prediction has to do differently.
Recently, I returned to Google to give a talk with Plug and Play to more than 200 people working across insurance, finance, healthcare, beverage and alcohol, consumer goods, and technology. The title was simple: Your Customers Are Not the Average.
The argument started with a story from my time on Google Maps. It ended with a test I think every company should apply to AI prediction: can you audit an answer all the way back to a real person, and then compare the prediction with what actually happened?
Talk deck
Your Customers Are Not the Average
Move through the 34-slide talk here, use the arrow keys, or open the original PDF.
1 / 34Your Customers Are Not the Average
The dashboard called them bad users
Google Maps gets much of its accuracy from ordinary people correcting the map. Move a pin, fix an address, report a closure. But we kept seeing people suggest an edit that moved a perfectly good address thousands of miles away. Internally, the chart tracking the behavior called them bad users.
We spent months looking at the data without understanding the behavior. The answer arrived over dinner with a friend: those users had moved homes. They thought changing the public map listing was how they updated their own home address.
They weren’t bad users. They were just using our product.
The dashboard captured what people did, but not why. We had designed for an average user who understood the distinction between a private home address and a public map edit. The real people did not arrive with that model in their heads. If we could have asked the people who had just moved, we would have learned that before we shipped.
The average gives you the wrong confidence
Every business decision is a bet on what people will do next. If we raise premiums, who leaves? Who opens a new card? Which patients stop refilling? There are always two questions inside the decision: what will happen, and why?
A generic language model can answer in seconds. The problem is that the answer tends to sound like the average. In the streaming example from the talk, a generic response estimated that 17% of subscribers would cancel after a $3 price increase and advised against the change. What happened was very different: 94% paid the higher price, fewer than 1% switched to the ad tier, and 5% cancelled.
The miss was not random. Averages smooth away inertia, status quo bias, mental accounting, and the thousand personal reasons somebody does not behave like the clean persona in a strategy document. A language model is very good at describing the center of a distribution. Your customers live everywhere else in it.
Prediction has to work at the individual level
Companies have three common options. Interview real people, which is accurate but slow and expensive. Simulate a population, which is fast but often detached from the people who actually buy the product. Or wait for a future model to solve the whole problem. None gives teams prediction at the speed of simulation with the why of every individual customer.
Our answer at Rehearsals is to own the full chain and hold every link to a standard:
- Person: begin with a real, consented individual and the context of their life.
- Twin: preserve what makes that person different from the mean.
- AI model: require the twin to respond the way the human does, not merely produce a fluent answer.
- Outcome: compare the prediction with what people actually do in the real world.
Rehearsals reports 92% accuracy to what real humans would say or do. The number matters, but the audit trail matters more. Every answer can be traced to the real person behind the twin, and every important prediction can be checked against a human response or an observed outcome.
Three things to take back
- People are not a mean average. Stop asking the mean average what they will do.
- Strange behavior is not bad behavior. It is an invitation to ask why.
- Whoever you buy prediction from, make sure you can audit the answer down to the real person.
Thank you to Google and Plug and Play for the room, and to everyone who stayed to ask hard questions after the talk. The industries represented were different, but each had its own version of the same problem: the most consequential customer is almost never the average one.
