Look at the Spread of Responses
A cohort’s value is in how the responses split across people. Say 30% of replicas react one way and 60% another: that split is the finding. Reporting only an average would blur those two groups into one middling number and hide the disagreement, which is the real range of opinion you are trying to see. You read this split in the project report once a Rehearsal finishes (see Running & Monitoring a Rehearsal).Stated intent from a cohort is directional, not definitive. “72% would consider buying” is best
read as which option wins and by roughly how much. It is a way to rank and compare, not a literal
forecast of real-world sales.
When a Simulation Surprises You
An unexpected result is not automatically a wrong result. Before you discount it, work through this checklist in order.Did you interview the right people?
Check the screening. A surprising number often means the cohort included people the study is not
really about. Predicting churn is meaningless if half the cohort never subscribed.
Was the question leading or biased?
Re-read every question as if you were trying to prove the opposite. Loaded framing, an anchored
price, or a presumed behavior can manufacture a result. Try the
rephrase check.
Is it a stated-versus-observed gap?
People (and replicas) overstate intent on discretionary things (“I would cancel over $3”) and
understate inertia. If the surprise runs in that direction, the result may be more honest than
your expectation.
Does the reasoning hold up?
Read the transcripts. If a counterintuitive result is backed by consistent, grounded reasoning
across replicas, it is likely real and worth understanding rather than dismissing.
A Unanimous No Is Also a Signal
When a full cohort rejects something, that is information. In one validation, every replica declined a status-priced product that later underperformed in the real market. A 0% result from a representative cohort is not a bug. Run before launch, it is an early warning you would otherwise pay for in ad spend and opportunity cost.Handling Results With Care
Sometimes a simulation contradicts a strongly held expectation, such as a creative someone loves or a price someone is sure about. Approach these the way a good researcher would.Suspect the question first
More often than not, a jarring result traces back to how the question was framed rather than the
replicas being wrong. Fix the framing and rerun before concluding anything.
Back it with the transcripts
When a result is surprising, read the actual reasons replicas gave in the transcripts, and
compare against a real-world benchmark if you have one. The number is far more convincing once
you can see the thinking behind it.
Creatives are genuinely hard to judge. Even experienced teams often cannot articulate why an ad
fails. In the Mera NYC case study, the founder had spent weeks and real budget on a creative she
was convinced would win. Replicas flagged it as the weaker option, matching the real ROAS
outcome. The lesson is not to trust the machine blindly. It is that a grounded replica panel is a
cheap check before you commit real money.
What to Do Next
- Rerun with neutral framing if the question checklist surfaced anything.
- Increase the cohort size for divisive topics where you need the full spread.
- Split by intent to see whether the result depends on the mindset you are testing.
- Read the transcripts. The reasoning behind the spread is usually the actual insight.
Back to Overview
Revisit how replicas differ from personas and a general AI.