I’ve spent a lot of time working with both experimentation and personalization, and one of the things I’ve noticed is how often they’re treated as separate disciplines.
A/B testing is typically focused on improving an existing experience. We come up with an idea, test it against what’s there today, and measure whether it performs better.
Personalization, on the other hand, has become so focused on what we know about the customer. What they bought last year, which segments they belong to, and whether we can get them to log in so we can finally identify them.
Meanwhile, they’re on the site right now, searching, comparing products, and giving us all kinds of information about what they’re interested in. But apparently we need to figure out who they are before we can help them.
I get why historical data matters. It can absolutely be useful. But it doesn’t necessarily tell us what someone is trying to accomplish today. And even when we think we’ve got the right segment, how do we know the experience we’re showing them is actually any better?
What happens when an experiment improves the experience for one group of visitors but makes it worse for another? Or when we discover that the same product page is being used by people with completely different needs? That’s where I think the two become much more interesting together.
A winning experiment isn’t necessarily the best experience for everyone
One of the limitations of traditional A/B testing is that we’re often looking for a single winning experience.
We identify a problem, create a variation, and split our traffic to see which performs better. If the variation improves our primary business metric and the evidence is strong enough, we have a reason to consider rolling it out.
That’s a useful process, and it’s one of the reasons I’ve always valued experimentation. It gives us a way to challenge our assumptions instead of relying entirely on opinions about what a better experience should look like.
But the result is generally an average across the population we tested, and averages can hide some pretty interesting differences.
Imagine we’re testing a guided product-finding experience on a category page. The existing page has thousands of products, filters, sorting, and all the usual ways someone can narrow down their options.
We introduce a guided experience that asks visitors a couple of questions to help them find something more relevant. At the end of the experiment, revenue per visitor is essentially flat. Does that mean the experience didn’t work? Maybe. But I’d want to understand a little more before deciding what to do next.
Someone arriving with a specific product in mind may not need any guidance. They want to search, filter, and get to the product as quickly as possible. Someone who’s still exploring might appreciate a little help narrowing thousands of choices into something manageable. Both visitors are using the same category page, but they’re trying to accomplish different things.
It’s entirely possible that the guided experience could help one group while creating unnecessary work for the other. If those effects offset each other, the overall result might look like nothing happened.
Of course, sometimes a flat result really does mean the change didn’t produce a meaningful improvement. I just wouldn’t assume that’s the only explanation without looking more closely.
Where personalization changes the question
This is where personalization starts to change how we think about experimentation. Instead of asking which version of the category page works best for everyone, we can ask whether different groups of visitors would benefit from different experiences.
For me, that starts with understanding what someone is trying to accomplish.
A visitor browsing engagement rings is probably approaching the purchase differently from someone looking for a birthday gift. Even within those groups, someone just starting their research may need different information from someone who’s ready to purchase.
We can use what we know about those visitors, including browsing behavior, entry context, previous activity, or something they’ve explicitly told us, to make a reasonable decision about what experience might help. Then we test that decision.
Going back to the guided product-finding example, we might hypothesize that visitors who are exploring a broad category would benefit from guidance, while those using specific searches or filters would prefer the existing experience.
Rather than showing the guided experience to everyone, we could target the visitors whose behavior suggests they might need it. Now we’re asking a different question. Not whether guided product finding is universally better, but whether providing it to a relevant audience improves the shopping experience and business outcomes.
Targeting a group doesn’t automatically mean we’ve personalized the experience well. We could identify a segment perfectly and still show them something completely useless. The value comes from connecting what we understand about their needs to a change that actually helps them.
That’s something I explored in more detail in Personalization for Anonymous Visitors, because we don’t necessarily need to know who someone is to recognize that they might need a different experience.
Finding a segment that performs better isn’t proof that personalization works
There’s a measurement problem that I think deserves more attention.
Let’s say we run that guided product-finding experiment across the entire category page. The overall result is flat, but when we look at the data afterward, we notice that visitors who spent more time exploring the category appeared to respond positively.
It’s tempting to conclude that we’ve found the audience for the personalization. And we might have. But I’d be careful about making that decision based on a single breakdown of the experiment.
When we start looking at enough segments, we’re eventually going to find differences. Some will be real, some will be random variation, and some may reflect differences in the visitors themselves rather than the treatment.
It also matters when we identify the audience. Looking at characteristics that existed before the visitor was assigned to the experiment is different from grouping people based on something they did after seeing the experience.
For example, comparing visitors who clicked the guided experience with those who didn’t isn’t a valid way to establish the feature’s incremental impact. The clickers may already have been more motivated or more interested in exploring.
What I’d want to do is use that initial observation to develop a more specific hypothesis. If we believe exploratory visitors benefit from guidance, define how we’re going to identify them before the treatment is delivered. Then run another randomized experiment within that eligible audience.
That gives us a much better opportunity to understand whether the personalization itself is creating value.
And if we want to establish that the targeted approach is better than simply giving everyone the same experience, we should compare those approaches directly or design an experiment that can estimate the difference.
How I think about testing personalized experiences
I don’t think personalization needs a completely different philosophy of measurement from traditional experimentation. We still need a clear problem, a hypothesis, a control, and a meaningful way to evaluate the result.
What’s different is how we define the experience and the audience.
In a traditional sitewide experiment, we might ask whether a redesigned product page improves revenue per visitor.
With personalization, we could ask whether visitors showing signs of uncertainty benefit from additional product guidance, or whether visitors who already know what they want perform better with a more direct path to purchase. The audience becomes part of the hypothesis.
We also need to be realistic about the size of those audiences. It’s easy to keep dividing visitors into smaller segments based on product interest, purchase intent, customer history, traffic source, and just about anything else we can measure.
The problem is that every additional layer reduces the number of visitors available for the experiment. Eventually, we’ve created an experience that’s so specific we can’t confidently determine whether it works.
That’s why I generally prefer starting with broader groups that have meaningfully different needs and becoming more specific when there’s evidence to justify it.
Measurement also needs to reflect what we’re trying to accomplish. For ecommerce, revenue per visitor is usually where I start, with conversion rate and average order value helping explain what happened. Depending on the experience, I might also look at product discovery, progression through the journey, or interaction with a particular feature.
But those supporting metrics shouldn’t replace the business outcome. We’ve all seen experiences that generate plenty of interaction without doing much for the business. A recommendation module can get clicks. A quiz can get completed. A shopping assistant can start conversations. None of those activities automatically means we’ve improved the shopping experience.
I also want to understand what might have gotten worse. A treatment could increase conversion while reducing average order value, or encourage more product exploration while slowing down visitors who were already ready to purchase.
That broader view is important whether we’re running a traditional A/B test or evaluating a personalized experience.
I’ve covered the measurement side in more detail in my personalization metrics guide.
Personalization doesn’t mean everyone needs a different experience
One thing I wouldn’t want anyone to take from this is that every experiment should eventually turn into personalization.
Sometimes a better experience really is better for everyone.
If visitors are struggling because the navigation is confusing, important product information is missing, or the checkout process is unnecessarily difficult, I’d rather fix those problems than figure out how to personalize around them.
There’s also the practical side of managing all these experiences.
Every new audience and treatment means more development, QA, analysis, and maintenance. It can also mean different teams competing for the same space on the page, with their own objectives and campaigns. That complexity can add up pretty quickly.
And there’s a danger in assuming that because we can identify different groups, those groups automatically need different treatments. Maybe they don’t.
Sometimes the original experience is already doing a reasonable job, or the difference between the groups isn’t meaningful enough to justify the effort.
That’s where experimentation becomes just as important to personalization as personalization is to experimentation. It helps us determine whether all that extra complexity is actually making things better.
They’re complementary, not competing approaches
I recently came across an interesting article from Spotify Engineering about why Spotify uses separate technology stacks for personalization and experimentation.
Their systems are obviously more sophisticated than what most ecommerce businesses need, but the distinction is useful. Personalization determines which experience to deliver, while experimentation evaluates whether that system improves outcomes.
I think that same principle applies even if you’re using a single platform to do both.
Traditional experimentation has helped us understand how to make websites better. Personalization gives us an opportunity to question whether better should always mean the same thing for everyone.
And combining the two gives us a way to make those decisions based on evidence instead of simply assuming that more relevant experiences will perform better.
I wouldn’t start by trying to personalize every page or build dozens of segments. I’d start with a meaningful visitor problem, determine whether different groups actually need different solutions, and test the experience that seems most likely to help.
Sometimes the result will be a personalized experience. Sometimes it’ll be a better default for everyone. Either way, we’ve learned something useful about how to make the site better for visitors and the business.
If you’re working through where to focus your ecommerce optimization efforts, my Ecommerce Growth Opportunity Sprint helps identify and prioritize opportunities across experience design, experimentation, and personalization using the evidence you already have.