Ecommerce Experimentation Strategy: What to Test and Why

An experimentation backlog can be full while a business is no closer to understanding what needs to improve. There are ideas to review, variations to build, results to report, and always another test waiting to launch. From the outside, it can look like a healthy program.

But I’d want to know what we’re actually learning. Are the experiments helping us understand why visitors struggle, what would make a meaningful difference, and where the business should invest next? Or have we just become very efficient at moving tests through a process?

I’ve spent a lot of time working with experimentation, personalization, and experience design, and I think the distinction matters. Running an A/B test is a method. An experimentation strategy is how we decide which questions deserve a test, what evidence would answer them, and what we’ll do with the result.

That doesn’t mean every experiment needs to be groundbreaking. Sometimes a fairly small change is exactly what’s needed. But it should have a reason to exist beyond filling a slot on the testing calendar.

What an ecommerce experimentation strategy should accomplish

A useful strategy connects visitor problems to business decisions. It helps a team determine what to investigate, where an experiment is appropriate, how to measure it, and what should happen when the results come back.

That sounds straightforward until five teams have different ideas for the same page. Marketing wants an offer, merchandising wants more product visibility, UX wants to simplify the experience, and someone has already promised leadership a new feature. All of those requests may have a reasonable business objective. That doesn’t mean they all deserve to be shown to the same visitor at the same moment.

For me, a good experimentation strategy provides a way to make those trade-offs. It shouldn’t just be a collection of test ideas or a quarterly target for how many experiments we’ll launch.

There’s also a distinction between the strategy for choosing and learning from experiments and the operating model for running an experimentation program. Resourcing, intake, QA, reporting, and executive support matter, but they deserve their own discussion. I’ve covered that side in how to build an experimentation program.

Start with a visitor problem, not a borrowed best practice

I’ve never had much interest in testing something solely because it worked on another website. That result may be worth investigating, but it doesn’t tell us why it worked there, whether the same problem exists on our site, or what we’d learn by copying it.

Imagine an article recommending that every ecommerce PDP needs a sticky add-to-cart button. It sounds sensible. Make the action easier to reach and more people might buy. But before building it, I’d want to know whether visitors are actually having trouble getting back to the purchase controls. Are they spending time deep in product details? Are they returning to the top of the page to act? Is a sticky chat control already competing for the same screen space?

If the evidence suggests that access to the purchase action is a problem, we have something worth testing. If it doesn’t, we’re just testing a feature because someone else made it look like a good idea.

The starting point can be quantitative or qualitative: unusual exit patterns, failed site searches, a product page that attracts paid traffic but leads nowhere useful, customer feedback, session recordings, or previous experiments that raised another question. None of those alone necessarily proves the cause. They’re evidence we can use to develop a hypothesis.

The first question I’d ask is not “What can we test?” It’s “What’s happening that we don’t understand well enough, and what decision are we trying to make?”

Understand what the visitor is trying to do

Ecommerce teams are good at classifying pages. We have PLPs, PDPs, carts, checkout, and homepages. Visitors don’t organize their shopping around our page taxonomy. They may be exploring, comparing alternatives, checking price, trying to understand a product, or returning to buy something they’ve already decided on.

Those are different jobs, even when they happen on the same page.

A product-ad visitor who opens a PDP might be ready to buy that exact product. Or the image might simply have been close enough to earn a click, and now they’re looking for something similar at a different price. A useful experiment might help them explore alternatives instead of assuming the PDP has only one job: sell the product they landed on.

I’d look at the entry point, searches, filters, product views, return behavior, and what visitors have explicitly told us. Historical customer information can help too. But I wouldn’t let a past purchase override everything someone is doing in the current visit.

We also have to be careful about calling every behavior an intent signal. Someone who views ten products might be highly engaged. They might also be struggling to find anything that fits. A low page count could mean low interest, or it could mean someone found exactly what they needed. Page depth can be useful in context; it is not a reliable intent label by itself.

The purpose of looking at behavior is to build a more plausible explanation of the visitor problem. Then we can decide what would distinguish that explanation from the alternatives.

You can borrow an idea, but you can’t borrow the learning. You still have to figure out whether it solves a problem on your own site.

Turn the problem into a testable hypothesis

Once we’ve identified a problem, I’d want the hypothesis to explain more than the proposed design change.

Consider a broad category with hundreds of products. We notice that some visitors repeatedly change filters and move between similar products without progressing. One possibility is that they aren’t confident they’re narrowing the choices in the right way. Another is that the filters are poorly designed. Those lead to different interventions.

A useful hypothesis might be: Visitors entering a broad category without a specific product in mind are having trouble narrowing their options. Offering an optional, short guided path will help more of them reach a suitable product and may improve revenue per visitor.

That gives us a visitor problem, a proposed treatment, a possible audience, and an expected outcome. It also gives us something to challenge. Perhaps visitors don’t want a guided path at all. Perhaps better filters would help everyone. We shouldn’t assume personalization is necessary simply because the audience can be identified.

Before launching, I’d want the team to agree on the essentials:

It doesn’t need to become a twenty-page document. I just don’t want the definition of success changing after someone sees a promising number in the report.

Prioritize experiments by opportunity, not convenience

A common trap is choosing experiments because they’re easy to build, easy to explain, or already have an enthusiastic stakeholder. I get it. Shipping a simple test is satisfying. But effort alone doesn’t tell us whether the experiment is worth running.

I’d weigh a few things together: how significant the visitor problem appears to be, how many relevant visitors encounter it, how strong the evidence is, what the potential business consequence might be, and what it will take to implement and measure the change. I also care about whether answering the question could influence decisions beyond a single component.

A test on a high-traffic page isn’t automatically the best opportunity. A very specific checkout problem affecting fewer visitors may deserve attention first if it prevents purchases. Conversely, a tiny segment with a clever personalized treatment might not generate enough traffic to measure reliably, however interesting the concept sounds.

This is where good ideas have to compete with one another. If everything gets priority, the site eventually becomes a collection of messages and features, each added for a perfectly reasonable reason in isolation. The visitor gets the combined experience, not the separate business cases.

I’d rather run fewer experiments that address meaningful questions than hit a testing-volume target with work nobody would have prioritized otherwise. That doesn’t mean small tests are worthless. It means small should describe the implementation, not the importance of the question.

Design the experiment to answer the actual question

A/B testing gives us a way to compare an experience against an alternative under controlled conditions. But the quality of the conclusion depends on what we randomized, who was eligible, whether the tracking is dependable, and whether the experiment had enough information to distinguish a meaningful effect from noise.

If we want to know whether a guided experience helps exploratory visitors, we need to define that audience before they encounter the treatment, then randomly assign eligible visitors to the appropriate experiences. Simply reporting that people who used the guide purchased more than people who didn’t use it won’t answer the question. The users chose to interact, and that choice may say something about their motivation before the feature helped them at all.

The same caution applies to post-test segment analysis. A flat overall result might hide meaningful differences, but looking through enough audiences will eventually produce interesting-looking numbers. I’d treat those patterns as clues, not proof, unless the experiment was designed and analyzed to support those conclusions.

Sometimes the right next step is a more targeted experiment. Sometimes it’s improving the default experience for everyone. And sometimes it’s accepting that the original idea doesn’t deserve more investment.

I’ve gone deeper into how this works in Personalization vs. A/B Testing: How They Work Together.

Choose metrics that reflect the business outcome

This is an area where experimentation can look successful without the business benefiting much. A new feature gets more clicks. A recommendation module gets more interaction. More visitors use a shopping assistant. Those things may tell us the feature is noticeable or useful to a particular group, but they don’t necessarily tell us whether we improved the overall shopping experience.

For a revenue-focused ecommerce test, I usually start with revenue per visitor (RPV). Conversion rate and average order value (AOV) help explain what happened, and margin may change whether an apparent revenue gain is attractive. For a checkout defect, completion rate might be the more direct measure. For an early product-discovery experience, finding a relevant product may be an important learning measure while downstream revenue remains a guardrail or longer-term outcome.

The right metric depends on the problem and the decision. There’s no reason to force every experiment into exactly the same scorecard. But there is a reason to be skeptical when the only reported success is people interacting with the thing we added.

I explain that in more detail in my guide to revenue per visitor versus conversion rate because those numbers can tell very different stories. A higher purchase rate can still produce less revenue when order value falls. And an experience that gets fewer clicks can still leave the business better off.

A real example: fewer promotional clicks, better business results

In one published promotion experiment, we moved prominent promotions into an on-demand drawer instead of showing them across the top of the experience. Visitors who wanted an offer could still find it, but everyone else had less messaging competing for attention.

Promotional clicks dropped by nearly 60%. If the team’s primary objective had been to get more people to click promotions, we’d have called it a failure. Yet the reported results showed improvements in conversion rate, AOV, and RPV for the general audience.

I find that more useful than a simple argument about whether banners are good or bad. The experiment challenged the assumption that more visibility necessarily creates more value. The published case study also noted a different response among deal-seeking visitors, which is a reason to investigate whether those visitors need a different approach, not proof that every deal seeker should receive another campaign.

The point isn’t to remove promotions from every ecommerce site. It’s to understand what the experience was supposed to accomplish and let the broader result challenge the metric we might otherwise have celebrated.

A flat or negative result can still change the next decision

Not every experiment produces a clean winner, and I don’t think it should have to. If the result is inconclusive, we need to distinguish between an idea that probably has little impact and a test that didn’t provide enough precision to answer the question. Those are not the same conclusion.

If the treatment performs worse, I’d want to understand whether it introduced friction, distracted visitors, or was shown to people who didn’t need it. But I wouldn’t automatically create three new variations just to rescue the original idea. There are times when the useful learning is that we should stop.

And if the overall result is positive, I still want to understand the trade-offs. Did AOV fall? Were there technical issues? Does the result depend on a promotional period or one traffic source? Is the improvement large enough to justify keeping another experience in production?

A test result should influence a decision, whether that’s rollout, iteration, a different experiment, further research, or leaving the current experience alone. Otherwise, we’ve collected another data point without getting much value from it.

Make the learning useful beyond the individual test

The most valuable experimentation programs build on what they’ve already learned. If we’ve tested several ways to help visitors compare products, those results should influence how we design the next PDP or PLP experience. We shouldn’t have to rediscover the same behavior every time a different team requests a change.

I’d want each meaningful experiment to leave behind a record of the visitor problem, the hypothesis, who was included, what changed, what the results do and don’t show, and what decision followed. Not a slide full of green and red arrows, but enough context for someone else to understand the learning later.

That also means sharing results with the people who make decisions about product, merchandising, marketing, and experience design. A finding isn’t particularly useful if it stays inside the experimentation team’s reporting deck.

Getting the resources, ownership, and reporting rhythm right is a separate challenge from choosing good experiments. If that’s what your team is struggling with, see how to build an experimentation program strategy. Here, the point is simpler: the strategy should make the next decision smarter than the last one.

Where I’d start

If I were reviewing an ecommerce experimentation roadmap, I wouldn’t begin by asking how many tests are planned for the next quarter. I’d look for the visitor problems the team believes matter most, what supports those beliefs, and whether the proposed experiments will actually tell us something useful.

I’d start with a small set of opportunities across the journey, select one where the evidence and business consequence are strong enough to justify attention, and define what we need to learn before building anything. Some opportunities will become A/B tests. Others will be obvious fixes, deeper research, or things we decide not to pursue.

That’s the distinction I keep coming back to. An experimentation strategy isn’t there to keep the testing calendar full. It’s there to help us make better decisions about the experience, for visitors and for the business.

If your team has analytics, previous test results, and a backlog of ideas but needs help deciding what deserves attention next, my Ecommerce Growth Opportunity Sprint brings the existing evidence together into a prioritized set of opportunities. The recommendation might be to test, personalize, fix something, or investigate further. It shouldn’t be predetermined by the service or tool we happen to be using.