The hypocritic oath taken by physicians states, “First, do no harm.” The same concept can apply to e-commerce website optimization and experimentation. Read on to find out how.

High win rates aren’t everything

When conducting experiments on e-commerce websites in an effort to boost optimization, the main goal usually is to achieve a high win rate. A win is defined as an experiment that generates a positive revenue uplift and return on investment, or ROI. The more wins we achieve for clients, the higher the win rate.

A high win rate indicates that a high percentage of experiments are resulting in greater revenue and higher ROI. The business is doing the right things and testing in the right areas. It has good ideas and a good understanding of what its customers need.

This is all good, right? Most of the time, yes. But experience has shown that testing for optimization isn’t always all about winning.

A self-fulfilling prophesy

The problem with this approach is that when you only focus on win rate, you only conduct experiments that are likely to win. That’s logical, right? But by avoiding experiments that might not result in a win, you may fail to conduct experiments that could actually be very revealing.

This can result in a site growing stagnant over time because changes are only made when justified by a revenue and ROI boost. Put another way: If you only test for changes that you are confident will win, this becomes a self-fulfilling prophesy. Innovation and growth are stunted and the site can become stale.

Test for minimal negative impact

The best way to avoid this problem is to change your criteria for testing from “winning” to “doing no harm.” In other words, test for minimal negative impact. If a test doesn’t produce a significantly negative impact, go ahead and implement the change.

An online retailer was justifying all website changes based on win rates and revenue uplift. However, research revealed that their customers thought the website was stagnant. So they changed their experiment criteria and started making some changes to the site as long as they didn’t result in a negative impact.

Adapt your metrics

Win rate and revenue uplift are important metrics for website optimization and experimentation, but they can cause harm if relied upon too heavily. As your program matures, adapt your metrics from time to time by testing for minimal negative impact. This can help keep your site fresh and innovative.

Download this PDF here