A/B testing is the preferred testing method for e-commerce websites that want to optimise conversions. During the past eight years, we have performed more than 13,000 A/B tests for our clients.

However, there are times when conducting an A/B test isn’t practical or feasible. This raises the question: What kinds of alternatives are there for running experiments when you can’t perform A/B tests?

When A/B testing isn’t feasible

There are a few scenarios where it’s not practical to conduct A/B tests. The first is when time is of the essence. Conducting an A/B test can take weeks — from producing designs to building the tests and then up to four weeks to conduct the experiments. You don’t always have this kind of time to devote to your testing.

Another scenario is when there is a low volume of transactions. This is common with luxury brand sites that have fewer high-value transactions, or smaller business that are still growing. We would typically look for around 1,000 transactions per month to obtain statistically relevant results. Yet another scenario is when site security is crucial, such as at the checkout.

User testing and prototyping

The best way to experiment when you can’t run A/B tests is to perform user testing and prototyping. This starts with performing research about a potential website problem or issue and then having a “how might we” session with team members to get ideas for how you might solve it. For example, maybe you want to increase how many users progress from basket to checkout or get more shoppers to add products to basket. Based on the ideas shared by team members, you will create an interactive prototype for user testing.

In user testing we recruit participants that match the target customer’s profile and observe how users interact with the prototype and the control. This gives us the opportunity to ask them for qualitative feedback while also taking quantitative measurements from their actions. For example, how many clicks did it take for users to complete the task set out in the prototype, or how many clicked the wrong element or triggered an error? What kinds of problems (if any) did they encounter in the process of trying to complete a purchase? Comparing the performance from the control to the prototype, we can measure if we are creating the improvement we wanted.

With user testing and prototyping, you can shrink the experimentation time from weeks to days, depending on the complexity. You also don’t need a large volume of transactions to perform these experiments, nor do you jeopardize site security.

Here are a few case studies of e-commerce businesses for whom we did this kind of testing to optimise conversions.

UK manufacturing company

A manufacturer in the UK wanted to identify friction points in the customer journey on their website — especially why customers hesitated to finalize a purchase. Research indicated that one hindrance was the login process, which made it difficult for customers to complete purchases.

Normally we would approach this using normal A/B testing, but they were worried about A/B testing in the checkout and how heavily they could be impacted by any changes that weren’t immediately positive.

We created a prototype in figma and redesigning the initial steps of the checkout. Next, we recruited in market participants and got them to go through the process of checking out using the new prototype. Then we measured how long it took customers to complete the checkout, where they hesitated and how many who started were able to complete the task. We kept tweaking the prototype until we had a version that significantly increased the conversion rate and improved the customer journey.

Once we were happy, we handed the designs to the client who was able to brief into their development team, confident the new version would deliver an improvement.

Bicycle retailer

A worldwide leader in the manufacturing and sale of foldable bicycles discovered that its customers were struggling to decide which bike was best for them based on the online descriptions and model names. The naming convention was not intuitive and customers were having a hard time picking from the wide variety, which included 36 different bicycle variations.

To simplify things, we created a prototype that narrowed it down to three main categories of bikes with just three models under each, while also simplifying the naming convention. We then ran 18 different user testing sessions across several different countries, comparing the prototype site to the existing control site and collecting qualitative customer feedback and quantitative data from feedback surveys along the way.

The sentiment in choosing between bicycle models was 94% positive on the prototype site, compared to just 41% positive on the control site. Eighty-nine percent of customers on the prototype site felt that it was easy to compare models, compared to just 41% on the control site. And 83% of customers on the prototype site believed that the reduced range of models was sufficient, compared to 76% who felt this way on the control site.

Home furnishings retailer

High-end home furnishings tend to be big-ticket but low volume for e-commerce retailers, which can make A/B testing difficult due to small sample sizes and the lack of statistically relevant results. A home furnishing retailer in the UK discovered that customers were having a hard time finding the product information they needed on the product details page (PDP).

Based on this, we redesigned the PDP to emphasize the information customers said was most important to them and make it more visually prominent. Then we did 50-50 user testing between the prototype and existing control sites and gathered qualitative customer feedback on the prototype site.

We iterated on the designs three times, rerunning the user testing session each time, until we reached a solution the client was happy with. The client took the final designs and updated their product pages across the site to match.

Optimising without A/B testing

Just because A/B testing isn’t a viable option doesn’t mean you can’t run experiments to optimise your e-commerce website. User testing and prototyping can achieve the same results, helping you optimise your site and improve conversion performance.

Download this PDF here