If you want to mature your optimisation program, you need to understand the different types of tests that can be run and when to use each kind of test. This is key to ensuring data integrity and increasing testing velocity and complexity, leading to better decisions for your e-commerce website.

There are a number of different types of optimisation tests and experiments you can run, including the following:

Validation tests

Running validation tests is crucial to verifying the integrity and reliability of your optimisation program. These tests mitigate the risk of unintended consequences or negative side effects that may arise from optimisation changes and safeguard against potential damage to the user experience or business metrics. Incorporating validation tests into your optimisation program builds confidence in the effectiveness and safety of implemented changes, leading to more informed decision-making and sustainable improvements.

There are two main types of validation tests: A/A tests and blackout tests.

1. A/A tests

These tests involve splitting traffic within the same user group but making no changes to the control or the variant. A/A tests are used to validate the accuracy of targeting conditions between the test and the control, or to verify that a testing tool is fairly splitting traffic groups.

You should always run an A/A test when onboarding a new testing tool to make sure that traffic is accurately split, there is parity within your baseline metrics, and data is passing correctly to your analytics tools. You can also run an A/A test before performing targeted tests. If you’re targeting customers who possess a specific set of attributes, you need to make sure they’re the ones who will be seeing your test in order to fully validate your hypothesis and ensure accuracy of the results. Running an A/A test will validate that the correct customers are falling in to your test pots.

2. Blackout tests

With blackout testing, you will retest a previous winning or losing test at a later stage to understand whether the performance increase or decrease still persists, as well as whether seasonality or external influences could have been a contributing factor to the results.

The performance of a test can often be related to people seeing something new on your site, and initially you might see a big spike in metrics that will decrease over time. This is especially useful to re-test if you are reporting the ongoing incremental benefit of a change. Blackout tests can also help you to understand if seasonality was a contributing factor to a previous test’s performance. For example, maybe the initial test was run close to the Black Friday period when traffic wasn’t your typical customer base. A blackout test will help you determine if those test results were accurate during a period when you know your typical customers are visiting the site.

Traditional tests

Every optimisation program begins with your basic A/B tests where you’re simply testing a new change, usually to your full traffic base. Ideally, these tests should not incorporate multiple changes because this makes analysing performance much more difficult, especially if things go wrong.

There are two main types of traditional tests: A/B tests and A/B/N tests:

1. A/B tests

These tests involve using a control group and a test group to understand the impact of making a change to a single variable, such as redesigning the size guide CTA to increase its prominence. Use A/B tests when only testing a single variable or when testing a single part of the site and there’s only one design to test against the control.

2. A/B/N tests

These tests involve using a control group and more than one test group to understand the impact of making a change to a single variable. Use A/B/N tests when you’re testing a single variable or when testing a single part of the site and there are multiple designs to test against the control, or designs with differing levels of change. For example, you could use an A/B/N test if you were redesigning the size guide on the product page and had different approaches on how best to tackle the change.

Complex tests

When your optimisation program is looking to mature, more complex testing can help increase velocity, mitigate the risk of larger scale changes and offer your customers experiences that are truly tailored to their specific needs. Some of these tests require robust data collection and analysis capabilities to understand user behaviour, preferences and characteristics. With this data, segmentation strategies can be developed to group users based on common attributes or behaviours.

1. Multivariant tests (MVTs)

With MVTs, you can simultaneously test multiple variations of different elements within a single webpage or user experience and get answers faster than with an A/B test. Unlike A/B tests, which compare two distinct versions of a webpage or experience, MVTs allow you to assess the combined impact of several different changes and ultimately shows you what the best combination is.

For example, one of our clients wanted to drive more customers into their custom fit journey by increasing the prominence of this throughout the site. We tested placements across several different areas of the site such as the homepage, PDP, PLP and sitewide banners. You might assume that having placements in all of these areas at the same time would best increase visibility and thereby have the highest impact to drive customers into this journey. However, when we ran a multivariant test, we discovered that showing all four placements actually had a negative effect on conversion. Instead, two placements was the optimal combination for increasing conversion.

2. Targeted tests

These tests use pre-defined profiled attributes to present a different shopping experience to a specific set of users.

One of our clients has a strong presence across Europe so there’s a broad country mix in their traffic. We ran a test on the PDP to show different messages about delivery costs and time scales to each country and saw that there wasn’t a one size fits all approach, with some users converting at a higher rate when they did or didn’t see delivery costs or delivery time scale estimates.

3. Personalisation tests

These tests use customers’ personal data to create a unique and individual user experience. You could tailor different versions of a webpage or user experience to specific user segments based on their unique characteristics, behaviours or preferences. For example, personalised variations might show different product recommendations based on a user’s browsing history, demographics or past purchase behaviour.

One of our clients offers a credit account so customers can finance their purchases. We utilised data for customers who had an available balance within their account and increased visibility of their available spend throughout their shopping journey to encourage them to purchase, especially when they were at risk of abandoning the shopping basket.

Choose the right type of test

Testing is essential to ensuring the changes that you’re making to your website are not only helping to boost sales and revenue but also providing your customers with the best and most optimal experience. The right type of test to use will depend on a wide range of different variables, including the maturity of your optimisation program and the data you have available. An optimisation agency can help you determine which type of testing is appropriate for any given situation and how you can maximise your data.

Download this PDF here