Your Guide to Running Effective Ecommerce A/B Testing

What is ecommerce A/B testing & why is it important?

A/B testing is a technique in which two versions of the same page are shown to users at random. The goal is to determine which version performs better. Many ecommerce businesses use A/B testing to help optimize their website experience and increase conversion rates. 

Common ecommerce A/B test examples: 

An infinite number of variations can be tested to improve each user's experience. Choosing the right elements is key to a successful experiment. Here are a few common A/B test examples you can try:

• Checkout page - How can you adjust your checkout flow? Try one page vs multi-page checkout.
• Product page - How are you displaying your products? Test different product image sizing.
• CTAs - Does shopper behavior change based on CTA phrasing, color, or placement?
• Promotions - Track how shoppers interact with promotions vs no promotions.

When should you use A/B testing?

It’s a good idea to do some A/B testing if you hear negative comments from your customers. Every ecommerce business gets the dreaded negative review from time to time. People have bad days, and sometimes they just need to vent. But if you start to see a pattern of negative reviews emerging in social comments, those negative reviews could signal that your website could use some improvement. A/B testing will help you discover what needs to change and how to change it. 

What if everything is running smoothly? You don’t need to conduct A/B testing, right? In this case, what you don’t know could be hurting you. Testing your ecommerce website will show you in tangible ways how to improve your UI/UX, increase conversions, and even optimize your shopping cart. 

A/B testing can benefit your business if things aren’t going well and even if they are. It’s almost never the wrong time to check in on user experience. However, it is important to test during times that are typical for the experience you want to capture. In other words, If you want to see how users search for information on a typical day, test during normal traffic times.

Let’s say you want to increase conversions. You could conduct a test of two different versions of a landing page with the CTA button in a different place or you could test a different color CTA button. A/B testing will show you if those small changes make a significant impact. 

As you get ready to conduct your testing, make sure you isolate only one variable to test. If you try to make too many changes between version A and version B, you won’t have clear answers on what to change. 

You can measure results in a variety of ways. For the example above, where you are testing the CTA button to see if it improves conversions, the number of users who click on the button may be all you need. It may also be useful to look at heatmaps to gain insight on user behavior, conduct user test surveys, or use other methods of gathering data.

What are the benefits and challenges of A/B testing?

Understanding the user experience through A/B testing is invaluable in building a better website. You’ll gain insight into your customers’ needs and benefit in many ways:

• Increase engagement

• Boost conversion rates

• Reduce bounce rates

• Improve content

• Target your audience more effectively

• Optimize the customer experience

Set up your tests carefully to make sure you get valuable results. As mentioned above, if you try to test more than one value at a time, your results won’t be clear in most cases. Keep things simple and your testing will be more effective. 

Another challenge you may run into is low volume. The more volume of traffic you have, the more statistically sound your results will be. 

Now that you have an idea of what you can test, when you can test, and the benefits of testing, we’ll cover some best practices to run an effective A/B test on your ecommerce website.

Ecommerce A/B testing best practices:

Have a hypothesis

Think of this test like a science experiment. You'll need to create a hypothesis for what you think will happen when you implement the change. For example, you can A/B test a new payment option for customers to see if it will increase conversion.

Calculate your A/B test sample size

Before starting your A/B test, make sure you have enough traffic to generate a statistically meaningful test in a reasonable amount of time. Your A/B testing tool may offer a sample size calculator. If not, there are many free sample size calculators that you can use. Calculate the estimated amount of traffic needed in the sample size calculator and divide that by your average daily traffic to estimate how long the test will take. 

Run the test when you have normal traffic (or as close as possible)

It's best to not run an A/B test if you're expecting an irregular influx of traffic to your site. Customers may behave differently around Black Friday or major holidays, shopping with increased urgency. During the holiday season, ecommerce websites may receive an increase in visitors that don't normally shop on their site. It's more effective to test regular visitors because what works during the holidays may not work during the rest of the year. 

Set up the test

Once you have enough traffic to complete the test in a reasonable amount of time, set up and start the test. If you have an A/B testing tool, it’s generally pretty easy to set up A/B tests that are straightforward like changing the text for CTAs or testing different images. More advanced tests (e.g., testing a new sight navigation) may require a developer. If you're A/B testing in Google Analytics, Google Optimize is a free tool that works well for most A/B tests.

Check for duplicate transactions

Google Analytics is one of the most popular web analytics tool in the ecommerce industry. However, unless you take the necessary steps to prevent Google Analytics from recording duplicate transactions, your numbers may be inflated. Every time a customer lands on the page that fires the transaction, Google Analytics will record a duplicate transaction for that transaction ID, as well as duplicate revenue.

Duplicate transactions typically happen when users restore the order confirmation page from a closed browser session on their desktop or mobile device. This occurrence can greatly skew data in general, especially for A/B tests. It's best to eliminate duplicate transactions before running an experiment.

More information on how to identify and prevent duplicate transactions in Google Analytics can be found here.

Remove non-human website traffic

Excluding bots and spiders from your ecommerce website is a great way to see how much actual human visitors you're getting. Bot traffic is any non-human traffic coming to your website. Similarly, Spiders are search engine bots that crawl webpages. Once removed you may see a significant decrease in traffic, however, this will make your data much more accurate.

Google analytics makes filtering out known bots simple. In your Admin go to View Settings and select the check box to begin filtering out bot traffic.

Filter out internal traffic

It's important to exclude internal traffic so that you're only measuring the reaction of shoppers on your website. Internal traffic can include call centers, sales reps, or development teams. Due to the pandemic's increase in remote work, you may want to also consider filtering out staff's IP addresses.

To filter out internal traffic in Google Analytics, go to Admin>All Filters then add a new filter. Leave Filter Type as predefined. It should read: Exclude + traffic from the IP addresses + that are equal to. Then, add the internal IP addresses you'd like to exclude. See Google's in depth tutorial. 

Analyze your results

Your A/B testing tool should indicate if your test has reached a statistically significant conclusion, based on all visitors during the experiment. It can be very useful to drill down to see the results by sub-groups that are important to your business. Try viewing the test results by New and Returning Visitors, by device category, and for your key acquisition channels. You can also use you results to run a more advanced A/B test or try a similar test on other pages or CTAs. 

Conclusion

We know how important A/B testing is for your ecommerce business, that’s why when we deploy our Shopping Guarantee, we conduct A/B testing to make sure we find optimal placement and messaging. Thanks to the extensive testing we have done, we are confident that our Shopping Guarantee can increase conversions on your ecommerce website. 

If executed properly, ecommerce A/B testing can be a powerful tool to drive online growth. A/B testing should be done regularly so that you can always give your customers an optimal experience, ultimately increasing conversions.