A or B: What to Do if Both Fail?

You are here

Testers at most QA companies probably know that situation when an A/B test has been successfully completed, but in a few days it occurs that it hasn’t raised your revenue as was expected.

This situation is not the time to get distressed.

Instead, you have to better perceive the A/B testing intricacies and explore what exactly you did wrong.

Below are some points which can help you understand what you did wrong and how to fix it.

  1. The test was set up at the funnel’s beginning while your additional traffic comes at the middle

For example, you could forget there was additional traffic that comes directly in the product page and this traffic has not been affected by your change to the page ‘Product category’. In the result your revenue appeared lower than your A/B test reported.

What to do:

Check additional streams of traffic which could affect the test results. Then exclude those from your test calculations and manage these expectations based on the traffic to undergo the test. This can be done in ‘Custom Variables’ of your Google Analytics account.

  1. You didn’t wait for enough statistical confidence

When you stop the test before it reaches a statistical confidence, the data will be unreliable. According to the standards, you should run the test till it gets at least 95% confidence level .

What to do:

Thanks to A/B testing tools widely available, the confidence level is easily calculated and available even without you getting involved in the calculations.

  1. You didn’t care for sample size much

If you implement the changes to your site based on a split test of insufficient sample size, the results will be surprising. It’s likely that your 60% revenue jump reported based on 50 test participants is not going to match the real results.

What to do:

Use a split duration calculator to plug in your site values and calculate the test’s sample size, or the visitors’ number you need to get conclusive split test results.

  1. You focused more on psychological tactics than customer needs

The conversion rate optimization’s primary aim is eliminating unsupervised thinking as well as helping consumers take the right decision providing them with value.

What to do:

If you understand the needs of your customers and try to make your website address them, it will likely give you a larger revenue lift than your psychological tactics.

  1. You didn’t track the right conversion goal

It’s okay to optimize your site for one goal only when its focus is narrow. For example, your sale goal will trump all other goals in case with an online shop. But if the website has many conversion goals, failing to track all of them can make you draw some false conclusions from the A/B tests.

What to do:

Make sure to tracking all important key performance indicators. You may sometimes see the primary test goal has the positive result, though it doesn’t favorably reflect on another significant metric of your site. In this case, you should decide which metric’s impact is better for your business indeed and go on with respective changes.

Be careful to mind all the obstacles above and get ready to eliminate them if necessary. It’s recommended to involve an experienced QA company in your A/B testing efforts.