A or B: What to Do if Both Fail?

by TestFortExpert on 04/24/2014

abtesting Testers at most QA companies probably know that situation when an A/B test has been successfully completed, but in a few days it occurs that it hasn’t raised your revenue as was expected.

This situation is not the time to get distressed.

Instead, you have to better perceive the A/B testing intricacies and explore what exactly you did wrong.

Below are some points which can help you understand what you did wrong and how to fix it.

The test was set up at the funnel’s beginning while your additional traffic comes at the middle

For example, you could forget there was additional traffic that comes directly in the product page and this traffic has not been affected by your change to the page ‘Product category’. In the result your revenue appeared lower than your A/B test reported.

What to do:

Check additional streams of traffic which could affect the test results. Then exclude those from your test calculations and manage these expectations based on the traffic to undergo the test. This can be done in ‘Custom Variables’ of your Google Analytics account.

You didn’t wait for enough statistical confidence

When you stop the test before it reaches a statistical confidence, the data will be unreliable. According to the standards, you should run the test till it gets at least 95% confidence level .

What to do:

Thanks to A/B testing tools widely available, the confidence level is easily calculated and available even without you getting involved in the calculations.

You didn’t care for sample size much

If you implement the changes to your site based on a split test of insufficient sample size, the results will be surprising. It’s likely that your 60% revenue jump reported based on 50 test participants is not going to match the real results.

What to do:

Use a split duration calculator to plug in your site values and calculate the test’s sample size, or the visitors’ number you need to get conclusive split test results.

You focused more on psychological tactics than customer needs

The conversion rate optimization’s primary aim is eliminating unsupervised thinking as well as helping consumers take the right decision providing them with value.

What to do:

If you understand the needs of your customers and try to make your website address them, it will likely give you a larger revenue lift than your psychological tactics.

You didn’t track the right conversion goal

It’s okay to optimize your site for one goal only when its focus is narrow. For example, your sale goal will trump all other goals in case with an online shop. But if the website has many conversion goals, failing to track all of them can make you draw some false conclusions from the A/B tests.

What to do:

Make sure to tracking all important key performance indicators. You may sometimes see the primary test goal has the positive result, though it doesn’t favorably reflect on another significant metric of your site. In this case, you should decide which metric’s impact is better for your business indeed and go on with respective changes.

Be careful to mind all the obstacles above and get ready to eliminate them if necessary. It’s recommended to involve an experienced QA company in your A/B testing efforts.

We Work With

Having one outside team deal with every aspect of quality assurance on your software project saves you time and money on creating an in-house QA department. We have dedicated testing engineers with years of experience, and here is what they can help you with.

Software is everywhere around us, and it’s essential for your testing team to be familiar with all the various types and platforms software can come with. In 21+ years, our QA team has tested every type of software there is, and here are some of their specialties.

There are dozens of different types of testing, but it takes a team of experts to know which ones are relevant to your software project and how to include them in the testing strategy the right way. These are just some of the testing types our QA engineers excel in.

The success of a software project depends, among other things, on whether it’s the right fit for the industry it’s in. And that is true not just for the development stage, but also for QA. Different industry have different software requirements, and our team knows all about them.

Full-Cycle Services

By Platform

By Type

Industries