A or B: What to Do if Both Fail?

TestFortExpert by TestFortExpert on 04/24/2014

A or B: What to Do if Both Fail?

abtestingTesters at most QA companies probably know that situation when an A/B test has been successfully completed, but in a few days it occurs that it hasn’t raised your revenue as was expected.

This situation is not the time to get distressed.

Instead, you have to better perceive the A/B testing intricacies and explore what exactly you did wrong.

Below are some points which can help you understand what you did wrong and how to fix it.

  1. The test was set up at the funnel’s beginning while your additional traffic comes at the middle

For example, you could forget there was additional traffic that comes directly in the product page and this traffic has not been affected by your change to the page ‘Product category’. In the result your revenue appeared lower than your A/B test reported.

What to do:

Check additional streams of traffic which could affect the test results. Then exclude those from your test calculations and manage these expectations based on the traffic to undergo the test. This can be done in ‘Custom Variables’ of your Google Analytics account.

  1. You didn’t wait for enough statistical confidence

When you stop the test before it reaches a statistical confidence, the data will be unreliable. According to the standards, you should run the test till it gets at least 95% confidence level .

What to do:

Thanks to A/B testing tools widely available, the confidence level is easily calculated and available even without you getting involved in the calculations.

  1. You didn’t care for sample size much

If you implement the changes to your site based on a split test of insufficient sample size, the results will be surprising. It’s likely that your 60% revenue jump reported based on 50 test participants is not going to match the real results.

What to do:

Use a split duration calculator to plug in your site values and calculate the test’s sample size, or the visitors’ number you need to get conclusive split test results.

  1. You focused more on psychological tactics than customer needs

The conversion rate optimization’s primary aim is eliminating unsupervised thinking as well as helping consumers take the right decision providing them with value.

What to do:

If you understand the needs of your customers and try to make your website address them, it will likely give you a larger revenue lift than your psychological tactics.

  1. You didn’t track the right conversion goal

It’s okay to optimize your site for one goal only when its focus is narrow. For example, your sale goal will trump all other goals in case with an online shop. But if the website has many conversion goals, failing to track all of them can make you draw some false conclusions from the A/B tests.

What to do:

Make sure to tracking all important key performance indicators. You may sometimes see the primary test goal has the positive result, though it doesn’t favorably reflect on another significant metric of your site. In this case, you should decide which metric’s impact is better for your business indeed and go on with respective changes.

Be careful to mind all the obstacles above and get ready to eliminate them if necessary. It’s recommended to involve an experienced QA company in your A/B testing efforts.

We Work With

Having one outside team deal with every aspect of quality assurance on your software project saves you time and money on creating an in-house QA department. We have dedicated testing engineers with years of experience, and here is what they can help you with.

Software is everywhere around us, and it’s essential for your testing team to be familiar with all the various types and platforms software can come with. In 21+ years, our QA team has tested every type of software there is, and here are some of their specialties.

There are dozens of different types of testing, but it takes a team of experts to know which ones are relevant to your software project and how to include them in the testing strategy the right way. These are just some of the testing types our QA engineers excel in.

The success of a software project depends, among other things, on whether it’s the right fit for the industry it’s in. And that is true not just for the development stage, but also for QA. Different industry have different software requirements, and our team knows all about them.

Icon Manual Testing

Maximum precision and attention to detail for a spotless result.

Icon Testing Automation

We’ll automate thousands of tests for all-encompassing coverage.

Icon Testing Outsourcing

Outsource your testing needs to a team of experts with relevant skills.

Icon Testing Consulting

Overhaul your QA processes to achieve even more testing efficiency.

Icon QA

Thorough Quality Assurance for a project of any scale or complexity.

Icon API Testing

Verify the correct operation of as many APIs as your project needs.

Icon IoT Testing

Stay ahead of the growing Internet of Things market with timely testing.

Icon Web App Testing

Reach out to even more customers with a high-quality web application.

Icon Mobile App Testing

Help users fall in love with your mobile app with our texting expertise.


Make sure your CRM/ERP system meets the needs of the stakeholders.

Icon Desktop Application Testing

We’ll check the stability, compatibility, and more of your desktop solution.

Icon Functional Testing

Is your app doing everything it’s supposed to? We’ll help you find out!

Icon Compatibility

Check how your solution works on different devices, platforms, and more.

Icon Usability

Find out if your software solution provides an engaging user experience.

Icon UI

Make sure your application’s UI logic works for all categories of users.

Icon Regression

We’ll verify the integrity of your application after recent code changes.

Icon Online Streaming & Entertainment

Stay on top of the media industry with a technically flawless solution.

Icon eCommerce & Retail

Does your store meet customer needs? We’ll help you know for sure!

Icon HR & Recruiting

Streamline HR processes with a solution that works like a clock

Icon Healthcare

Test the functionality, stability, scalability of your app and more.

Icon Fintech & Banking

Give your users what they want: a powerful, secure fintech product.

We use cookies to ensure your best experience. By continuing to browse this site, you accept the use of cookies and "third-party" cookies. For more information or to refuse consent to some cookies, please see our Privacy Policy and Cookie Policy