The Science of testing

You are here

It’s a real science!

This article is not about bugs or methodologies. This article may not be as deep and tech savvy as many others. This article was not meant to be. We will be discussing tests any business is to conduct to do what it was destined: earn more money. Despite your software requires proper tests to be computable on the market you still need to sell it to earn money. You may be selling milk for all’s sake and you are testing it’s qualities as well to make sure it is safe. And you still need to sell it. Why not test that? Sales? That’s what A/B testing is created for as many other useful things you may implement.

The thing is that every marketer today is well aware of A/B like tests even without any tech skills whatsoever (which is true to most cases) and is capable of running them in a well-designed interface where all is simple. That is the problem. Zero skill and button clicking. A great example of non-testers doing testing and failing most of the time. In reality few marketers can actually design such tests. Clicking on the ‘start’ button is their limit. Surely they are professionals nonetheless in whatever considers marketing, yet proper testing is a science. And this approach is leading to nothing but false-positive results or no results at all.

But it is a great thing people are nowadays realizing the value of testing and are applying it. It’s not that hard to learn science if there is such a desire. Here are several tips that will improve your businesses marketing strategy and make you better testers.


DOE or Design of Experiments is some form of statistics you have applied to planning, executing or analysis of your series of tests. This allows deeper understanding of signals in a really complex test environment. Some of DOE major concepts are:

  • Random participants should be assigned between control groups and tests
  • Testing is to be conducted against control
  • Only repeated tests ensure results and accuracy

Thus if a test, or even an experiment (its science we are talking about) is designed appropriately you will be receiving accurate results more likely. One of truly vital components of a properly designed experiment is a vast sample size. Let’s get closer to it?

Size does matter

It’s all about statistics. If your testing sample is too small you are likely to end up with false or false-positive results. Your sample are people when A/B is considered. If the crowd is appropriate you will have several opinions that may be gathered into statistics. If there are two people voting for the same thing that is but a coincidence. How do you know the right amount of your sample? In the world of science there are answers to almost all questions.

There are tests to test tests. Some of them may determine whether your sample is of appropriate size. All that effort is expensive? Quality always is. But no need to get desperate when you are in possession of a rather slim budget. Go for A/A tests in parallel with A/B tests. If the result is similar you may presume you are facing a false-positive. Easy.

Don’t be carried away with fast results

It’s internet we are talking about. People love new things. That may effect on your tests from the start. Some people love things the way they used to be. That may also effect. What do these kinds of people have in common? Well, new things are transforming into old things fast. Especially in th web. So if tests are being run for longer you will possess better results that were affected by actual preferences rather than reaction to change.

That is science and rather than testing just because the process is in trend today. And this will help your business a lot.