Regression testing is an integral part of the software development and testing process that teams absolutely cannot afford to overlook. Regression testing is used after every change in the code to make sure that all of the existing functionality is still there and that the software product was not negatively affected by the change. So it’s not really a question of whether you need regression testing on your project, but a question of how to do it with maximum efficiency.
Teams often come to realize that regression after each iteration takes too long, and the functionality being verified often has no changes. Consequently, the possibility of defects is quite low. As a natural result, we arrive at an idea to automate all regression scenarios and refuse manual regression testing. But is this really the most sensible idea, and does manual regression testing still have a place in the modern software QA landscape? This is the subject we’ll investigate in today’s article.
Automated vs. manual testing
Both automated and manual testing is necessary to conduct whenever you want your digital product to be of great quality. However, when it comes exactly to regression testing, the choice of solution highly depends on the type of regressions. For instance, partial and unit regressions can be tested manually, while complete regression is better to be tested automatically.
Manual regression testing has the main indisputable advantage, which is better code coverage. In that aspect, it is a bit more reliable than automated regression testing. Still, for bigger projects with repetitive test case scenarios — automation is a must. It saves dozens of hours and turns out to be a more cost-effective way of testing projects with plenty of changes implemented.
The advantages of automated testing here are obvious:
- automated tests are faster than a human;
- automated tests can be performed at any time;
- automated tests are very precise;
- automation can be used in almost all testing processes;
- automatic creation of reports is possible.
Limitations of automated regression testing
While the benefits of automated regression testing are undisputable, regression test automation has some pitfalls and limitations that are not always considered before the start of a project. Here are the possible issues of automated regression testing methods:
- An automated regression testing project takes a while to set up;
- Only engineers with specific expertise can do automation, and they usually cost more to hire;
- Running the entire automation suite may be unnecessary with partial regression.
If you want to make your regression testing — manual or automated — project successful, there is a mandatory preparation process. Before you start to automate regression testing, it is necessary to solve a few questions:
- What functionality is to be covered by the tests;
- What the autotests architecture is going to be like;
- The choice of a tool for regression test automation.
Our experience with automated regression testing
Let us take a closer look at automating regression testing in practice and look at a real-life regression testing example from our own projects. At the stage of switching from manual testing to automated tests, our company became involved in a project where we had to test the registration of two types of accounts for the site on 43 domains depending on the country.
The sheer volume of data and possible environment combinations was almost too much to handle. Things were made even more difficult by the client, who didn’t care much about the quality of the information displayed. The requirements were as follows:
- accounts are created without any fatal errors;
- input information coincides with the values that are displayed in the settings after creation.
After a couple of test runs, we realized that we’ve had enough. At that moment, we could only see that the number of registration fields is different for some countries. Coincidentally, one of the project team members also had some coding experience in C#. Another regression was close at hand and we did not have time to hesitate. After many questions, tries, and errors, a simple navigation test was born.
We discovered locators and Selenium web drivers — and lo and behold, all the domains used the same locators. The team had a little left to do — to run the last manual regression on all domains and fill the table with the fields and corresponding domains. Another round of manual regression — a long process with completing the table — and it seems like it is all over.
We enjoyed every other regression. It was almost as if we could watch some TV series while regression was running. By the way, it was just as long as one episode. While we used to spend about 4-5 hours on registration testing on 43 domains, automated regression testing took us 40-45 minutes, provided non-optimal code, no report system, and no proper exception handlers.
Considering the above said, we can make conclusion #1: to see the advantages of test automation, we should start with the most repeated tests, which are usually positive, but they need to be run after each release.
This continued for several releases until the company decided to automate most of the functionality officially. Since the client was not well-versed in automation tests, we decided to automate any and all that was within our knowledge of test automation.
As a result, we covered almost the entire system with autotests. All would be fine, but after any fixes, some tests started to fail. Mostly, it happened because of long scripts that contain a lot of steps:
- Test could fail even at the second step, and as the first part of the test usually brings the system to a certain state, running the whole test had no sense;
- Long scenarios contain a lot of code; as a result, repeating the first step might lead to the failure of many tests.
- Because of the code repetition, autotests check the same point many times; this results in a needless increase in the test duration.
On that basis, we can make conclusion #2: too-long scenarios should be run manually, as a lot of time is spent on writing them and they can often fail because of some minor bugs or errors in the second step.
Let us consider another very interesting situation. The client ordered autotests as usual and asked to write a couple of examples. They approved the idea, and in several days we added test automation to the plan. But our programmers did not meet the deadline, and eventually, we had to write autotests based on the current stage. With a delay of a couple of days, the programmers had a release, but more than half of the autotests failed.
Conclusion #3: never start writing automated tests without a stable build.
Only a short time ago, we had one more notable situation. We were handed a new project for automating website testing and have completed the entire scope of testing quickly. Everything went better than ever. A few weeks later, the customer sent a letter complaining that tests did not work. As it turned out, they started a site redesign and all the tests had failed. It taught us to do stable tests.
Therefore, conclusion #4 is: when possible, use the locators so that the elements can be found after the slightest changes.
Bottom line
Automated regression testing can take the project to the next level in terms of quality and efficiency. However, the advantages of manual testing are clear — it doesn’t depend on anything and can be done all the time. Abandoning manual testing will do nothing good. Automated and manual testing are interrelated and complementary testing methods that each has pros and cons. Before thinking of regression scenarios automation, consider time expenditures for both writing tests and their support. Also, pay attention that manual specialists are usually paid less than those who have skills in test automation.
So, as always, the whole issue is mostly about money. If automation of regression testing scenarios saves money without losing the quality of the product, you can expand the range of automated scenarios until the product quality is high and you save costs. The way of balancing here is up to you to decide and depends on personal experience and preferences. You can also cut costs on a project by entrusting some or all of your testing needs to a trusted provider — that way, you don’t need to spend any money or time on hiring, onboarding, or providing office space for new employees.
