Quality Assurance Automation for A Virtual Training Lab Service

Challenge

The client approached us to perform quality assurance on the product over six years ago, when the solution had a bare minimum of functionality and was in its early stages. Our team was tasked with automating the tests from the ground up, which is a challenge of its own, as there were barely any contributions from the manual QA team to base our work upon. Other key objectives and challenges for this project included:

Achieving the most practical use of the client’s resources while maintaining the highest quality of testing.

• Developing a comprehensive testing strategy from scratch, given that there was little to no documentation at the start of the project;
• Creating in-depth integration tests to check how the core solution interacts with additional products it’s paired with;
• Finding an effective way to work with large databases that are constantly getting bigger;
• Checking how accurately the application calculates the time and money spent on cloud services within one event;
• Ensuring smooth communication and collaboration between our team, the manual QA team, and the development team, which are all located in different parts of the world;
• Implementing various complex technologies, mainly cloud-related, to fulfill the requirements of the stakeholders;
• Setting up an effective CI/CD process despite the ever-growing number of tests;
• Verifying the correct localization of the project, which is available in 5+ languages;

Solutions

After analyzing the existing state of the application and setting the groundwork for the collaboration with the manual QA team, we created a test plan that we have since updated and enhanced countless times to meet the changing requirements of the product. At different stages of the project, our team consisted of:

• 1 Automation QA Lead
• 3 Automation QAs

From the very beginning, our team focused on end-to-end testing to deliver the most consistent and reliable results. Besides that, we used:

• Smoke testing
• Regression testing
• UI testing
• Functional testing
• Localization testing
• Performance and load testing
• Security testing

Over the course of the project, we have covered over 77% of functionality tests, up to 92% of API tests, and a large chunk of performance, load, and security tests. Overall, we have automated over 1,100 tests, and that number continues to grow. Through daily meetings and effective communication with the development and manual QA teams, we have ensured seamless collaboration across all departments.

In the early stages of the product’s development, the team on the client’s side was dedicating most of their time and effort to the functionality of the product, while the performance of the app was often treated as an afterthought. After our team discovered significant performance issues, we updated the testing strategy to include this aspect as well. As a result of our input, the development team has been successfully optimizing the application’s performance for over a year now.

As soon as the application’s user base started growing, we began testing not only the core product, but also the features and modules of the product tailored to the needs of the end customers. To date, we have tested over 20 of those solutions and continue to provide ongoing support.

Results

Our work on this project has been continuing for over 6 years and is still going strong, so much so that we are currently the only testing team working on the project. Thanks to the continuous use of testing automation, we have been able to significantly increase the efficiency of testing and locate many of the bugs missed by the manual QA team. The feedback from the client has been largely positive, and our level of responsibility has grown accordingly.

25%

User engagement increased by

92%

Automation coverage reached

5 to 3.5 seconds

App load time dropped from

37%

Support tickets decreased by