Performance testing is impossible to do without proper automated testing tools. Depending on your performance testing needs, experience, and budget, there are numerous performance testing tools and frameworks available on the market. And even though there are lots of pricey options out there, the majority of tools quality assurance professionals use today are open-source and can be accessed absolutely for free.
Here’s a list of popular performance testing tools with their features, advantages, and possible drawbacks explained.
Probably the most popular of all performance testing tools, Apache JMeter is an open-source Java application designed to measure the performance of web applications (and a variety of other kinds of systems since recent updates). It is used to simulate heavy loads on servers, networks, or system components to test and analyze their ability to perform well under different load types.
Some of its biggest advantages are:
- the ability to performance test a lot of different static and dynamic resources out of the box, including Java Objects, HTTP/HTTPS, SOAP and REST web services, FTP, databases, queries, and mail protocols
- a robust Integrated Development Environment that simplifies planning, recording, building, and debugging your performance tests
- a flexible core that can be easily extended with a huge variety of readily available free plugins provided by the community
Although JMeter is one of the most popular performance testing choices, it still does have some shortcomings:
- even though the UI is pretty intuitive, the learning curve to use JMeter effectively is still quite steep
- it’s pretty resource heavy, and high memory consumption when executing large JMeter tests may cause errors
JMeter, the tool described above, was pretty much built to serve as an open-source alternative to LoadRunner from Micro Focus, a highly sophisticated commercial tool for measuring system behavior and performance under load. While JMeter is primarily focused on performance testing web applications and services, LoadRunner features broader options, including testing of ERP software and legacy systems.
LoadRunner’s most distinctive features are:
- native monitors which means there’s no need to install the tool on the server under test
- excellent UI that enables you to monitor performance metrics in the form of concise, colored charts
- ease of creating virtual users and no limitations on the number of virtual users when running the test cases
Except for the most obvious downside (which is the ridiculously high cost of a license), LoadRunner:
- requires a ton of resources to work efficiently and avoid crashing
- arranging scripts takes up a lot of time and resources, especially in the case of frequent changes to the build of your software solution
Gatling is an open-source performance testing framework built on top of a toolkit called Akka. It is a highly capable tool that focuses on stress testing. It’s biggest advantage over other performance testing tools is the advanced architecture with high performance and maintainability. Virtual users in Gatling are messages not threads. This gives Gatling a huge scaling advantage over thread-based tools such as JMeter and LoadRunner. The message-driven architecture allows you to run thousands of virtual users on a single machine and scale them up without facing any resource-related issues.
Among other benefits are:
- great HTTP Protocol support
- GUI recorder that converts captured traffic into Gatling scenarios that can be used to simplify script creation for a new application under test
- test scenarios are easier to maintain as you can keep your code in the VCS (version control system) alongside the production code; this also enables you to run tests as part of CI (continuous integration)
The biggest downsides of Gatling compared to JMeter and LoadRunner are:
- supports only a handful of protocols out of the box such as HTTP, WebSockets, Server-sent events, and JMS
- the default metrics you get are quite limited; for more you need to use third-party plugins and extensions
Locust is an open-source distributed user load testing tool that focuses on measuring response times on websites and applications. It’s based on Python and features a simple UI which makes it very easy for you to check how many concurrent users your system can effectively handle. Locust creators have also moved away from the traditional thread-based approach and built their tool using an event-based architecture that uses two-three times less resources compared to JMeter and LoadRunner.
Some of the awesome things you can do with Locust are:
- customize the behavior of each of your virtual users and scale them easily
- monitor the load generation process in real time from a web UI
- create your test scenarios using Python; maintain and configure performance tests for different environments under a version control system
Considering that the tool is rather new and the community is still growing, it’s biggest drawback is a very small library of plugins. Among other downsides are the lack of script recording functionality and ramp-up flexibility (the amount of time it takes a tool to add all test users to test execution).
Other advantages of using k6 are:
- ease of use due to its intuitive CLI (command-line interface) that shares a lot of UX aspects with popular DevOps tools
- test cases can be separated into different modules and the code reused among different tests
The shortcomings of using k6 are:
- a tool with no GUI can’t be efficiently utilized by testers who are used to no-code UIs
- supports only a handful of protocols out of the box such as web (HTTP/1.1, HTTP/2), WebSockets, gRPC, and SOAP/REST web services