Pokemon Go Showed the Importance of Stress and Load Testing

You are here

Pokemon Go load testingThe enormous success of Pokemon GO is a demonstration of how a single application can change the world in a couple of days. Millions of active users started to wander around their neighborhoods searching for mysterious creatures and fighting for the gyms. Only two weeks after its release, Pokemon GO managed to earn the title of the biggest US app ever, surpassing the Twitter App in daily active users.

This is an anomaly, one that we witness rarely—a true manifest of the cultural phenomenon evolution from one media type into another. However, this undoubtedly glorious transition didn’t happen flawlessly. There were some hiccups and many users were dissatisfied with the performance. We cannot continuously praise this amazing game without highlighting some glaring faults. Let’s keep things objective.

There is no need to list the “goodies” that Pokemon GO brings to the table. Dozens of articles all over the internet cover them fairly well, going deep into the details. We want to highlight some obvious issues. Some of them have been fixed, others still exist.

  • Stability issues. The game reportedly crashed after catching Pokemons or entering PVP. The latest patch seemed to fix some cases, but the issue still persists on many devices.
  • Clunky controls. Many users expressed their concerns about the interface, unreliable hit targets, and controls less responsible than desired.
  • Connectivity issues. This is a double-edged sword. Servers still struggle with unexpectedly high load. On the other hand, many mobile internet providers do not have sufficient coverage to satisfy all their users. There are places where you simply have poor connection accompanied with regular service disruptions. This is not Niantic’s fault.

There were other minor complaints like game design problems and poor game balance. However, these are easily solvable and do not drastically affect user experience. All in all, Pokemon GO successfully “survived” the after-deployment period and managed to satisfy the majority of users. The undeniable triumph of Pokemon GO is the reason for the contemplation. Why do modern startups and enterprises have to hire QA engineers?

Stress and Load Testing

There two different types of testing that are directly related to reliability. We call them Stress Testing and Load Testing. Scalable applications like Pokemon GO need to be ready for extensive load. At the same time, ensuring that the backend is versatile and stable enough to support continuous changes and alterations to the application is pivotal in the long run. For example, the first patch fixing the issue with Pokemon GO accessing Google+ accounts was released less than two weeks after the release.

Stress Testing. During stress tests we literally try to break the application. Using bots, vicious scripts, and denial tools is the way to go. The main goal of Stress Testing is to examine how the application and its backend handle crashes and errors. Simultaneously, we try to identify critical software vulnerabilities and crucial stability/security aspects that need additional attention.

The goals of Stress Testing:

  • To identify what can break the application
  • To determine safe usage limitations
  • To confirm that intended specifications are intact
  • To identify failure patterns
  • To test the application in the environment exceeding standard in terms of load/intensity

Frequently, enterprises and software development companies hire software testers with experience in Stress Testing. Many freelancers participate in such testing activities. Just like white-hackers are necessary when it comes to security testing, stress testers who know how to write malicious software and service denial scripts are more than welcome during Stress Testing.

Load Testing. While Stress Testing is an example of destructive methodologies commonly used in various industries, Load Testing focuses on checking whether the application can handle the pressure from anticipated load extremums. We do not want to break the application in order to identify critical issues, but rather examine how it works under heavy load.

The goals of Load Testing:

  • To test QOS (quality of service) performance
  • To identify various issues occurring under heavy load
  • To confirm that intended specifications are met
  • To identify the limitations of the application and its backend

Load testing is necessary for any application that allows multiple users to interact with its backend. While Stress Testing may be questionable when it comes to applications that do not expect extreme loads, Load Testing is undoubtedly important for any project. This is why there are dozens of frameworks and methodologies that can be used in Load Testing.

How do these things relate to Pokemon GO?

We cannot stress enough (no pun intended) how important it is to conduct extensive testing activities before releasing an application. We all know that judging the book by its cover is wrong, but there are no other ways to judge it when you first see it. You need to ensure that the initial impression left by your application satisfies the customer, that the end-user likes the application.

Imagine that you want to try out an application. You install it, you launch it, the excitement grows, and BOOM! Suddenly it crashes. Obviously such an experience will negatively affect you. Most likely, you will give it a couple of tries, but if crashes keep happening, you will simply forget about it. The revolutionary idea, ground-breaking features, and other cool aspects of your applications will never spark the interest of this specific user.

Stress Testing and Load Testing focus on preventing such scenarios. Performance testing in general ensures that the application works correctly in various environments. A properly tested application will handle errors appropriately and won’t “faint” under unexpectedly heavy load. This is very important for an application like Pokemon GO that, without a doubt, expected enormous loads right after the release.

Millions of users are enjoying this game right now as we speak. Even if they couldn’t experience the entirety of the feature set, they could at least play the game and check out its core features. Occasional crashes and other inconveniences were not significant enough to create a meaningful obstacle for Pokemon GO on its way to total domination over the industry. Therefore, we believe that their Stress and Load Testing activities were successful.

This game would have never achieved such popularity if it constantly crashed or if its servers were unavailable all the time for the majority of users. However, this was not the case. The fact that millions of users were satisfied and only a minority found this application unusable is the biggest proof that performance testing is more than important for modern applications.

The future of Stress and Load Testing

Numerous development teams create their own testing teams and departments. Many still prefer to cooperate with certain specialists with rich experience in software testing. The latter is especially just for startups and small app development teams that simply cannot afford their own testing departments. So where can one find good software testers? Ukraine is one of the best countries to search for such professionals.

This is a country renowned for its IT expertise and fair prices. The Ukrainian IT industry is about to catch up to the most advanced countries with fourteen IT clusters and more than 100,000 IT specialists available for hire. Their IT-outsourcing market grew by 4.5 times over the course of six years from 2008 to 2014. Simultaneously, this is the biggest IT industry in Eastern Europe in terms of generated revenue, with its market estimated at USD 3.6 billion in 2013.

There are notable outsourcers that show constant progress in India and Indonesia. These are the cheapest countries to outsource in, but they are notorious for poor quality. However, with the growth in demand for software testing, the industry will surely evolve, bringing more talent under the spotlight.

Pokemon GO sets up a number of positive precedents for our industry:

  • Augmented reality projects can be amazingly interesting and engaging
  • Augmented reality games with online play features can bring in crazy profits
  • Large media universes like Pokemon can be similarly huge in the world of mobile apps
  • Proper Load and Stress testing ensure the success of the application

The hype surrounding both the game and the industry is warranted for the bright future of the genre. More AR projects are on their way. New ground-breaking wearables are about to hit the market. More investors are contemplating whether they should start opening their wallets and fund promising startups. All of the above tells us that the true competition between software developers is just about to begin.

With big money on the line, you will have to invest more in proper testing. If your application fails horribly or won’t meet estimated specifications resulting in negative reviews, your competitors and investors will eat you alive. This means that testers will have to improve and that QA will demand even more attention than previously.


Pokemon GO is surely the brightest phenomenon of the decade in our industry. The game pushes the AR segment of the industry further, opens up new opportunities, and proves that QA is pivotal for the success of any application. Hopefully, more startups will take note and start emphasizing the importance of performance testing.