What if the customer can’t reproduce your bug?

Software quality assurance and testing is a complicated procedure that requires dedication, technical expertise, eye for detail, and hours of work. Testers are people who dive deep into your product and have a major impact on the final project outcomes. That’s why it is so important that the product owners understand well not only the importance of software testing, but its pain points from within for more effective communication. Our today’s post focuses on bug reproduction issues that usually remain behind the scenes for everyone but testers, which isn’t always good for the whole software project quality.

“Can’t reproduce”: what are we doing wrong?

Many software testers sooner or later face a problem when a customer/their team can’t reproduce a reported bug. A bug that you spent time and effort investigating. You tried so hard to report it back in a clear and concise manner, and yet they throw the “can’t reproduce” verdict in your face.

The reason why a person experienced troubles reproducing a bug can be in a tester’s actions as well. For example, he/she might have provided a vague bug summary, listed uncertain steps to reproduce or enclosed a useless attachment that either doesn’t demonstrate the bug correctly or doesn’t want to open at all. Even worse if you didn’t check the bug reproduction before getting to the report, and it stopped reproducing after the cache is cleared.

In case you are dead sure that the bug was reported correctly as well as it can be found by the application end users, you have no other choice but to defend it. This is achieved through communication with all stakeholders that can investigate and fix that bug in future. 

So what are the questions you should ask your QA team to find out why can’t they reproduce your bug?

1. Do you use the same testing environments (platform, browser, browser version, device, OS, screen resolution)?
2. What type of Internet connection did you and the testing team use?
3. Has the team counted in all the specified by you preconditions and steps for bug reproduction?
4. Does your expected result of software behavior match these of the team as well the expected testing outcomes stated in the project requirements?
5. Do you or your team use any third-party apps or browser extensions that could affect the software’s performance?

So, let’s look at all these questions in more detail.

First of all, in most cases the team doesn’t finish reading your bug. In the best case scenario, a person trying to reproduce your bug has read more than the bug summary and its attachment. That’s why it is no wonder some of the bug report attributes often get missed. Yes, it is as unfortunate as it sounds.

Do your testing environments match?

You have to specify which exact environment the team checked your bug in. 

It is supposed that the environment of a reproducing bug was stated in the description of it. Therefore, you have to double check whether your colleague who’s trying to reproduce the bug is using the same platform to do it (browser, device, operating system, screen resolution, of course if we’re talking about an environment-specific bug) as you did.

As an example, we once had a case when our tester reported a bug that could only be reproduced on iPhone 6 but no other versions. That’s why he specified the particular environment in the bug report description. A developer who had to process that bug report didn’t pay attention to the fact that the bug can only be seen on a specified device. Since he only had an iPhone 7 near him, he couldn’t reproduce the reported issue. Only after we double checked everything, the misunderstanding was finally solved: the bug that used to occur on iPhone 6 exclusively was found and fixed.

What type of Internet connection did you and the testing team use?

For the most part, such a question should be asked about device-associated bugs, because the type of Internet connection (wi-fi, 3G, 4G, etc.) can affect the performance of an application or site, so this information is really important and has to be clarified. 

Although, it turned out this issue is not only relevant for Mobile. On one of our projects, we found an interesting bug: the online store did not display a map of post offices available for certain delivery methods. We discovered it when all were working from home due to the quarantine. The customer team could not reproduce the bug, for them it seemed as if everything was perfectly displayed. After a few discussions, we decided to put the bug under investigation. After a part of our team got back to the office work, we found that the bug can no longer be reproduced.

Since a part of employees continued working remotely, we managed to discover a fascinating fact. With the home internet connection, the bug went on reproducing, meanwhile in our office, where the connection was more stable, the post office map was displayed successfully, just like on the customer’s side. That’s when we decided to include the information on the internet connection type and its speed to the bug report. As you can see, the internet connection itself can be a reason for a bug, which in our case could lead to certain customers being unable to place their orders in that online store.

Has the team counted in all the specified by you preconditions and steps for bug reproduction?

Getting back to the issue of the team not paying enough attention to the bug summary and its attachment, missing some important information. Namely, how to reproduce a particular bug.

Frequently, qualified employees tend to think they have mastered a project’s functionality to the point they don’t need to examine steps to reproduce and preconditions thoroughly. That’s why it’s worth double checking how exactly developers tried to reproduce your bug, whether they missed a step or didn’t take into account any preconditions.

We had a case when a bug was reproducing only after we marked one out of four checkboxes for one country in the settings of a software under testing. We added all this information to the bug description, which our colleague didn’t finish reading and, as a result, could not reproduce the bug, since he was following the usual steps to reproduce. 

Does your expected result of software behavior match these of the team as well the expected testing outcomes stated in the project requirements?

It happens. You reported a bug because it didn’t match the expected result specified in the requirements or even ones that weren’t specified, but you know exactly how the functionality should perform based on your experience, general standards or just common sense. You should discuss this issue with your colleagues. Maybe someone on the team is relying on inaccurate information about the expected result of the software behavior.

Do you or your team use any third-party apps or browser extensions that could affect the software’s performance?

And the last question to discuss is the one that could be asked from the bug report author. Sometimes, certain apps can conflict with one another. Naturally, this provokes software errors that QA analysts classify as bugs. Even VPN can affect the work. Therefore, you have to make sure that during the bug reproduction, there are no third-party applications, able to affect the performance of the tested product and mislead the entire team, installed. Or, for example, you use a virtual machine for testing, which can be less stable than the original platform or a browser itself. This also has to be clarified.

Our colleague one reported a bug displaying unnecessary web elements on a page. None on the team but him couldn’t reproduce that bug. As it was later revealed, the bug author used a spelling assistant application which reacted inadequately to certain web elements. 

Summing up

The aim of a software tester is to assure stable and error-free software functionality trying to prevent users from spotting bugs. That’s why we examine and classify all the bugs you may ever encounter. We take full responsibility for the quality of a tested product, high-quality bug report in case of finding a bug, as well as high-quality processing of it. 

And if you find a bug that to you seems a serious reason to cancel the product release, and the team cannot reproduce it, you should definitely find out the reasons by asking the above-listed questions. This will not only shape your professional reputation, but also affect the image of the entire QA department in the eyes of the development team. Needless to say about the company’s position in front of its end users ― a fragile thing that is so hard to get yet so easy to lose through a single serious mistake.