7. How to effectively manage an automation QA team and who should do it?
Nitish Bhardwaj: The automation QA team should be managed by the automation QA Lead, who should provide guidance, support, and feedback to team members. It is essential to ensure that team members have the necessary resources and tools to perform their jobs effectively. Regular communication, team meetings, and performance reviews can help to ensure that the team is meeting its goals and objectives. The QA automation lead can report to the QA manager/director, an engineering manager, or a technology leader.
8. When you already have a team, how do you evaluate it? How do you know it’s efficient? Are there some metrics you can use?
Yarden Ingber: It can be challenging to track the success of an automation developer because they don’t develop the product itself, meaning you don’t have bug reports from the customers. Still, there are definitely some metrics you can use. For example, positive feedback from product developers — If they are happy with the test suite that the automation developers created — that’s a sign of a good automation developing team. I think the stability of the test suite is another very important metric because if the test suite is unstable, then the trust of the entire company in the test suite fails. At the same time, I don’t like the idea of testing or measuring the team by the number of tests created per week or the number of tests fixed, or similarly hard metrics. I would rather have softer metrics.
Andrew Knight: You essentially want to build a scorecard that indicates the health and maturity of the team in their testing journey. These metrics shouldn’t be punitive, but rather a way to reveal opportunities for areas of improvement. If you make the metrics punitive, then the whole thing is off. You’ve got to make sure that you message that appropriately. I believe metrics should reflect part of the story, but not the whole story. Some of the things I would look for when scoring a team would be things like:
- How long does it take you to develop a new test?
- Does the framework help you or hinder you from making progress?
- How long does it take to run a test suite —15 minutes, an hour, 20 hours?
- What is the amount of time for the feedback, from the developer pushing the change to getting the result?
9. Is it possible to run a distributed automation testing team? How to build efficient communication with a distributed/offshore team?
Yarden Ingber: All of my automation developers are outsourced and do not share an office with me. On the other hand, most of the software developers do share an office with me. What I see is that there is a very big advantage for people that work together. And you can see it in day-to-day actions, when I sit with someone to eat lunch. We talk about work and ideas can pop up. This is something you can hardly have if you just communicate through text or Zoom meetings. But we’re trying! It’s very important for me to open each meeting with at least 15-20 minutes of talking about everything except work. We talk about the weather in everyone’s country, personal matters, family stuff, traveling, and even how their weekend was. It can be complicated to have this personal communication when everyone is in different locations, so these personal discussions absolutely help.
Andrew Knight: I definitely enjoy working with an in-house team more, but we also successfully adjusted to the shift to remote work. Plus, working in tech, we were more prepared to go remote than many other industries. But there isn’t much I’m doing differently than with in-house teams. You do have to be more intentional with communication to make sure that you can build both personal and professional connections. I would love to be able to meet up in person regularly, but when that is not available, you can still make it work.
Nitish Bhardwaj: I think with remote working and effective collaboration tools, it’s not going to be a challenge. It is essential to establish clear communication channels and set expectations for communication and feedback. Additionally, it is important to ensure that the team members have the necessary resources and support to perform their jobs effectively, regardless of their location — for example, it’s vital to ensure equal access to the test environment from all locations, and take into account things like network lags.
10. What is team culture to you when we’re talking about automation testing?
Yarden Ingber: I think that both manual and automation developers are very dependent on the product developers. So both for manual and automation QAs, I value the ability to communicate well. They can explain their ideas and what is blocking the progress. When they have a request from the developers and are able to communicate their needs, the developers can quickly help. As for other aspects of team culture, I don’t think a company should be absolutely homogenous. Different people with different views and different personalities can benefit from one another, so differences should not be an issue.
Andrew Knight: I think there are certain things that people on a test automation team would understand that people from other teams would not, such as test automation diversity or different testing conferences and events. That’s part of the team culture. You also want a healthy team culture. You want people who are respectful of each other, optimistic, energetic, with a go-getter attitude, who love what they’re doing and enjoy working with people on the team, who are not afraid to give critical feedback but do so in a healthy manner. Those are the qualities I look for in a team.
3. Automation testing tools: Choosing the right one for the job
Find out how to pick the perfect tool and technology for your testing automation project.
1. What are your favorite automation testing tools and why?
Marcus Merrel: I don’t tend to choose tools until I understand what the project is and what they’ll be used for. I guess my answer is that my favorite automation testing tool is the one that fits best into the culture of the team and the rest of the stack the developers are using. The tools I’ve used the most often in my automation career have been some variations of Java/TestNG/Selenium, but I would never say that those are adequate for all testing purposes. It usually takes over a dozen different tools sprinkled throughout the SDLC to make automated testing happen.
2. Do you pick tools based on the project specifics, or do you have a universal go-to tool set?
3. Who is involved in the process of selecting the tools at your organization?
Marcus Merrel: It’s generally left up to the teams, but in some cases, we don’t have many choices. Our teams are oriented towards microservices to provide APIs to internal and external customers. We have some flexibility, but Sauce Labs is a software company that’s fundamentally in the business of managing hardware. When you have several thousand iOS and Android devices in a lab, you’re somewhat limited to the choices offered by Apple and Google, not to mention all the low-level networking and USB cable-level code we have to deal with.
4. What are the features you are looking for in an ideal tool?
Marcus Merrel: Personally, I start by looking for open-source tools with a strong community. The community around an open-source project is a lot like the corporation behind a commercial product: a strong community means strong support and ongoing maintenance/feature work. A weak community is similar to what you get when you buy a small product from a large corporation. When choosing commercial tools, I rely on references and my network to help me understand the product’s roadmap, stability, support, etc. I also favor tools that do one thing and do it well. I’ve been involved with projects where people ask us to put some random features in, because “competitive projects have XYZ features, why don’t you?”. My preference is always to explain why they don’t actually want us to implement a poor version of XYZ, but to integrate with an amazing tool that specializes in XYZ. This way, we can continue to be experts in our thing, while they stay experts in theirs. I’m a big fan of specializing.
5. What other principles do you use in your decision-making process?
Marcus Merrel: I come from the open-source world, so I have a natural inclination toward open-source tooling. I like projects that have a rich ecosystem of commercial support that surround an open source product — this means I pay people to help me use a product that’s free, rather than paying some seat or consumption price for the software. There are pros and cons to this approach, and I’ve definitely bought my share of commercial software, but this is my initial approach.
6. Do the available automation tools fully meet your needs, or do you feel like the market is missing something?
Marcus Merrel: The market is absolutely missing some critical things, and as far as I can tell, nobody is even seriously trying to attain them. While the entire ecosystem of testing products is trying to automate the human tester out of a job, we’re not getting answers to simple questions like “What is the value of my automated tests?”, “How can I make my testing better, more effective, and faster?”, “What kind of testing am I missing?”. The analysis of test signals is only being considered by a couple of products, and it drives me a bit crazy because this is such a solvable problem. We have all this data but no idea how to analyze it. In my opinion, this is more valuable than an effort at low-code test automation.