Tag: qa

Demystifying Bug Triage

Rachel

I adapted this post from an internal guide I made for one of our teams. My goal was to demystify bug triage, lay out the basic hows and whys, and get buy-in from the team. I wanted everyone to feel comfortable triaging the issues reported in the team’s GitHub repositories (or other bug trackers).

The term “triage” comes from medicine, where it’s the process of determining the order in which patients will receive treatment based on the severity and urgency of their medical condition. At Automattic we apply the term “triage” to the processes we use to determine the severity and urgency of bug reports (and the potential positive impact of enhancement requests) so we can prioritize open issues. In other words, it’s how we keep our GitHub repos organized and make sure we can identify the next most important thing to work on.

How to Triage

What processes do we use for triage? Triage is primarily the initial review and prioritization of all new issues as they are opened in GitHub:

  • Add a label identifying the topic, feature, or epic related to the issue.
  • Add a label identifying the type of issue (e.g. bug or enhancement).
  • Add a label identifying the priority, if it’s clearly a high or low priority issue.
  • Check the issue to see if it’s missing any critical information, such as steps to reproduce or the device or app version where the bug occurs.
  • Add the issue to relevant projects or milestones for followup. If it’s a critical/blocking bug, escalate the issue in other ways, such as a direct ping to a team member.
  • Especially important when someone outside the team opened the issue: leave a comment to acknowledge the contribution and set expectations for followup.

I also use the term “triage” as an umbrella term for all the processes we use to review issues, and this includes reviewing all open GitHub issues on a regular basis:

  • Make sure that open issues are still valid and complete.
  • Look for trends, e.g. a group of issues related to a specific feature or component.
  • Re-prioritize issues when team goals and priorities change, or in response to trends you identified.

The exact timing for triaging new issues and reviewing existing issues depends on the team and project. If you’re just getting started, I’d suggest triaging new issues at least once per week and reviewing existing issues at least once per quarter (or whenever there’s a larger conversation about what to work on next).

Why to Triage

Why do these processes matter? They make it easier to:

  • Identify related issues that can be fixed at the same time, that show a potential weakness in a particular part of the app, or that point to a potential longer-term project.
  • Gauge the health of the app, in terms of number of issues and their severity.
  • Prioritize issues for regular maintenance.
  • Respond to all reports, especially from external contributors and reporters, to make sure nothing falls through the cracks.

Get Started

If you’re new to triage, here are some next steps you can take to get yourself and your team started:

  1. Agree on a consistent set of labels and what they’ll be used for. If you’re using GitHub, there is a set of default labels you can start with — but most important is to think of what’s useful for your team and how you work.
  2. Set up any projects or milestones you have or are planning to use to organize your work.
  3. Review all open issues (add labels, assign priority, check for completeness, etc.).
  4. Practice labeling new issues with appropriate topic, type, and priority labels. Hold yourself and your team accountable for doing this on all new issues you open.
  5. Identify a triage DRI (“Directly Responsible Individual”) and set a cadence for triaging new issues and reviewing existing issues going forward.

As with any work, be prepared to reflect and iterate on your processes. So far this approach has worked well for me and the teams I work with, but you may need to add or subtract steps to make it fit the way you work.

What do you think? Are your teams already doing this kind of triage? Are there any other steps or processes that you use to keep open issues organized and prioritized?

Advertisement

Testing as Exploration

Rachel

In the beginning, there was testing.

Thus begins James Bach and Michael Bolton’s essay on Exploratory Testing 3.0. The point they make is that, at the start, there wasn’t a clear distinction made between exploratory testing and automated testing. It was only after the rise of automated, scripted testing that the term “exploratory testing” came about to define human, interactive, ad hoc testing.

Bach and Bolton describe the evolution of exploratory testing over time. They note how the concept of agency came to characterize exploratory testing as opposed to scripted testing, and how they eventually decided to do away with the distinction altogether. That is, their new definition of testing is not exploration versus scripting — it characterizes scripting as just one technique through which we can explore and test our software:

Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.

As someone who loves tinkering with, exploring, and trying to break new things, I wholeheartedly support that perspective.

Bug Tracking for [Software] Health and [User] Happiness

Rachel

Timothy Western’s article “Clean Up Your Bug Tracker and Keep Numbers Manageable” is a straightforward, honest reminder of the importance of staying on top of reported software bugs. A backlog of reported bugs can be demoralizing and demotivating for everyone involved:

While the best teams will collaborate and try to fix bugs as soon as they are discovered, even their best efforts may not always keep up with the growing defect count. Features get notice; bugs that were deferred last week are likely to get deferred again. When push comes to shove and the team has to decide on deferring new bugs, what’s ten more added to a pile of ten thousand, anyway? When the technical staff realizes that product management cares about features only, consistently asking if the bug is deferrable, they lose pride in work, wondering, “Why bother to build something good when shoddy makes people happier?”

Western suggests some ways to combat that kind of bug pileup:

  • Use code reviews and paired programming to improve quality from the start
  • Test early and often (while the features and issues are fresh in everyone’s minds)
  • Find bugs that are quick to fix or high priority and put those in front of the team
  • Focus on improving the user experience rather than collecting as many bug reports as possible

I like Western’s advice because it focuses on finding approaches that make the most sense for everyone involved. If developers and testers aren’t collaborating and communicating throughout the process, it’s ultimately the users who suffer.

On Letting Go

Rachel

… it is not your job to stop the bugs (or the versions containing them) from walking out the door.

Your job is to provide visibility into the status of your product and your project, and give everyone on the team the information they need to make their decisions.

After you have given everyone the correct information they will need to make the choice whether to release the product into the field or not. You may be part of this team making the decision, but your voice will never be the only one that counts!

From Joel Montvelisky’s Letter to a starting tester