scrum101

How important are your bugs?

As with all software development, you are almost certainly going to encounter bugs in your code, so how important are your bugs? And how do you decide when to fix them?

This article describes one simple method for how to incorporate bug fixes into your agile development process.

What is a bug?

software bug is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways

https://en.wikipedia.org/wiki/Software_bug

How do you know how important a bug fix is?

Welcome to the bug triage process! When I first started in my current role, we had a fairly familiar way of categorising bugs:

  • P1 = Fix it now
  • P2 = Fix it soon
  • P3 = We definitely should fix this one day
  • P4 = We can live with this bug, maybe one day we’ll fix it

And that very simple system worked fine. We’d get together as a group, once a week, to go through all the new bug reports, and categorise them into their appropriate categories. This meeting was attended by the Scrum Master, Product Owner, a few developers from the teams and was led by a member of the QA team.

However, using this method we saw two main issues that made this process not really work for us in the long term:

  1. We saw arguments over categorisation; as some people have their favourite bugs, e.g. if it affects them personally in a certain way, or they happen to be more senior.
  2. We also saw A LOT of P3 bugs. Most things were categorised as this. In the sea of P3 bugs, how do you know which of the P3s to tackle first?

How we tackled them once categorised

P1 bugs bug were fixed immediately, so their numbers stayed low.

In each sprint planning session, we’d review the P2 bugs, and decide what to take in.

If you were lucky (very lucky); there would be no P2 bugs, and we’d start looking at P3 bugs. This was very rare though.

And finally, the P4 bugs. When they got to 6 months old without being fixed, we just obsoleted them all. They rarely got touched. There were a lucky few, but not many. Setting a bug as P4 is basically saying we’re not fixing that.

We decided we could improve our bug triage process.

The new bug triage process

Each bug fix still has it’s own “bug score”, but this time it’s on a scale from 0 to 100 (100 being the most important thing to fix).

We answer a fixed set of simple questions about the bug, and dependent on the answers given, you generate an overall bug score.

This removes (to some extent) people’s opinions of a bug, and converts it into a raw number. We’re putting data behind our decisions now.

Here are the 4 questions we ask of every single bug fix:

1. Bug Type

  • No user impact. i.e. the user won’t even notice
  • Visual polish. It doesn’t affect navigation or branding
  • Minor Usability issue (or minorly affecting branding)
  • Impairs usability in some non-key scenarios (or significantly affects branding)
  • Impairs usability in key scenarios (or a major branding issue)
  • Major usability issue

2. Severity

  • Trivial. Has no/low impact on the site/app. Does not affect user experience
  • Minor. Has low impact on the site/app. User will notice but won’t affect user experience
  • Starts to hurt. Has a decent impact on the site/app. User experience will be slightly affected
  • Major. Has a strong impact on the site/app. User experience is strongly affected
  • Direct loss of revenue
  • Critical! Blocks progress, legal issue, crashes, 500 errors etc…

3. Likelihood: (how likely or % of users affected)

  • Most unlikely (0-10% users)
  • Very unlikely (11-20% users)
  • Unlikely (21-30% users)
  • Kind of unlikely (31-40% users)
  • Likely (41-50% users)
  • Most likely (51-60% users)
  • Very likely (61-70% users)
  • Almost sure (71%-80% users)
  • For sure (81-100% users)

4. Workaround

  • Quite easy to get around the issue (or a piece of config)
  • Fairly obvious, but, some users might not get it
  • Less than ideal workaround (most users won’t get it)
  • No workaround

TOTAL SCORE

In order to generate the bug score, you need to decide a number for each of the above scenarios. You could just assign sequential numbers within each list. Then, with the score for each section, you can simply add up the numbers to get a total.

Worked Example:

I have a bug that scores the following:

  • Bug Type = Visual polish (2)
  • Severity = Direct loss of revenue (5)
  • Likelihood = Kind of unlikely (4)
  • Workaround = no workaround (4)

If you add those up, you get 13 i.e. you don’t get a score out of 100. So we need to apply some weighting to our answers in each section to get a better view of how important this bug is to fix.

If you decide each section is just as important as all others, then you need to divide the amount each section is worth as a whole (25%) by the number of options available to choose from in that section. You then multiply the result of this, by the score that you gave it within that section, and you arrive at the weighted score for each section. Add all of these together to get the total weighted bug score (out of 100):

e.g.

  • Bug Type = ( (25% / 6 options ) * 2 (score) ) = 8.33 (there are 6 answers in this section, the section is weighted as 25 percent of the whole score, and we answered number 2 in the list)
  • Severity = ((25/6) * 5) = 20.83
  • Likelihood = ((25/9) * 4) = 11.11
  • Workaround = ((25/4) * 4) = 25

Total bug score = 65.27

Choose your own method

The exact weighting method and questions you choose are completely flexible and can be tailored to your product. You might decide for example, that bugs that affect the most people should generate higher scores, so you increase the weighting of the likelihood section and decrease the weightings of the other sections. It’s up to you.

When to tackle bug fixes

If you use this simple scoring mechanism for all of your known bugs; you will have an ordered list of bugs by total bug score. You can then just get started tackling them in that order if you want to.

However, what if you have other work to do? How do you mix these bug fixes into your everyday development process?

Well, we use team working agreements in sprint planning to help make sure our bugs aren’t getting out of control.

Team Working Agreements

These are just things that the team agree to do when working with each other. It’s kind of a reference guide on what we do in certain situations e.g. a team working agreement might be “no mobile phones in Sprint Planning”. The team hold each other to account for breaching their own team working agreements.

Part of my team working agreements

  • Bug score = 100 means FIX THIS BUG NOW and deploy it to production in the quickest possible time. Drop everything and get anyone to help.
  • Bug score = 90 means do not do another release of this product without fixing this bug first (it might not be live yet, but doing a release could make this a HUGE problem)
  • Bug score 80 means we need to tackle this bug in our current sprint; but we’ve still got a bit of time to fix this.

We tackle bugs by looking at the whole bug backlog in sprint planning and seeing what the highest scoring bug is.

Say our highest scoring bug is a 62. The team then look a bit lower than that and might see how many bugs are > 55, and bring all of them into the sprint. If any other bugs > 55 get raised during the sprint, they too are added to the sprint by the team.

In our next sprint planning session, we’ve fixed all of those bugs (hopefully), so we start at bugs scoring 55 and over and look lower down the backlog, maybe taking in everything over 50 this time. If any new bugs get raised that score over 50 during the sprint, they automatically get in and fixed.

Following this process ensures that our bugs stay low, and the quality of our products remains high.

Should you estimate bug fixes?

A thorny subject this one, and my answer is, it depends. For me, it depends how you are measuring your teams velocity (converting stories into working software):

  • If you are measuring velocity purely by “how much work gets done” each sprint, regardless of it’s type; then I’d say estimate bugs as well. You’ll get a clearer picture.
  • However, if you are measuring the quality and speed of delivery, and we are assuming that all bugs created are due to coding errors; then don’t estimate bugs. Fixing a bug instead of doing another story this sprint will lower your velocity (as the bug is worth 0 points). Therefore your average velocity over time will decrease to account for the mistakes made.

Summary

Some questions, a quick google docs spreadsheet and a short (30 min) meeting once per week, is all you need to get started with this process. When you have an ordered list of bugs, you can then update your team working agreements to include how and when you fix bugs.

I hope you found this article useful, I’ve been using this process for years, and the results are generally good. Our weekly bug triage meeting means that everyone knows what all the new bugs are and our clear process for when we fix them boosts transparency. If you like this, or think it’ll help anyone else, please share using the links below.

Till next time…

Stephen Waring
Stephen Waring

I’m an enthusiastic and energetic Scrum Master who loves everything about Agile!