06

Reporting bugs (Éibhear's opinion, FWIW)

Prologue

I'm old. I'm so old, I was already a development team lead and systems designer before DevOps was a twinkle in anyone's eye, or before The Agile Manifesto was anything more than some scratchings on a paper napkin.

What follows is a guideline for how testers were to report bugs to me and my team. Younger people (or those not-so-young people who pretend they're trendy) will argue that with all these new, modern automated testing systems, this should be unnecessary. To that I have this to say: there are organisations out there who aren't yet in a position to make use of those tools and methods, and this post is for them.

Introduction

Purpose

This document outlines what a developer should be provided as part of a well-formed bug report.

Background

A bug is a behaviour in software that is not expected. Often, the unexpected behaviour arises from code failure. However, this is not always the case. In order to assist the developer in determining the root cause of the problem, the bug/issue report should provide as much information as possible. Most, if not all of the information to be provided to the developer is information that the reporter would have to hand.

Information to be in a bug/fault report

Subject

A single, short, sentence that nearly-uniquely describes the problem as the reporter sees it.

  • Make it read like an e-mail subject
  • It's not possible to ensure that the description is unique, but it should, at least, be distinct from the other open issues.
  • The subject should state the problem, and not what the fix is, or should be.
  • If using a standardised information entry form, the subject should not include information that is recorded elsewhere (e.g. in Bugzilla, Product and Component are recorded in one of the fixed-format fields, etc).

Examples

Good
"Can't access profit centre field in FT Map screen for contract of product XYZ".
Bad
"FT Map fields not working"

Steps to reproduce the bug

A detailed set of steps that the developer would follow to reproduce the defect. This information will allow the developer to do this before returning to the reporter (if necessary) for more information.

Where possible, avoid using information that is unique to the environment (e.g. specific server names, or URLs), so that the developer can attempt the steps in the development environment. Of course, this may not be possible if the fault depends on specific data or configurations in the environment. If this is the case, these details should be noted in the report.

Examples

Good
  1. Log onto ADS and search for customer '000123456'
  2. From search screen, drill down to customer screen, and then drill down to view the details on the contract '11123415'.
  3. Create a new contact record with the following settings:
    • Loan letter 1
    • Handler code 'ABCD'
  4. Attach this case to the contract '72615344' by pressing the "Attach" button.
  5. Save contact record.
  6. Quit the ADS.
  7. Log back in and navigate to the same contact record for both contracts.
  8. The handler code for '11123415' is recorded correctly as 'ABCD', but the handler code for '72615344' is blank.
Bad
"when I create a contact record for a contract and attach it to another contract not all the fields are updated properly"
Bad

The following was the full report on a recent ADS bug

"Hi…

When I try to generate a letter from the arrears system, I'm getting an error 'Internet explorer cannot display the webpage'. I have attached the screenshot - could you take a look please?

Thanks.."

Expected behaviour versus Actual behaviour

This is critical.

The developer must be provided with a description of what was expected and a description of what actually happened. It's not fair to make the developer guess as to what was supposed to have happened.

Good example
"We expected that the interest surcharge field in the output file for this record would be the same as the value for the contract in the EIS"
Bad example
"it's not right"

Opinion

This is also critical.

It may not be obvious that this is a bug or fault. The reporter should provide an analysis as to why this is to be considered a bug.

Other subjective information that should be included are severity and scope of impact.

Example

Good

"Section x.y.z of the requirement states that for this type of contract, the profit centre field is required and must be supplied. However, as the field is disabled, this isn't possible. This is why we're raising this issue. We find that this bug effects contracts from products A, B and C, but for products D and E, this bug isn't present.

As these are the most common product types, and as this field is required for the ticket to be uploaded, this is a critical bug."

Bad
"Please have a look to see if this is supposed to be the case."

Supporting information

The following lists the other, supporting, information that the developer might require for investigation purposes.

  • A contact person for the fault/bug (tester, analyst, ITI staff member etc.)
    Good example
    "As I will be on leave tomorrow and the next day, I'll be handing this bug over to X, and (s)he will be aware of the details."
  • The investigations that the reporter did to confirm that the fault is something for the developer to investigate.
    Good example
    I looked into the following: I confirmed that in the CONTACT table, the handler code is populated for the contact record for '11123415', but is reporting NULL for the contact record on '72615344'."
  • Date and time on which the error occured, both system date and time, and application date, if different.

    This would let us know where in log files, for example, to look (or what log files to look for, in the event that they have been archived).

    Good example
    "This occured at around 10:30 on the 5th. The core system, though, was at the 1st February."
  • Scope of the fault. This will give us information on the domain range of the fault.
    Good examples

    "The bug occurs just for this contract."

    "All accounts of class DEPSIA with an open date of before 1st January 2007 show this fault. We haven't seen it in any other accounts."

  • Log files. This could include application log files (e.g. output files produced by the application's code) and system log files (e.g. log files from Apache, Oracle, IIS, etc.)

    The trouble shooting document of the system at hand will provide information on where to find these output files.

    Good example
    "The following files are attached:
    • Error and access log files from apache web server
    • The application error and access log files."
  • Discussion threads regarding the bug. If the reporter has e-mails or instant message logs that discuss the fault/bug, please help the developer by including the full conversations in the bug report.
    Good example
    "See below for an IM conversation I had with the DBA on this problem, confirming that the DAD configuration is correct."

Other suggestions

Don't assume: if you encounter the bug in system X, report the bug as being in system X. If you think the bug is in system Y, then provide the information you used to come to that conclusion.

Avoid screen shots: pictures are very difficult to search, and they can also be missing some critical information.

  • For example, a screenshot of a terminal session may not include all the output of the command. Instead, provide the log file, or copy-and-paste the output from the terminal session into the bug (or a text file). If you feel the two are materially different, then provide both.
  • In some circumstances, a screen shot is necessary1. However, please use it only to support the report, not as the report.
  • Finally, if you have to provide a screenshot, please provide them as image files (e.g. jpg, png, etc.) and not pasted into MS Word documents. This just increases the amount of time the developer needs to spend on coming to understand the bug report.

If you think it's an environment issue (but you're not sure), rather than a code problem, log the bug anyway. The developer can help to demonstrate and confirm that it is an environment issue.

Footnotes:

1
Jesus, did I really write that?!