Vous êtes sur la page 1sur 26

The Fine Art of

Writing a Good Bug Report


What Is a Bug Report and Why
Do We Write Them?
• Bug report is a technical document
– Describes failure mode in system under test (SUT)
– The only tangible “product” of testing
• Not a management problem escalation tool
– “Build not delivered on time” is not a bug report
summary
– “Build 781 fails to install” is a bug report summary
• Written to increase product quality
– Documents a specific quality problem quality of SUT
– Communicates to developers
Are Bad Bug Reports a Problem?
•Anecdotally, developers return many bug reports as
unreproducible, leading to:
oWasted time writing the report
oFrustration for tester and developer
alike
oNo increase in product quality
•Bug reports can be unreproducible due to:
oIntermittence
oInconsistent test/development environments
oDisputes over “correct” behavior

•But many unreproducible bug reports are poorly


conceived and poorly written
Ten Tips for a Good Bug Report
1. Structure: test carefully
2. Reproduce: test it again
3. Isolate: test it differently
4. Generalize: test it elsewhere
5. Compare: review results of similar tests
6. Summarize: relate test to customers
7. Condense: trim unnecessary information
8. Disambiguate: use clear words
9. Neutralize: express problem impartially
10. Review: be sure
• Remember, writing is creative: two good bug reports on
one problem can differ in style and content (but not
substance)
Case Study: SpeedyWriter

• Let’s use a case study to illustrate these


techniques
• SpeedyWriter
• A Java-based word processor
• Supports the usual word processing
functions
Structure

• Structured testing foundational to good bug


reports
– Use deliberate, careful approach to testing
– Follow written test cases or run automated ones per
written or standardized process
– Take careful notes
• Bug reporting begins when expected and observed
results differ
• Sloppy testing results in sloppy bug reports
Structured Testing Paradigms
Good software
testing should be
more like a
carefully designed
laboratory
experiment than a
random walk in an
electrical storm or
worse. Testers are
engineers, after all.
Ben Franklins standing in a M. Maria
thunderstorm hoping to get Sklodowska- Curie
hit by lightning performing a carefully
designed experiment
Reproduce

• Always check reproducibility of the failure as part


of writing bug report
 Three times is a good rule of thumb
• Document a crisp sequence of actions that will
reproduce the failure
• Report intermittent, hard-to-repeat failures
– Note failure incidence rate (i.e., 1 in 3 tries)
* Clean steps to reproduce addresses issue of
unreproducibility head-on
Good Bad
Steps to Reproduce
1. I started the SpeedyWriter Nasty bug trashed contents of new
editor, then I created a new file. file that I created by formatting
2. I then typed in four lines of text, some text in Arial font, wasting my
repeating “The quick fox jumps time.
over the lazy brown dog” each
time, using different effects each
time, bold, italic, strikethrough,
and underline.
3. I highlighted the text, then
pulled down the font menu, and
selected Arial.
4. This nasty bug trashed all the
text into meaningless garbage,
wasting the user’s time.
5. I was able to reproduce this
problem three out of three tries.
Isolate

• Change variables that may alter symptom


– Make changes one by one
– Requires thought and understanding of SUT
– May not be immediately obvious
• Can be extensive
– Match amount of effort to severity of problem
– Avoid getting into debugging activities
* Good isolation shows due diligence and gives
developers head start on debugging
Better Good
Steps to Reproduce Steps to Reproduce
1.I started the SpeedyWriter editor, then I created 1. I started the SpeedyWriter editor, then I
a new file. created a new file.
2.I then typed in four lines of text, repeating “The
2. I then typed in four lines of text,
quick fox jumps over the lazy brown dog” each
repeating “The quick fox jumps over
time, using different effects each time, bold,
the lazy brown dog” each time, using
italic, strikethrough, and underline.
different effects each time, bold, italic,
3.I highlighted the text, then pulled down the font strikethrough, and underline.
menu, and selected Arial.
3. I highlighted the text, then pulled down
4.This nasty bug trashed all the text into
the font menu, and selected Arial.
meaningless garbage, wasting the user’s time.
5.I was able to reproduce this problem three out 4. This nasty bug trashed all the text into
of three tries. meaningless garbage, wasting the user’s
Isolation time.
On the vague suspicion that this was just a 5. I was able to reproduce this problem
formatting problem, I saved the file, closed three out of three tries.
SpeedyWriter and reopened the file. The garbage
Isolation
remained. If you save the file before Arializing
the contents, the bug does not occur. The bug Doesn’t happen on Solaris.
does not occur with existing files. This only
happens under Windows 98.
Generalize

• Look for related failures in SUT


– Does the same failure occur in other modules or
locations?
– Are there more severe occurrences of the same fault?
• Avoid confusing unrelated problems
– Same symptom can arise from different bugs
* Generalizing reduces duplicate bug reports and
refines understanding of failure
Better Good
Steps to Reproduce Steps to Reproduce
1. I started the SpeedyWriter editor, then I 1. I started the SpeedyWriter editor, then I created
created a new file. a new file.
2. I then typed in four lines of text, repeating
“The quick fox jumps over the lazy brown 2. I then typed in four lines of text, repeating
dog” each time. “The quick fox jumps over the lazy brown
3. I highlighted the text, then pulled down the dog” each time, using different effects each
font menu, and selected Arial. time, bold, italic, strikethrough, and underline.
4. This nasty bug trashed all the text into 3. I highlighted the text, then pulled down the
meaningless garbage, wasting the user’s font menu, and selected Arial.
time.
5. I was able to reproduce this problem three 4. This nasty bug trashed all the text into
out of three tries. meaningless garbage, wasting the user’s time.
Isolation 5. I was able to reproduce this problem three out
Also happens with Wingdings and Symbol fonts. of three tries.
On the vague suspicion that this was just a
formatting problem, I saved the file, closed Isolation
SpeedyWriter and reopened the file. The garbage
remained. If you save the file before changing the On the vague suspicion that this was just a formatting
font of the contents, the bug does not occur. The problem, I saved the file, closed SpeedyWriter and
bug does not occur with existing files. This only reopened the file. The garbage remained. If you save
happens under Windows 98. the file before Arializing the contents, the bug does
not occur. The bug does not occur with existing files.
This only happens under Windows.
Compare

• Examine results for similar tests


– – Same test run against earlier versions
– – Similar conditions, other tests, same version
• Is failure a regression?
– Change introduces defect not in earlier versions
– Usually found when previously passed tests fail
• • Not always possible
– Test previous blocked, reinstall impractical
– Tested feature unavailable in earlier versions
Better Good
Steps to Reproduce Steps to Reproduce
1. I started the SpeedyWriter editor, then I 1. I started the SpeedyWriter editor, then I created
created a new file. a new file.
2. I then typed in four lines of text, repeating
“The quick fox jumps over the lazy brown 2. I then typed in four lines of text, repeating
dog” each time. “The quick fox jumps over the lazy brown
3. I highlighted the text, then pulled down the dog” each time.
font menu, and selected Arial. 3. I highlighted the text, then pulled down the
4. This nasty bug trashed all the text into font menu, and selected Arial.
meaningless garbage, wasting the user’s
time. 4. This nasty bug trashed all the text into
5. I was able to reproduce this problem three meaningless garbage, wasting the user’s time.
out of three tries.
5. I was able to reproduce this problem three out
Isolation of three tries.
New to build 1.1.018; same test case passed
against builds 1.1.007 (System Test entry) Isolation
through 1.1.017. Also happens with Wingdings
Also happens with Wingdings and Symbol fonts. On
and Symbol fonts. On the vague suspicion that
this was just a formatting problem, I saved the the vague suspicion that this was just a formatting
file, closed SpeedyWriter and reopened the file. problem, I saved the file, closed SpeedyWriter and
The garbage remained. If you save the file before reopened the file. The garbage remained. If you save
changing the font of the contents, the bug does the file before changing the font of the contents, the
not occur. The bug does not occur with existing bug does not occur. The bug does not occur with
files. This only happens under Windows 98. existing files. This only happens under Windows 98
Summarize

• Put a short “tag line” on each report


– Capture failure and impact on customer
 Analogy: Bumper sticker
• Harder than it seems
– Testers must spend time thinking about this
* Advantages of good summaries
– Get management attention
– Name bug report for developers
– Help set priorities
Better Good
Summary
Steps to Reproduce
Arial, Wingdings, and Symbol fonts corrupt new
files. 1. I started the SpeedyWriter editor, then I created
Steps to Reproduce a new file.
1. I started the SpeedyWriter editor, then I 2. I then typed in four lines of text, repeating
created a new file. “The quick fox jumps over the lazy brown
2. I then typed in four lines of text, repeating dog” each time.
“The quick fox jumps over the lazy brown
dog” each time. 3. I highlighted the text, then pulled down the
3. I highlighted the text, then pulled down the font menu, and selected Arial.
font menu, and selected Arial.
4. This nasty bug trashed all the text into
4. This nasty bug trashed all the text into meaningless garbage, wasting the user’s time.
meaningless garbage, wasting the user’s
time. 5. I was able to reproduce this problem three out
5. I was able to reproduce this problem three of three tries.
out of three tries.
Isolation
Isolation
New to build 1.1.018; same test case passed New to build 1.1.018; same test case passed against
against builds 1.1.007 (System Test entry) builds 1.1.007 (System Test entry) through 1.1.017.
through 1.1.017. Also happens with Wingdings Also happens with Wingdings and Symbol fonts. On
and Symbol fonts. On the vague suspicion that the vague suspicion that this was just a formatting
this was just a formatting problem, I saved the problem, I saved the file, closed SpeedyWriter and
file, closed speedyWriter and reopened the file. reopened the file. The garbage remained. If you save
The garbage remained. If you save the file before
the file before changing the font of the contents, the
changing the font of the contents, the bug does
not occur. The bug does not occur with existing bug does not occur. The bug does not occur with
files. This only happens under Windows 98. existing files. This only happens under Windows 98.
Condense

• Eliminate extraneous words or steps


– Reread report carefully
– Avoid both cryptic commentary and droning on
• Are any details or actions irrelevant?
• Everyone’s time is precious, so don’t waste
any of it on unnecessary verbiage, but don’t
cut any meat, either
Better Summary
Good
Summary
Arial, Wingdings, and Symbol fonts corrupt new Arial, Wingdings, and Symbol fonts corrupt new files.
files.
Steps to Reproduce Steps to Reproduce
1. Started SpeedyWriter editor, then created 1. I started the SpeedyWriter editor, then I created a new
new file. file.
2. Typed four lines of text, repeating “The
quick fox jumps over the lazy brown dog” 2. I then typed in four lines of text, repeating “The quick
each time. fox jumps over the lazy brown dog” each time.
3. Highlighted text, then pulled down the font 3. I highlighted the text, then pulled down the font menu,
menu, and selected Arial. and selected Arial.
4. This nasty bug trashed all text into
meaningless garbage, wasting the user’s 4. This nasty bug trashed all the text into meaningless
time. garbage, wasting the user’s time.
5. Reproduced three out of three tries. 5. I was able to reproduce this problem three out of three
Isolation tries.
New to build 1.1.018; same test case passed
against builds 1.1.007 (System Test entry) Isolation
through 1.1.017. Also happens with Wingdings New to build 1.1.018; same test case passed against builds
and Symbol fonts. On vague suspicion this was a 1.1.007 (System Test entry) through 1.1.017. Also happens
formatting problem, saved file, closed
with Wingdings and Symbol fonts. On the vague suspicion
SpeedyWriter and reopened file. Garbage
remained. that this was just a formatting problem, I saved the file, closed
speedyWriter and reopened the file. The garbage remained. If
Saving file before changing font prevents bug.
Bug does not occur with existing files. Only you save the file before changing the font of the contents, the
happens under Windows 98. bug does not occur. The bug does not occur with existing files.
This only happens under Windows 98.
Disambiguate

• Remove, rephrase, or expand vague,


misleading, or subjective words and
statements
• Make sure report is not subject to
misinterpretation
* Goal: Clear, indisputable statements of fact
÷ Lead developer by the hand to bug
Better Summary
Good
Summary
Arial, Wingdings, and Symbol fonts corrupt new Arial, Wingdings, and Symbol fonts corrupt new files.
files.
Steps to Reproduce Steps to Reproduce
1. Started SpeedyWriter editor, then created 1. Started SpeedyWriter editor, then created new file.
new file.
2. Typed four lines of text, repeating “The quick fox
2. Typed four lines of text, repeating “The
quick fox jumps over the lazy brown dog” jumps over the lazy brown dog” each time.
each time. 3. Highlighted text, then pulled down the font menu, and
3. Highlighted all four lines of text, then pulled selected Arial.
down the font menu, and selected Arial.
4. This nasty bug trashed all text into 4. This nasty bug trashed all text into meaningless
meaningless garbage, including control garbage, wasting the user’s time.
characters, numbers, and other binary junk, 5. Reproduced three out of three tries.
wasting the user’s time.
5. Reproduced three out of three tries. Isolation
Isolation New to build 1.1.018; same test case passed against builds
New to build 1.1.018; same test case passed against 1.1.007 (System Test entry) through 1.1.017. Also happens
builds 1.1.007 (System Test entry) through 1.1.017. with Wingdings and Symbol fonts. On vague suspicion this
Reproduced with same steps using Wingdings and was a formatting problem, saved file, closed SpeedyWriter
Symbol fonts. On vague suspicion this was a
and reopened file. Garbage remained. Saving file before
formatting problem, saved file, closed
SpeedyWriter and reopened file. Garbage changing font prevents bug. Bug does not occur with existing
remained. Saving file before changing font prevents files. Only happens under Windows 98.
bug. Bug does not occur with existing files. Only
happens under Windows 98, not Solaris, Mac, or
other Windows flavors
Neutralize

• Deliver bad news gently


• Be fair-minded in wording and implications
• Avoid:
– Attacking developers
– Criticizing the underlying error
– Attempting humor or using sarcasm
• Confine bug reports to statements of fact
You never know who’ll read the reports
Better Summary
Good
Summary
Arial, Wingdings, and Symbol fonts corrupt new Arial, Wingdings, and Symbol fonts corrupt new files.
files.
Steps to Reproduce Steps to Reproduce
1. Started SpeedyWriter editor, then created 1. Started SpeedyWriter editor, then created new file.
new file.
2. Typed four lines of text, repeating “The quick fox
2. Typed four lines of text, repeating “The
quick fox jumps over the lazy brown dog” jumps over the lazy brown dog” each time.
each time. 3. Highlighted all four lines of text, then pulled down the
3. Highlighted all four lines of text, then pulled font menu, and selected Arial.
down the font menu, and selected Arial.
4. All text converted to control characters, 4. This nasty bug trashed all text into meaningless
numbers, and other apparently random garbage, including control characters, numbers, and
binary data. other binary junk, wasting the user’s time.
5. Reproduced three out of three tries. 5. Reproduced three out of three tries.
Isolation
Isolation
New to build 1.1.018; same test case passed against
builds 1.1.007 (System Test entry) through 1.1.017. New to build 1.1.018; same test case passed against builds
Reproduced with same steps using Wingdings and 1.1.007 (System Test entry) through 1.1.017. Reproduced with
Symbol fonts. On vague suspicion this was a same steps using Wingdings and Symbol fonts. On vague
formatting problem, saved file, closed
suspicion this was a formatting problem, saved file, closed
SpeedyWriter and reopened file. Garbage
remained. Saving file before changing font prevents SpeedyWriter and reopened file. Garbage remained. Saving
bug. Bug does not occur with existing files. Only file before changing font prevents bug. Bug does not occur
happens under Windows 98, not Solaris, Mac, or with existing files. Only happens under Windows 98, not
other Windows flavors. Solaris, Mac, or other Windows flavors.
Review

• Each tester should submit each bug report to one


or more test peers for a review
• Reviewing peers should:
– Make suggestions to improve report
– Ask clarifying questions
– Even challenge “bugginess” if appropriate
Test team should only submit best possible bug
reports, given time constraints appropriate to
priority of bug
Conclusions

• A bug report is a technical document


– Accurate
– Concise
– Well-conceived
– High-quality
• Instill tester focus on writing good reports
• Quality indicators for good bug reports
– Clear to management (especially the summary)
– Actionable by development
– Move quickly from opened to closed
ROI for Good Bug Reporting

• Improved Test team communication to senior and


peer management
– – Can enhance credibility, standing, and resources
• Better tester/developer relationships
– Helps developers be effective, reduces arguments
• Faster, more efficient bug lifecycles
– Quicker fixes and fewer reopens
Support increased product quality