`
BugCache
Predicting Defects
Sung Kim • MIT Tom Zimmermann • U. of Calgary Jim Whitehead • UC Santa Cruz Andreas Zeller • Saarland University
All modules are bug-free!
Bug-free module
Dream
X
X X
X
X
Some modules are buggy!
Bug-free module
X Buggy module
Some are Buggy
X
X X
X
X
Predicting Buggy Files
Bug-free module
X Buggy module
Predicting most bug prone files
Motivation
Which files should we focus on?
Where are bugs?
Temporal locality:
Defected files are
likely to have more soon.
[Ostrand, Weyuker]
In modified files!
[Nagappan et al.]
In new files!
[Graves et al.]
Spatial locality:
In nearby other bugs!
[Zimmermann et al.]
Our Solution
• List of most bug-prone files
• Dynamically adaptive and intuitive
• Combine bug prediction models
10% BugCache predicts 73~95% of bugs
Outline
•
Bug Cache Model and Operation•
Evaluation•
Seven open source projects•
Related Work•
Applications•
SummaryBug Cache Model
Identifying Bugs
……
Development history of JEditTextArea.java
...
...
...
...
Rev 102 ...
...
...
...
Rev 1
...
...
...
...
Rev 100
...
...
...
...
Rev 101
change change
Identifying Bugs
...
insertTab() ...
...
Rev 102 (no BUG)
...
setText(“\t”) ...
...
Rev 101 (with BUG)
fixed
……
Development history of JEditTextArea.java
...
...
...
...
Rev 102 ...
...
...
...
Rev 1
...
...
...
...
Rev 100
...
...
...
...
Rev 101
change change
Identifying Bugs
...
insertTab() ...
...
Rev 102 (no BUG)
...
setText(“\t”) ...
...
Rev 101 (with BUG)
fixed
……
Development history of JEditTextArea.java
...
...
...
...
Rev 102 ...
...
...
...
Rev 1
...
...
...
...
Rev 100
...
...
...
...
Rev 101
change change
Change message:
“fix for bug 28434”
Cache Operation
Miss
Cache size: 2
A B C
C
Cache Update
Parameter: Block size (neighborhood size)
• Load missed files
• Load nearby files (spatial locality)
File Number of common changes with .
1 4 0
C A
B D
4
B
Cache Operation
Hit
Miss Miss
Cache size: 2 Block size: 2
Hit
A B C C A B B D A
C A
B
Which one should be replaced?
Replacement Policies
•
Least recently used (LRU)Unload the files that have the least recently found defect.
•
Least frequently changed (CHANGE)Unload the files that have the fewest changes.
•
Least frequent defects (BUG)Unload the files that have the fewest defects.
Parameter: Replacement Policy
Cache Model
Hit
Miss Miss
Cache size: 2 Block size: 2
Hit
Replacement: BUG
A B C C A B B D A
C A
Block size: 1 Cache size: 2
File LRU CHANGE BUG
-5 2 2
-3 3 1
B C
BUG 2
1 (replace)
B
Cache Evaluation
Hit
Miss Miss
Cache size: 2 Block size: 2
Hit
Replacement: BUG
A B C C A B B D A
C A
Hit rate = #Hits / #Defects = 50%
Subject Programs
PostgreSQL jEdit
Mozilla Columba
0 25 50 75
Subversion PostgreSQL Mozilla JEdit Eclipse Columba Apache 1.3
File Function
Hit Rates
Cache size = 10%
Block size = 50% of the cache size Replacement policy = LRU
67
76
85 83
93 79
71
43
59 55 46
69 67 60
0 25 50 75
Subversion PostgreSQL Mozilla JEdit Eclipse Columba Apache 1.3
File Function
43
59 55 46
69 67 60
0 25 50 75
Subversion PostgreSQL Mozilla JEdit Eclipse Columba Apache 1.3
File Function
Exhaustive Evaluation
• Cache size: fixed to 10%
• Vary block size:
0% to 100% of cache size
• Vary replacement: LRU, CHANGE, BUG
Optimal Hit Rates
Cache size = 10%
73
79
88 85
95 83
82
46
59 55 49
72 68 62
0 25 50 75
Subversion PostgreSQL Mozilla JEdit Eclipse Columba Apache 1.3
Function File
0 25 50 75 100 BugCache. Top 10%
Hassan et al. Top 10%
Ostrand et al. Top 20%
Khoshgoftaar et al. Top 20%
Khoshgoftaar et al. Top 10%
Related Work
64%
82%
71~93%
44~78%
0 25 50 75 100 BugCache. Top 10%
Hassan et al. Top 10%
Ostrand et al. Top 20%
Khoshgoftaar et al. Top 20%
Khoshgoftaar et al. Top 10%
Related Work
64%
82%
71~93%
44~78%
73~95%
0 25 50 75 100 BugCache. Top 10%
Hassan et al. Top 10%
Ostrand et al. Top 20%
Khoshgoftaar et al. Top 20%
Khoshgoftaar et al. Top 10%
Related Work
64%
82%
71~93%
44~78%
73~95%
Previous State of the Art, 10% predicts 44%~78%
20% predicts 71~93%
10% BugCache predicts 73~95%
Applications
•
Additional QA efforts•
Selective assertion/error checking•
Dynamic and static (bug cache) analysis combinationStatic and Dynamic Analysis
•
Combine static and dynamic analysis•
BugCache + Dynamic AnalysisOverhead False positives
Static Low High
Dynamic High Low
•
Reproducing crashes (faults) is hard!• Require the exact configuration of crash (in field)
• Crashes usually involve nondeterministic facts
•
Must be able to reproduce crashes to fix bugs and validate fixesReproducing Crashes
Subject program
ReCrash
13-64% performance overhead
Test case
ReCrash
ReCrash
• Monitoring all modules
ReCrash + BugCache
X
X
X X
X
X
ReCrash
• Monitoring all modules
BugCache
• Identify crash- able modules
ReCrash + BugCache
X
X
X X
X
X
ReCrash
• Monitoring all modules
BugCache
• Identify crash- able modules
ReCrash + BugCache
• Monitoring only identified
modules
ReCrash + BugCache
X
X
X X
X
ReCrash
• Monitoring all modules
BugCache
• Identify crash- able modules
ReCrash + BugCache
• Monitoring only identified
modules
• 10% of modules account for 70% of crashes
• ReCrash + BugCache can possibly
• run with 1~6% overhead
• reproduce 70% of crashes
ReCrash + BugCache
Summary