iseweek3
3.1 Intelligent Software Engineering: Classic Metrics
Software Defect Prediction
- The foundation of software defect prediction lies in metric identification.
- This was a key research direction in the 1980s.
- Metrics aim to quantify properties of code to detect potential defects and improve quality.
Classic Code Metrics
1. McCabe Cyclomatic Complexity
- Purpose: Measures the complexity of code based on the number of linearly independent paths in the code’s flow graph.
- Why it Matters
- More conditional statements = More possible execution paths = Higher complexity.
- Useful for identifying complex, hard-to-test, and error-prone code.
- Simple Definition
- McCabe Complexity = Number of simple conditions + 1
- What is a “Simple Condition”?
- A conditional without logical connectors (AND, OR).
- Examples:
- if (a > b)
- while (a > b)
- for (a=b; a > b; b++)
- do {…} while (a > b)
- Compound Conditions
- Count each simple condition inside:
- if (a > b || a > 2) → 2 simple conditions
- if (a > b && a > 2) → 2 simple conditions
- Count each simple condition inside:
- Use Case
- Helps determine test case count needed for complete branch coverage.
2. Halstead Complexity Measures
- Purpose: Measures complexity based on the operators and operands used in code.
- Definitions:
- n1: Number of distinct operators (e.g., !=, !, %, /, *, +, &&, ||)
- n2: Number of distinct operands (e.g., variable names, constants, types like bool, char)
- N1: Total occurrences of operators
- N2: Total occurrences of operands
- Why Use Halstead?
- Evaluates:
- Code length
- Code vocabulary
- Effort required to implement or understand the code
- Potential bugs
- Evaluates:
3. Lines of Code (LOC)
- LOC: Total number of lines in a program.
- Comment Lines: Lines containing only comments.
- Usefulness:
- Simple indicator of:
- Code size
- Code density
- Maintainability and readability
- Simple indicator of:
3.2 Intelligent Software Engineering: Within-Project Prediction
Just-in-Time (JIT) Defect Prediction
- Based on the classic work by Kim et al. (2008).
- Focuses on predicting defects at the commit/change level rather than file or module level.
Steps in the JIT Defect Prediction Pipeline:
- File-level changes are extracted from a project’s revision history.
- Bug fix changes are identified using keywords in SCM (Source Code Management) change log messages.
- Bug-introducing and clean changes are identified by tracing backwards from the bug fix commits.
- A classification model (e.g., SVM) is trained on these labeled examples.
- Once trained, the classifier can predict if new code changes are likely to be buggy or clean.
Change-wise Prediction Details
Change History Extraction
- Collected information includes:
- Change log
- Author
- Change date
- Source code
- Change delta
- Change metadata
Identifying Bug-Introducing Changes
- Step 1: Search for Bug Fixes
- Use keywords (e.g., “fix”, “bug”, “patch”) to find bug-fixing commits.
- Step 2: Use the SZZ Algorithm
- Determine what was changed in bug fixes.
- Produces a list of regions (“hunks”) showing differences between two revisions.
- Deleted or modified code in each hunk is treated as the location of a bug.
- Traces origin of this code to find the earlier bug-introducing changes.
Example Walkthrough
Revision 1:
- Initial creation of a function `bar`.
- Introduces a bug: `if (report == null)` (should be `!=`).
- SCM annotate shows all lines as modified in revision 1 by “kim”.
Revision 2:
- Two changes:
- Function `bar` renamed to `foo`.
- Argument changed from `report` to `report.str` in `println`.
- Annotate output shows lines 1 and 4 were last modified by “ejw” in revision 2.
Revision 3:
- Bug fix applied: changes `==` to `!=` on line 3.
- SZZ algorithm compares revisions 3 and 2, identifying line 3 as modified.
- Traces line 3’s origin back to revision 1 — identifying the bug-introducing change.
3.3 Intelligent Software Engineering: Cross-Project Prediction (HDP)
Heterogeneous Defect Prediction (HDP)
- Based on the work by Nam and Kim (2015).
- Motivation: Metrics used for defect prediction often differ across projects.
- Goal: Address the metric mismatching problem across projects (heterogeneous settings).
- Classifier agnostic — can be used with any machine learning model.
HDP Architecture
Metric Selection in Source Datasets
- Uses well-known feature selection methods:
- Gain ratio
- Chi-square
- Relief-F
- Significance attribute evaluation
- Empirical testing used to choose the best approach.
- Top 15% metrics per source project are selected.
- Metric mismatching arises because each project may prioritize different metrics.
Matching Source and Target Metrics
- Key Steps:
- Pair all metrics from source and target projects.
- Remove poorly matched metrics based on a cutoff threshold for matching scores.
- Apply maximum weighted bipartite matching to select the best group of matched metric pairs:
- Goal: Maximize sum of matching scores.
- Ensure no duplicated metrics are selected.
- Example:
- 2 source metrics: X1, X2
- 2 target metrics: Y1, Y2
- Matching pairs: (X1,Y1), (X1,Y2), (X2,Y1), (X2,Y2)
After applying a cutoff threshold of 0.30:
- Group 1: (X1,Y1) and (X2,Y2) with total score 1.3 (=0.8+0.5)
- Group 2: (X2,Y1) with score 0.4 (Stands alone (can’t be paired with any other remaining pair without duplication)).
- Group 1 is chosen as the matched metric set.
Methods for Calculating Matching Scores
Percentile-Based Method
- Compares 9 percentiles (10th, 20th, …, 90th) between source and target metric values.
- Uses the formula:
Pij(n) = 1 - |spij(n) - bpij(n)| / bpij(n)
- spij(n): smaller percentile value
- bpij(n): bigger percentile value
- Matching score is 1 when all percentiles are identical.
Kolmogorov-Smirnov (KS) Test Method
- Non-parametric two-sample test.
- Useful when distributions are unknown or have unequal variances.
- Computes a p-value to indicate the similarity.
- Matching score derived from the p-value.
Spearman’s Rank Correlation Coefficient Method
- Measures correlation between two sets of values.
- If dataset sizes differ, randomly sample the larger set to match sizes.