Wave changes
This commit is contained in:
268
Uni/20250329200158-ise_week_4.org
Normal file
268
Uni/20250329200158-ise_week_4.org
Normal file
@@ -0,0 +1,268 @@
|
||||
:PROPERTIES:
|
||||
:ID: ebf874d9-0554-47f0-be8b-5c9a948738bf
|
||||
:END:
|
||||
#+title: ise_week_4
|
||||
#+filetags: :uni:notes:
|
||||
|
||||
|
||||
|
||||
* DONE Different Coverage Metrics and Branch Concepts (pages 6-12)
|
||||
* DONE Evolutionary Algorithm (pages 24-32)
|
||||
* DONE Test Case Generation (EvoSuite) (pages 44-55)
|
||||
* DONE Multi/Many-objective Software Testing (Sapienz) (pages 80-97)
|
||||
|
||||
want to figure out how to translate the real world phenotype to a genotype
|
||||
* 4.2 Evolutionary Algorithms - Intelligent Software Engineering
|
||||
|
||||
** 1. Illustrative Optimization Problem
|
||||
|
||||
- Problem: Maximize the objective function \( f(x) = x^2 \)
|
||||
- Design variable: \( x \in \{-15, -14, ..., 0, 1, ..., 15\} \)
|
||||
- Search space: All integers between -15 and 15 inclusive
|
||||
- Objective function: \( f(x) = x^2 \), to be maximized
|
||||
- Constraints: None
|
||||
- This problem is simple and allows us to demonstrate the application of evolutionary algorithms without involving additional complexity from constraints.
|
||||
|
||||
** 2. Representation
|
||||
|
||||
Evolutionary algorithms operate on representations of solutions called genotypes, which map to actual solutions (phenotypes). The choice of representation is crucial and problem-dependent.
|
||||
|
||||
*** Binary Representation
|
||||
|
||||
- The solution is represented as a fixed-length binary string.
|
||||
- For the example problem (maximizing \( f(x) = x^2 \)), we use a 5-bit binary representation:
|
||||
- The first bit indicates the sign of \( x \): 0 for positive, 1 for negative.
|
||||
- The remaining bits represent the magnitude in binary.
|
||||
- The genotype space is \( \{0,1\}^L \), where L is the length of the binary string.
|
||||
|
||||
*** Other Common Representations
|
||||
|
||||
- **Binary**: Suitable for many simple problems.
|
||||
- **Integer**: Useful for categorical or discrete variables (e.g., car brands such as Toyota, Volkswagen, etc.).
|
||||
- **Floating Point**: Used for problems with continuous variables. For example, optimizing \( f(x_1, x_2) = x_1 + x_2 \), where \( x_1, x_2 \in [0,1] \).
|
||||
- **Permutations**: Suitable for ordering problems like the Traveling Salesman Problem.
|
||||
- **Matrices**: Employed in more complex problems such as staff allocation or scheduling.
|
||||
|
||||
** 3. Evolutionary Algorithm Steps
|
||||
|
||||
The typical steps in an evolutionary algorithm include:
|
||||
|
||||
1. **Initialization**:
|
||||
- Start with a randomly generated population of candidate solutions.
|
||||
- Ensure a diverse set of individuals to explore the search space effectively.
|
||||
- Optionally include known solutions or use heuristics to seed the initial population.
|
||||
|
||||
2. **Evaluation**:
|
||||
- Each individual is evaluated using a fitness function.
|
||||
- The fitness function quantifies how well an individual performs with respect to the problem objective.
|
||||
|
||||
3. **Main Loop** (repeats until termination condition is met):
|
||||
a. **Selection**:
|
||||
- Select parent individuals based on their fitness.
|
||||
- Higher fitness individuals have a higher chance of being selected.
|
||||
|
||||
b. **Recombination (Crossover)**:
|
||||
- Combine selected parents to produce new offspring.
|
||||
- Occurs with probability \( P_c \) (crossover probability).
|
||||
|
||||
c. **Mutation**:
|
||||
- Randomly alter offspring genes to maintain diversity.
|
||||
- Occurs with probability \( P_m \) (mutation probability).
|
||||
|
||||
d. **Evaluation of Offspring**:
|
||||
- Assess the fitness of each newly created individual.
|
||||
|
||||
e. **Survivor Selection**:
|
||||
- Decide which individuals (from parents and offspring) will make up the next generation.
|
||||
- Can use various strategies like elitism or generational replacement.
|
||||
|
||||
** 4. Fitness Function
|
||||
|
||||
- The fitness function is derived from the problem’s objective or quality function.
|
||||
- It assigns a single real-valued score to each individual (phenotype).
|
||||
- The function reflects the degree to which a solution meets the desired criteria.
|
||||
- Typically, the aim is to **maximize** fitness.
|
||||
- If the problem is better posed as a minimization task, it can be transformed accordingly (e.g., minimizing \( f(x) \) is equivalent to maximizing \( -f(x) \)).
|
||||
|
||||
* 4.3 Test Case Generation using EvoSuite - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction to EvoSuite
|
||||
|
||||
- EvoSuite is a tool developed by Fraser and Arcuri (2011) for automated test case generation.
|
||||
- It generates whole test suites (not just individual test cases) for a given software system.
|
||||
- The tool accepts a list of input classes to be tested and produces corresponding JUnit test case code.
|
||||
- It leverages **genetic algorithms**, a form of evolutionary computation, to evolve effective test suites.
|
||||
|
||||
** 2. Motivation and Limitations of Traditional Methods
|
||||
|
||||
- Conventional test generation tools typically focus on **single coverage goals** (e.g., a single program branch).
|
||||
- Assumes:
|
||||
- All coverage goals are equally important.
|
||||
- All goals are equally difficult to reach.
|
||||
- Goals are independent of each other.
|
||||
- These assumptions are problematic:
|
||||
- The sequence in which goals are selected can significantly affect the quality of the resulting test suite.
|
||||
- Interdependencies among goals are often ignored.
|
||||
|
||||
*** Solution:
|
||||
- Generate **whole test suites** rather than isolated test cases.
|
||||
- Takes into account relationships between methods/classes.
|
||||
|
||||
** 3. Architecture and Representation
|
||||
|
||||
*** Test Suite Representation
|
||||
|
||||
- A test suite \( T \) consists of multiple test cases.
|
||||
- Each **test case** is a sequence of statements of varying types and lengths.
|
||||
- The total length of a test suite is the sum of the lengths of its individual test cases.
|
||||
|
||||
*** Statement Types in Test Cases
|
||||
|
||||
1. **Primitive statements**: Initialize basic types (e.g., `int var0 = 54`)
|
||||
2. **Constructor statements**: Create new instances (e.g., `Stack var1 = new Stack()`)
|
||||
3. **Field statements**: Access object members (e.g., `int var2 = var1.size`)
|
||||
4. **Method statements**: Call methods (e.g., `int var3 = var1.pop()`)
|
||||
|
||||
** 4. Fitness Function
|
||||
|
||||
- Guides the **selection of parents** in the genetic algorithm.
|
||||
- Aims to **maximize code coverage**.
|
||||
- If two test suites achieve the same coverage, the one with fewer statements is preferred (parsimony).
|
||||
- Uses **branch coverage** as the primary metric.
|
||||
- Employs the **branch distance heuristic**:
|
||||
- Measures how close an input is to flipping a predicate’s boolean outcome.
|
||||
|
||||
** 5. Bloat Control
|
||||
|
||||
- A known issue in Genetic Algorithms is **bloat**, where test cases grow unnecessarily large.
|
||||
- Can lead to memory exhaustion and inefficiency.
|
||||
|
||||
*** Techniques Used:
|
||||
- Set limits:
|
||||
- Maximum number of test cases \( N \)
|
||||
- Maximum length per test case \( L \)
|
||||
- Discard offspring that do not provide improved coverage.
|
||||
|
||||
** 6. Search Operators
|
||||
|
||||
*** Crossover Operator
|
||||
|
||||
- Combines two parent test suites (P1 and P2) to generate two offspring (O1 and O2).
|
||||
- O1 = first \( a \cdot |P1| \) test cases from P1 + remaining from P2.
|
||||
- O2 = similar combination from P2 and P1.
|
||||
- Valid since test cases are independent.
|
||||
- Helps reduce difference in length between resulting test suites.
|
||||
|
||||
*** Mutation Operator
|
||||
|
||||
- Mutation is applied with a probability of \( 1/T \), where \( T \) is the number of test cases.
|
||||
- New test cases may be added with a probability \( p \), up to a maximum count \( N \).
|
||||
|
||||
*** Mutation Operations (applied with equal probability 1/3):
|
||||
|
||||
1. **Remove**:
|
||||
- Each statement \( s_i \) is deleted with probability \( 1/n \), where \( n \) is the number of statements.
|
||||
- If needed, replace deleted statements to keep test case valid.
|
||||
|
||||
2. **Change**:
|
||||
- Each statement \( s_i \) may be altered.
|
||||
- For primitives: change numeric value randomly within ±Δ.
|
||||
- For others: change to a method/field/constructor of the same type.
|
||||
|
||||
3. **Insert**:
|
||||
- A new statement is inserted at a random position in the test case.
|
||||
|
||||
** 7. Results and Evaluation
|
||||
|
||||
- Key takeaway: **EvoSuite outperforms traditional single-goal test generation tools**.
|
||||
- Reported improvement: Up to **18x better branch coverage** than single-branch strategies.
|
||||
|
||||
|
||||
* 4.4 Multi/Many-objective Software Testing with Sapienz - Intelligent Software Engineering
|
||||
|
||||
** 1. Introduction to Sapienz
|
||||
|
||||
- Sapienz is an automated software testing tool developed by Mao et al. (2016).
|
||||
- It uses evolutionary algorithms to generate test cases for Android apps.
|
||||
- Notable Achievements:
|
||||
- Tested the top 1000 most popular Google Play apps.
|
||||
- Discovered 558 unique and previously unknown app crashes.
|
||||
- Led to a commercial spinout company named **MaJiCkE**.
|
||||
- Acquired by **Facebook/Meta**.
|
||||
|
||||
- Sapienz customizes the **NSGA-II** algorithm (a multi-objective genetic algorithm) for test case generation.
|
||||
|
||||
Reference: Mao, Ke, Mark Harman, and Yue Jia. *"Sapienz: Multi-objective automated testing for android applications."* ISSTA 2016.
|
||||
|
||||
** 2. NSGA-II: An Overview
|
||||
|
||||
- NSGA-II is a Genetic Algorithm (GA) adapted for **multi-objective optimization**.
|
||||
- Key Differences from standard GA:
|
||||
- Uses **Pareto dominance** for survival selection.
|
||||
- A solution **a dominates** solution **b** if:
|
||||
- \( a_i \leq b_i \) for all objectives, and
|
||||
- \( \exists j \) such that \( a_j < b_j \)
|
||||
- A **Pareto optimal** solution is one that is not dominated by any other in the population.
|
||||
|
||||
- NSGA-II also uses:
|
||||
- **Non-dominated sorting**: Separates population into Pareto fronts.
|
||||
- **Crowding distance**: Prefers diverse solutions within the same front.
|
||||
|
||||
** 3. Representation
|
||||
|
||||
- Specific representation details were not included in the slides but are tailored to represent Android GUI interaction sequences.
|
||||
|
||||
** 4. Objective Functions in Sapienz
|
||||
|
||||
Sapienz optimizes multiple objectives simultaneously:
|
||||
|
||||
*** a. Code Coverage
|
||||
|
||||
- Types of coverage used:
|
||||
- **Statement Coverage**: Measures how many individual code statements are executed.
|
||||
- **Method Coverage**: Measures the number of methods invoked.
|
||||
- **Android Activity Coverage**: Tracks which screens (activities) of the app are accessed.
|
||||
- Example: A dialer app may include separate activities for contacts, keypad, call history, etc.
|
||||
|
||||
*** b. Test Case Length
|
||||
|
||||
- Shorter test cases are generally preferred to improve efficiency and reduce overhead.
|
||||
- Multiple slides (86–89) emphasize the importance of minimizing test length.
|
||||
|
||||
*** c. Crash Discovery
|
||||
|
||||
- The number of test cases that lead to app crashes is also a key metric.
|
||||
- Objective: Maximize the number of crash-inducing test cases.
|
||||
|
||||
** 5. Search Operators
|
||||
|
||||
*** a. Crossover
|
||||
|
||||
- Combines parts of two parent test sequences to form new offspring.
|
||||
- Details of the crossover structure are tool-specific but follow the typical GA-style recombination.
|
||||
|
||||
*** b. Mutation
|
||||
|
||||
- Mutations are applied to test cases to explore new behaviors.
|
||||
- Types of Mutation:
|
||||
|
||||
1. **High-Level Mutation**:
|
||||
- Alters the structure or intent of test sequences.
|
||||
|
||||
2. **Low-Level Mutation (Same Size)**:
|
||||
- Changes test actions without altering the sequence length.
|
||||
|
||||
3. **Low-Level Mutation (Different Size)**:
|
||||
- Adds or removes actions to vary the length of test cases.
|
||||
|
||||
4. **Low-Level Mutation (Shuffling)**:
|
||||
- Reorders existing actions in the test case.
|
||||
|
||||
** 6. Results and Observations
|
||||
|
||||
- Sapienz significantly **outperforms other automated testing tools** in terms of:
|
||||
- Number of crashes detected.
|
||||
- Coverage achieved.
|
||||
- Efficiency in test generation.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user