:PROPERTIES: :ID: ebf874d9-0554-47f0-be8b-5c9a948738bf :END: #+title: ise_week_4 #+filetags: :uni:notes: * DONE Different Coverage Metrics and Branch Concepts (pages 6-12) * DONE Evolutionary Algorithm (pages 24-32) * DONE Test Case Generation (EvoSuite) (pages 44-55) * DONE Multi/Many-objective Software Testing (Sapienz) (pages 80-97) want to figure out how to translate the real world phenotype to a genotype * 4.2 Evolutionary Algorithms - Intelligent Software Engineering ** 1. Illustrative Optimization Problem - Problem: Maximize the objective function \( f(x) = x^2 \) - Design variable: \( x \in \{-15, -14, ..., 0, 1, ..., 15\} \) - Search space: All integers between -15 and 15 inclusive - Objective function: \( f(x) = x^2 \), to be maximized - Constraints: None - This problem is simple and allows us to demonstrate the application of evolutionary algorithms without involving additional complexity from constraints. ** 2. Representation Evolutionary algorithms operate on representations of solutions called genotypes, which map to actual solutions (phenotypes). The choice of representation is crucial and problem-dependent. *** Binary Representation - The solution is represented as a fixed-length binary string. - For the example problem (maximizing \( f(x) = x^2 \)), we use a 5-bit binary representation: - The first bit indicates the sign of \( x \): 0 for positive, 1 for negative. - The remaining bits represent the magnitude in binary. - The genotype space is \( \{0,1\}^L \), where L is the length of the binary string. *** Other Common Representations - **Binary**: Suitable for many simple problems. - **Integer**: Useful for categorical or discrete variables (e.g., car brands such as Toyota, Volkswagen, etc.). - **Floating Point**: Used for problems with continuous variables. For example, optimizing \( f(x_1, x_2) = x_1 + x_2 \), where \( x_1, x_2 \in [0,1] \). - **Permutations**: Suitable for ordering problems like the Traveling Salesman Problem. - **Matrices**: Employed in more complex problems such as staff allocation or scheduling. ** 3. Evolutionary Algorithm Steps The typical steps in an evolutionary algorithm include: 1. **Initialization**: - Start with a randomly generated population of candidate solutions. - Ensure a diverse set of individuals to explore the search space effectively. - Optionally include known solutions or use heuristics to seed the initial population. 2. **Evaluation**: - Each individual is evaluated using a fitness function. - The fitness function quantifies how well an individual performs with respect to the problem objective. 3. **Main Loop** (repeats until termination condition is met): a. **Selection**: - Select parent individuals based on their fitness. - Higher fitness individuals have a higher chance of being selected. b. **Recombination (Crossover)**: - Combine selected parents to produce new offspring. - Occurs with probability \( P_c \) (crossover probability). c. **Mutation**: - Randomly alter offspring genes to maintain diversity. - Occurs with probability \( P_m \) (mutation probability). d. **Evaluation of Offspring**: - Assess the fitness of each newly created individual. e. **Survivor Selection**: - Decide which individuals (from parents and offspring) will make up the next generation. - Can use various strategies like elitism or generational replacement. ** 4. Fitness Function - The fitness function is derived from the problem’s objective or quality function. - It assigns a single real-valued score to each individual (phenotype). - The function reflects the degree to which a solution meets the desired criteria. - Typically, the aim is to **maximize** fitness. - If the problem is better posed as a minimization task, it can be transformed accordingly (e.g., minimizing \( f(x) \) is equivalent to maximizing \( -f(x) \)). * 4.3 Test Case Generation using EvoSuite - Intelligent Software Engineering ** 1. Introduction to EvoSuite - EvoSuite is a tool developed by Fraser and Arcuri (2011) for automated test case generation. - It generates whole test suites (not just individual test cases) for a given software system. - The tool accepts a list of input classes to be tested and produces corresponding JUnit test case code. - It leverages **genetic algorithms**, a form of evolutionary computation, to evolve effective test suites. ** 2. Motivation and Limitations of Traditional Methods - Conventional test generation tools typically focus on **single coverage goals** (e.g., a single program branch). - Assumes: - All coverage goals are equally important. - All goals are equally difficult to reach. - Goals are independent of each other. - These assumptions are problematic: - The sequence in which goals are selected can significantly affect the quality of the resulting test suite. - Interdependencies among goals are often ignored. *** Solution: - Generate **whole test suites** rather than isolated test cases. - Takes into account relationships between methods/classes. ** 3. Architecture and Representation *** Test Suite Representation - A test suite \( T \) consists of multiple test cases. - Each **test case** is a sequence of statements of varying types and lengths. - The total length of a test suite is the sum of the lengths of its individual test cases. *** Statement Types in Test Cases 1. **Primitive statements**: Initialize basic types (e.g., `int var0 = 54`) 2. **Constructor statements**: Create new instances (e.g., `Stack var1 = new Stack()`) 3. **Field statements**: Access object members (e.g., `int var2 = var1.size`) 4. **Method statements**: Call methods (e.g., `int var3 = var1.pop()`) ** 4. Fitness Function - Guides the **selection of parents** in the genetic algorithm. - Aims to **maximize code coverage**. - If two test suites achieve the same coverage, the one with fewer statements is preferred (parsimony). - Uses **branch coverage** as the primary metric. - Employs the **branch distance heuristic**: - Measures how close an input is to flipping a predicate’s boolean outcome. ** 5. Bloat Control - A known issue in Genetic Algorithms is **bloat**, where test cases grow unnecessarily large. - Can lead to memory exhaustion and inefficiency. *** Techniques Used: - Set limits: - Maximum number of test cases \( N \) - Maximum length per test case \( L \) - Discard offspring that do not provide improved coverage. ** 6. Search Operators *** Crossover Operator - Combines two parent test suites (P1 and P2) to generate two offspring (O1 and O2). - O1 = first \( a \cdot |P1| \) test cases from P1 + remaining from P2. - O2 = similar combination from P2 and P1. - Valid since test cases are independent. - Helps reduce difference in length between resulting test suites. *** Mutation Operator - Mutation is applied with a probability of \( 1/T \), where \( T \) is the number of test cases. - New test cases may be added with a probability \( p \), up to a maximum count \( N \). *** Mutation Operations (applied with equal probability 1/3): 1. **Remove**: - Each statement \( s_i \) is deleted with probability \( 1/n \), where \( n \) is the number of statements. - If needed, replace deleted statements to keep test case valid. 2. **Change**: - Each statement \( s_i \) may be altered. - For primitives: change numeric value randomly within ±Δ. - For others: change to a method/field/constructor of the same type. 3. **Insert**: - A new statement is inserted at a random position in the test case. ** 7. Results and Evaluation - Key takeaway: **EvoSuite outperforms traditional single-goal test generation tools**. - Reported improvement: Up to **18x better branch coverage** than single-branch strategies. * 4.4 Multi/Many-objective Software Testing with Sapienz - Intelligent Software Engineering ** 1. Introduction to Sapienz - Sapienz is an automated software testing tool developed by Mao et al. (2016). - It uses evolutionary algorithms to generate test cases for Android apps. - Notable Achievements: - Tested the top 1000 most popular Google Play apps. - Discovered 558 unique and previously unknown app crashes. - Led to a commercial spinout company named **MaJiCkE**. - Acquired by **Facebook/Meta**. - Sapienz customizes the **NSGA-II** algorithm (a multi-objective genetic algorithm) for test case generation. Reference: Mao, Ke, Mark Harman, and Yue Jia. *"Sapienz: Multi-objective automated testing for android applications."* ISSTA 2016. ** 2. NSGA-II: An Overview - NSGA-II is a Genetic Algorithm (GA) adapted for **multi-objective optimization**. - Key Differences from standard GA: - Uses **Pareto dominance** for survival selection. - A solution **a dominates** solution **b** if: - \( a_i \leq b_i \) for all objectives, and - \( \exists j \) such that \( a_j < b_j \) - A **Pareto optimal** solution is one that is not dominated by any other in the population. - NSGA-II also uses: - **Non-dominated sorting**: Separates population into Pareto fronts. - **Crowding distance**: Prefers diverse solutions within the same front. ** 3. Representation - Specific representation details were not included in the slides but are tailored to represent Android GUI interaction sequences. ** 4. Objective Functions in Sapienz Sapienz optimizes multiple objectives simultaneously: *** a. Code Coverage - Types of coverage used: - **Statement Coverage**: Measures how many individual code statements are executed. - **Method Coverage**: Measures the number of methods invoked. - **Android Activity Coverage**: Tracks which screens (activities) of the app are accessed. - Example: A dialer app may include separate activities for contacts, keypad, call history, etc. *** b. Test Case Length - Shorter test cases are generally preferred to improve efficiency and reduce overhead. - Multiple slides (86–89) emphasize the importance of minimizing test length. *** c. Crash Discovery - The number of test cases that lead to app crashes is also a key metric. - Objective: Maximize the number of crash-inducing test cases. ** 5. Search Operators *** a. Crossover - Combines parts of two parent test sequences to form new offspring. - Details of the crossover structure are tool-specific but follow the typical GA-style recombination. *** b. Mutation - Mutations are applied to test cases to explore new behaviors. - Types of Mutation: 1. **High-Level Mutation**: - Alters the structure or intent of test sequences. 2. **Low-Level Mutation (Same Size)**: - Changes test actions without altering the sequence length. 3. **Low-Level Mutation (Different Size)**: - Adds or removes actions to vary the length of test cases. 4. **Low-Level Mutation (Shuffling)**: - Reorders existing actions in the test case. ** 6. Results and Observations - Sapienz significantly **outperforms other automated testing tools** in terms of: - Number of crashes detected. - Coverage achieved. - Efficiency in test generation.