org_roam/20250329200158-ise_week_4.org~

:PROPERTIES:
:ID:       ebf874d9-0554-47f0-be8b-5c9a948738bf
:END:
#+title: ise-week-4
#+filetags: :uni:ise:notes:



* DONE Different Coverage Metrics and Branch Concepts (pages 6-12)
* DONE Evolutionary Algorithm (pages 24-32)
* DONE Test Case Generation (EvoSuite) (pages 44-55)
* DONE Multi/Many-objective Software Testing (Sapienz) (pages 80-97)

want to figure out how to translate the real world phenotype to a genotype
* 4.2 Evolutionary Algorithms - Intelligent Software Engineering

** 1. Illustrative Optimization Problem

- Problem: Maximize the objective function \( f(x) = x^2 \)
- Design variable: \( x \in \{-15, -14, ..., 0, 1, ..., 15\} \)
- Search space: All integers between -15 and 15 inclusive
- Objective function: \( f(x) = x^2 \), to be maximized
- Constraints: None
- This problem is simple and allows us to demonstrate the application of evolutionary algorithms without involving additional complexity from constraints.

** 2. Representation

Evolutionary algorithms operate on representations of solutions called genotypes, which map to actual solutions (phenotypes). The choice of representation is crucial and problem-dependent.

*** Binary Representation

- The solution is represented as a fixed-length binary string.
- For the example problem (maximizing \( f(x) = x^2 \)), we use a 5-bit binary representation:
  - The first bit indicates the sign of \( x \): 0 for positive, 1 for negative.
  - The remaining bits represent the magnitude in binary.
- The genotype space is \( \{0,1\}^L \), where L is the length of the binary string.

*** Other Common Representations

- **Binary**: Suitable for many simple problems.
- **Integer**: Useful for categorical or discrete variables (e.g., car brands such as Toyota, Volkswagen, etc.).
- **Floating Point**: Used for problems with continuous variables. For example, optimizing \( f(x_1, x_2) = x_1 + x_2 \), where \( x_1, x_2 \in [0,1] \).
- **Permutations**: Suitable for ordering problems like the Traveling Salesman Problem.
- **Matrices**: Employed in more complex problems such as staff allocation or scheduling.

** 3. Evolutionary Algorithm Steps

The typical steps in an evolutionary algorithm include:

1. **Initialization**:
   - Start with a randomly generated population of candidate solutions.
   - Ensure a diverse set of individuals to explore the search space effectively.
   - Optionally include known solutions or use heuristics to seed the initial population.

2. **Evaluation**:
   - Each individual is evaluated using a fitness function.
   - The fitness function quantifies how well an individual performs with respect to the problem objective.

3. **Main Loop** (repeats until termination condition is met):
   a. **Selection**:
      - Select parent individuals based on their fitness.
      - Higher fitness individuals have a higher chance of being selected.

   b. **Recombination (Crossover)**:
      - Combine selected parents to produce new offspring.
      - Occurs with probability \( P_c \) (crossover probability).

   c. **Mutation**:
      - Randomly alter offspring genes to maintain diversity.
      - Occurs with probability \( P_m \) (mutation probability).

   d. **Evaluation of Offspring**:
      - Assess the fitness of each newly created individual.

   e. **Survivor Selection**:
      - Decide which individuals (from parents and offspring) will make up the next generation.
      - Can use various strategies like elitism or generational replacement.

** 4. Fitness Function

- The fitness function is derived from the problem’s objective or quality function.
- It assigns a single real-valued score to each individual (phenotype).
- The function reflects the degree to which a solution meets the desired criteria.
- Typically, the aim is to **maximize** fitness.
- If the problem is better posed as a minimization task, it can be transformed accordingly (e.g., minimizing \( f(x) \) is equivalent to maximizing \( -f(x) \)). 

* 4.3 Test Case Generation using EvoSuite - Intelligent Software Engineering

** 1. Introduction to EvoSuite

- EvoSuite is a tool developed by Fraser and Arcuri (2011) for automated test case generation.
- It generates whole test suites (not just individual test cases) for a given software system.
- The tool accepts a list of input classes to be tested and produces corresponding JUnit test case code.
- It leverages **genetic algorithms**, a form of evolutionary computation, to evolve effective test suites.

** 2. Motivation and Limitations of Traditional Methods

- Conventional test generation tools typically focus on **single coverage goals** (e.g., a single program branch).
- Assumes:
  - All coverage goals are equally important.
  - All goals are equally difficult to reach.
  - Goals are independent of each other.
- These assumptions are problematic:
  - The sequence in which goals are selected can significantly affect the quality of the resulting test suite.
  - Interdependencies among goals are often ignored.

*** Solution:
- Generate **whole test suites** rather than isolated test cases.
- Takes into account relationships between methods/classes.

** 3. Architecture and Representation

*** Test Suite Representation

- A test suite \( T \) consists of multiple test cases.
- Each **test case** is a sequence of statements of varying types and lengths.
- The total length of a test suite is the sum of the lengths of its individual test cases.

*** Statement Types in Test Cases

1. **Primitive statements**: Initialize basic types (e.g., `int var0 = 54`)
2. **Constructor statements**: Create new instances (e.g., `Stack var1 = new Stack()`)
3. **Field statements**: Access object members (e.g., `int var2 = var1.size`)
4. **Method statements**: Call methods (e.g., `int var3 = var1.pop()`)

** 4. Fitness Function

- Guides the **selection of parents** in the genetic algorithm.
- Aims to **maximize code coverage**.
- If two test suites achieve the same coverage, the one with fewer statements is preferred (parsimony).
- Uses **branch coverage** as the primary metric.
- Employs the **branch distance heuristic**:
  - Measures how close an input is to flipping a predicate’s boolean outcome.

** 5. Bloat Control

- A known issue in Genetic Algorithms is **bloat**, where test cases grow unnecessarily large.
- Can lead to memory exhaustion and inefficiency.

*** Techniques Used:
- Set limits:
  - Maximum number of test cases \( N \)
  - Maximum length per test case \( L \)
- Discard offspring that do not provide improved coverage.

** 6. Search Operators

*** Crossover Operator

- Combines two parent test suites (P1 and P2) to generate two offspring (O1 and O2).
- O1 = first \( a \cdot |P1| \) test cases from P1 + remaining from P2.
- O2 = similar combination from P2 and P1.
- Valid since test cases are independent.
- Helps reduce difference in length between resulting test suites.

*** Mutation Operator

- Mutation is applied with a probability of \( 1/T \), where \( T \) is the number of test cases.
- New test cases may be added with a probability \( p \), up to a maximum count \( N \).

*** Mutation Operations (applied with equal probability 1/3):

1. **Remove**:
   - Each statement \( s_i \) is deleted with probability \( 1/n \), where \( n \) is the number of statements.
   - If needed, replace deleted statements to keep test case valid.

2. **Change**:
   - Each statement \( s_i \) may be altered.
   - For primitives: change numeric value randomly within ±Δ.
   - For others: change to a method/field/constructor of the same type.

3. **Insert**:
   - A new statement is inserted at a random position in the test case.

** 7. Results and Evaluation

- Key takeaway: **EvoSuite outperforms traditional single-goal test generation tools**.
- Reported improvement: Up to **18x better branch coverage** than single-branch strategies.


* 4.4 Multi/Many-objective Software Testing with Sapienz - Intelligent Software Engineering

** 1. Introduction to Sapienz

- Sapienz is an automated software testing tool developed by Mao et al. (2016).
- It uses evolutionary algorithms to generate test cases for Android apps.
- Notable Achievements:
  - Tested the top 1000 most popular Google Play apps.
  - Discovered 558 unique and previously unknown app crashes.
  - Led to a commercial spinout company named **MaJiCkE**.
  - Acquired by **Facebook/Meta**.

- Sapienz customizes the **NSGA-II** algorithm (a multi-objective genetic algorithm) for test case generation.

Reference: Mao, Ke, Mark Harman, and Yue Jia. *"Sapienz: Multi-objective automated testing for android applications."* ISSTA 2016.

** 2. NSGA-II: An Overview

- NSGA-II is a Genetic Algorithm (GA) adapted for **multi-objective optimization**.
- Key Differences from standard GA:
  - Uses **Pareto dominance** for survival selection.
  - A solution **a dominates** solution **b** if:
    - \( a_i \leq b_i \) for all objectives, and
    - \( \exists j \) such that \( a_j < b_j \)
  - A **Pareto optimal** solution is one that is not dominated by any other in the population.

- NSGA-II also uses:
  - **Non-dominated sorting**: Separates population into Pareto fronts.
  - **Crowding distance**: Prefers diverse solutions within the same front.

** 3. Representation

- Specific representation details were not included in the slides but are tailored to represent Android GUI interaction sequences.

** 4. Objective Functions in Sapienz

Sapienz optimizes multiple objectives simultaneously:

*** a. Code Coverage

- Types of coverage used:
  - **Statement Coverage**: Measures how many individual code statements are executed.
  - **Method Coverage**: Measures the number of methods invoked.
  - **Android Activity Coverage**: Tracks which screens (activities) of the app are accessed.
    - Example: A dialer app may include separate activities for contacts, keypad, call history, etc.

*** b. Test Case Length

- Shorter test cases are generally preferred to improve efficiency and reduce overhead.
- Multiple slides (86–89) emphasize the importance of minimizing test length.

*** c. Crash Discovery

- The number of test cases that lead to app crashes is also a key metric.
- Objective: Maximize the number of crash-inducing test cases.

** 5. Search Operators

*** a. Crossover

- Combines parts of two parent test sequences to form new offspring.
- Details of the crossover structure are tool-specific but follow the typical GA-style recombination.

*** b. Mutation

- Mutations are applied to test cases to explore new behaviors.
- Types of Mutation:

  1. **High-Level Mutation**:
     - Alters the structure or intent of test sequences.

  2. **Low-Level Mutation (Same Size)**:
     - Changes test actions without altering the sequence length.

  3. **Low-Level Mutation (Different Size)**:
     - Adds or removes actions to vary the length of test cases.

  4. **Low-Level Mutation (Shuffling)**:
     - Reorders existing actions in the test case.

** 6. Results and Observations

- Sapienz significantly **outperforms other automated testing tools** in terms of:
  - Number of crashes detected.
  - Coverage achieved.
  - Efficiency in test generation.