13

Career/Concepts.md

@@ -64,12 +64,18 @@ Some core programming concepts that are essential for software development. The
 - [[OWASP Top 10]]
 - [[Cross Site Scripting (XSS)]]
 - [[Output Encoding]]
+
+# Tech Talks
+
+- [[OpenShift migration Tech Talk]]
+- [[MA4 Notes]]
 # Concepts to learn and talk about (cs and maths)
 
    - [x] xor
    - [x] big O notation
    - [x] time complexity
    - [x] space complexity
+   - [x] ternary operator
    - [ ] data structures
    - [ ] algorithms
    - [ ] design patterns

Career/Database - SQL/SQL Execution Plans.md

@@ -0,0 +1,179 @@
+---
+note type:
+  - database
+  - sql
+  - note
+date: 2026-07-03
+done:
+link: https://bertwagner.com/posts/introduction-to-sql-server-execution-plans/
+---
+## What is an Execution Plan?
+
+- SQL Server is a **declarative language**:
+    - You specify _what_ data you want, not _how_ to retrieve it.
+- An **execution plan** shows:
+    - How SQL Server decides to retrieve the data.
+    - The sequence of operations (joins, scans, seeks, etc.).
+
+### Key Idea
+
+- The **Query Optimizer**:
+    - Evaluates multiple possible ways to execute a query.
+    - Chooses a plan that is **“good enough”**, not necessarily the absolute fastest.
+- Reason:
+    - Evaluating every possible plan would be too slow and impractical.
+
+### When Problems Occur
+
+- Poor performance happens when:
+    - The optimizer picks an inefficient plan.
+- Causes may include:
+    - Inaccurate statistics
+    - Incorrect assumptions about data distribution
+- Execution plans help diagnose these issues.
+
+---
+
+## Types of Execution Plan Views
+
+### 1. Text-Based Execution Plan
+
+To see an execution plan for your query, you can run `SET SHOWPLAN_ALL ON`. This will provide a text-based tree representation of the plan:
+
+`SET SHOWPLAN_ALL ON GO --Query 
+`GO SET SHOWPLAN_ALL OFF GO`
+
+![[Pasted image 20260703150027.png]]
+
+### 2. Graphical Execution Plan (Preferred)
+
+- Generated via:
+    - **“Display Estimated Execution Plan”** in SSMS
+- Features:
+    - Visual flow of query execution
+    - Easier to interpret than text-based plans
+![[Pasted image 20260703150047.png]]
+#### XML View
+
+- Access via:
+    - Right-click → “Show Execution Plan XML”
+- Contains:
+    - Detailed metadata and properties not always visible graphically
+
+---
+
+## Estimated vs Actual Execution Plans
+
+### Estimated Execution Plan
+
+- Created **before execution**
+- Based on:
+    - Table statistics
+    - Metadata
+- Does **NOT include runtime data**
+
+---
+
+### Actual Execution Plan
+
+- Created **after execution**
+- Includes:
+    - Actual number of rows processed
+    - Number of executions
+    - Runtime metrics
+
+### Key Difference
+
+|Feature|Estimated Plan|Actual Plan|
+|---|---|---|
+|When generated|Before execution|After execution|
+|Uses real data|No|Yes|
+|Structure|Same|Same|
+|Runtime metrics|No|Yes|
+
+**Important Insight:**
+
+- The plan structure is usually identical.
+- Only the **runtime statistics differ**.
+
+---
+
+## Live Query Statistics (SQL Server 2016+)
+
+### What It Does
+
+- Shows execution progress in **real time**
+- Combines:
+    - Estimated plan + live runtime data
+
+### Benefits
+
+- Helps identify **bottlenecks quickly**
+- Useful for:
+    - Long-running queries
+    - Beginners learning execution plans
+
+### Example Insight
+
+- If a **Clustered Index Scan** takes most time:
+    - Focus optimisation efforts there.
+
+---
+
+## Historical Execution Plans
+
+### 1. Plan Cache
+
+- SQL Server stores plans for reuse.
+- Access via DMV:
+    - `sys.dm_exec_query_plan`
+
+#### Key Notes
+
+- Contains:
+    - Cached (previously executed) query plans
+- Limitations:
+    - Plans may be removed from cache
+    - Typically shows **estimated plans only**
+
+---
+
+### 2. Query Store
+
+- Stores historical query performance data
+- Requirements:
+    - Must be enabled on the database
+
+#### Benefits
+
+- Allows:
+    - Retrieval of past execution plans
+    - Query performance analysis over time
+
+---
+
+## Why Execution Plans Matter
+
+- Provide visibility into:
+    - How SQL Server retrieves data
+- Essential for:
+    - **Performance tuning**
+    - Identifying inefficient operations
+    - Diagnosing slow queries
+
+---
+
+## Summary
+
+- Execution plans reveal **how SQL Server executes queries**.
+- The optimizer selects a **“good enough” plan**, not always optimal.
+- Main viewing options:
+    - Text plan
+    - Graphical plan (most useful)
+    - XML plan
+- Types of plans:
+    - **Estimated** → Pre-execution
+    - **Actual** → Includes runtime data
+- Advanced tools:
+    - **Live Query Statistics** → real-time insights
+    - **Plan Cache & Query Store** → historical analysis

Career/Database - SQL/SQL Joins.md

@@ -3,9 +3,8 @@ note type:
   - database
   - theory
 date: 2026-06-03
-done:
+done: true
 ---
-
 # The SQL JOIN Clause
 
 The JOIN clause is used to combine rows from two or more tables, based on a related column between them.
@@ -17,4 +16,6 @@ Here are the different types of JOINs in SQL:
 - RIGHT (OUTER) JOIN: Returns all rows from the right table, and only the matched rows from the left table
 - FULL (OUTER) JOIN: Returns all rows when there is a match in either the left or right table
 
-![[Pasted image 20260603221345.png]]
+![[Pasted image 20260603221345.png]]
+
+![[Pasted image 20260619152151.png]]

Career/Database - SQL/Using Joins, Constraints, Normalization, and Subqueries.md

@@ -4,7 +4,7 @@ note type:
   - database
   - theory
 date: 2026-06-03
-done:
+done: true
 link: https://app.pluralsight.com/ilx/video-courses/sql-joins-constraints-normalization-subqueries/course-overview
 ---
 # Common Aggregate Functions
@@ -771,130 +771,199 @@ Movies_Promotions
     - `*` = many
 
 
-# SQL INNER JOINs (Level 4 Notes)
+# SQL INNER JOIN — Explained with an Example
 
-## Overview
+## Example Tables
 
-- **INNER JOIN** is used to combine rows from two (or more) tables based on a related column.
-- It returns **only the matching records** between the tables (the overlapping part).
-- Common use case: retrieving related data in a **single query instead of multiple queries**.
-
-## Key Concepts
-
-### 1. Problem Without JOIN
-
-- To get reviews and corresponding movie titles:
-    1. Query reviews: SELECT review, movie_id FROM Reviews;
-    2. Use returned `movie_id`s to query movies: SELECT title FROM Movies WHERE id IN (1, 3, 4);
-- This requires **multiple queries**, which is inefficient.
-
-### 2. INNER JOIN Syntax
+### Movies
 
 ```
-SELECT *
-FROM Movies
-INNER JOIN Reviews
-ON Movies.id = Reviews.movie_id;
+id | title
+---|-------------
+1  | Don Juan
+2  | The Lost World
+3  | Peter Pan
+4  | Robin Hood
 ```
 
-#### Explanation:
-
-- `INNER JOIN Reviews`: specifies the table to join.
-- `ON Movies.id = Reviews.movie_id`: defines how rows relate:
-    - `Movies.id` = primary key
-    - `Reviews.movie_id` = foreign key
-
-### 3. Result Behaviour
-
-- Only rows with **matching keys in both tables** are returned.
-- Examples:
-    - Movies without reviews → **excluded**
-    - Reviews without movies → **excluded**
-    - A movie with multiple reviews → appears **multiple times**
-
-#### Example:
-
-- Movie **"Don Juan"** with 3 reviews → appears 3 times.
-- Movie **"Peter Pan"** with no reviews → does not appear.
-
-### 4. Order of Tables
-Both queries return the same result:
-
-`FROM Movies INNER JOIN Reviews`
-
-or
-
-`FROM Reviews INNER JOIN Movies`
-
-- Because INNER JOIN returns only **matching data**, order doesn't change the result.
-
-### 5. Selecting Specific Columns
-
-Instead of retrieving all columns (`SELECT *`), specify only what you need:
+### Reviews
 
 ```
-SELECT Movies.title, Reviews.review
-FROM Movies
-INNER JOIN Reviews
-ON Movies.id = Reviews.movie_id;
+id | movie_id | body
+---|----------|-------------------
+1  | 1        | Great movie
+2  | 1        | Loved it
+3  | 1        | Classic
+4  | 2        | Not bad
+5  | NULL     | Random review
 ```
 
-- Important: Prefix columns with table names to avoid ambiguity.
+## What INNER JOIN Does
 
-## 6. INNER JOIN Across Multiple Tables
+Returns **only rows where both tables match**.
 
-You can join more than two tables in a single query.
+- Ignores:
+    - Movies with no reviews
+    - Reviews with no valid movie
 
-### Example: Get movie title and genres
+## Basic Query
 
 ```
-SELECT Movies.title, Genres.name
-FROM Movies
-INNER JOIN Movies_Genres
-  ON Movies.id = Movies_Genres.movie_id
-INNER JOIN Genres
-  ON Movies_Genres.genre_id = Genres.id
-WHERE Movies.title = 'Peter Pan';
+SELECT m.title, r.body
+
+FROM Movies m
+
+INNER JOIN Reviews r
+
+ON m.id = r.movie_id;
 ```
-### How it works:
 
-1. Join **Movies → Movies_Genres** via `movie_id`
-2. Join **Movies_Genres → Genres** via `genre_id`
-3. Filter for `"Peter Pan"`
-
-## Visual Understanding
-
-- Think of INNER JOIN like the **intersection of two circles (Venn diagram)**:
-    - Left circle = Movies
-    - Right circle = Reviews
-    - Result = only the overlapping middle
-
-## Key Takeaways
-
-- Use **INNER JOIN** to retrieve related data from multiple tables in one query.
-- It only returns **matching rows**.
-- Always use the `ON` clause to define relationships.
-- Specify columns for cleaner results.
-- Multiple joins can be chained for more complex relationships.
-
-## Quick Example Summary
+## Result (conceptually)
 
 ```
--- Basic join
-SELECT Movies.title, Reviews.review
-FROM Movies
-INNER JOIN Reviews
-ON Movies.id = Reviews.movie_id;
-
--- Multi-table join
-SELECT Movies.title, Genres.name
-FROM Movies
-INNER JOIN Movies_Genres
-  ON Movies.id = Movies_Genres.movie_id
-INNER JOIN Genres
-  ON Movies_Genres.genre_id = Genres.id;
+Don Juan       | Great movie
+Don Juan       | Loved it
+Don Juan       | Classic
+The Lost World | Not bad
 ```
 
+## Key Observations
+
+- Only matching data is returned
+- `Don Juan` appears 3 times (3 reviews)
+- `Peter Pan` is missing (no reviews)
+- `Random review` is missing (`movie_id = NULL`)
+
+## Why Use INNER JOIN
+
+Instead of doing this:
+
+```
+-- Query 1
+
+SELECT movie_id, body FROM Reviews;
+
+-- Query 2
+
+SELECT title FROM Movies WHERE id IN (1, 2);
+```
+
+You can do it in one query:
+
+```
+SELECT m.title, r.body
+
+FROM Movies m
+
+INNER JOIN Reviews r
+
+ON m.id = r.movie_id;
+```
+
+Cleaner and more efficient.
+
+## Table Order
+
+These are equivalent:
+
+```
+FROM Movies m
+
+INNER JOIN Reviews r
+
+ON m.id = r.movie_id;
+
+FROM Reviews r
+
+INNER JOIN Movies m
+
+ON m.id = r.movie_id;
+```
+
+Because INNER JOIN only keeps matches.
+
+## Selecting Specific Columns
+
+Avoid `SELECT *`:
+
+```
+SELECT m.title, r.body
+
+FROM Movies m
+
+INNER JOIN Reviews r
+
+ON m.id = r.movie_id;
+```
+
+This keeps output clean and avoids ambiguity.
+
+
+## INNER JOIN with Multiple Tables
+
+### Example: Add Genres
+
+#### Additional Tables
+
+```
+Movies_Genres
+movie_id | genre_id
+---------|----------
+1        | 10
+1        | 11
+3        | 12
+
+Genres
+id | name
+---|---------
+10 | Drama
+11 | Romance
+12 | Fantasy
+```
+
+### Query
+
+```
+SELECT m.title, g.name
+
+FROM Movies m
+
+INNER JOIN Movies_Genres mg
+
+  ON m.id = mg.movie_id
+
+INNER JOIN Genres g
+
+  ON mg.genre_id = g.id;
+```
+
+### Result (conceptually)
+
+```
+Don Juan   | Drama
+Don Juan   | Romance
+Peter Pan  | Fantasy
+```
+
+## Mental Model
+
+Think of INNER JOIN as:
+
+- “Give me only rows that exist in **both tables**”
+
+##  Behaviour Summary
+
+| Situation             | Included? |
+| --------------------- | --------- |
+| Movie with reviews    | Yes       |
+| Movie with no reviews | No        |
+| Review with no movie  | No        |
+|                       |           |
+
+## One-Line Summary
+
+INNER JOIN = **only matching rows between tables**
+
 # SQL Aliases (Columns & Tables)
 
 ## Column Aliases
@@ -982,123 +1051,250 @@ INNER JOIN Genres g ON mg.genre_id = g.id;
 
 
 
-# SQL Outer Joins (LEFT & RIGHT)
 
-## Overview
+# SQL Outer Joins 
 
-Outer joins allow you to combine rows from two tables even when there is no match in one of them. They help ensure that you don’t lose data from one side of the relationship.
+## Example Tables
+
+### Movies
+
+```
+id | title
+---|-------------
+1  | Don Juan
+2  | The Lost World
+3  | Peter Pan
+4  | Robin Hood
+```
+
+### Reviews
+
+```
+id | movie_id | body
+---|----------|-------------------
+1  | 1        | Great movie
+2  | 1        | Loved it
+3  | 1        | Classic
+4  | 2        | Not bad
+5  | NULL     | Random review
+```
 
 ## LEFT OUTER JOIN
 
-### Purpose
+### What it means
 
-- Returns **all records from the left table** (`Movies`)
-- Returns **matching records from the right table** (`Reviews`)
-- If no match exists, the right-side columns are filled with `NULL`
+“Show me **all movies**, and any reviews they have.”
 
-### Syntax
-
-```
-SELECT *
-FROM Movies
-LEFT OUTER JOIN Reviews
-ON Movies.id = Reviews.movie_id;
-```
-
-### Key Observations
-
-- Every movie appears in the result
-- Movies with multiple reviews appear multiple times
-- Movies with **no reviews still appear**, with `NULL` values for review columns
-
-### Example Output Insight
-
-- `Don Juan` appears **3 times** (3 reviews)
-- `Peter Pan` appears **once** with no review (NULL values)
-
-## Refining the LEFT JOIN
-
-### Improvements
-
-1. Use table aliases
-2. Select only relevant columns
-3. Order results
-
-### Example Query
+### Query
 
 ```
 SELECT m.title, r.body
+
 FROM Movies m
-LEFT OUTER JOIN Reviews r
-ON m.id = r.movie_id
-ORDER BY r.id;
+
+LEFT JOIN Reviews r
+
+ON m.id = r.movie_id;
 ```
 
-### Result Behaviour
+### Result (conceptually)
 
-- Movies without reviews (e.g. `Peter Pan`) appear **last** due to ordering by `review id`
+```
+Don Juan        | Great movie
+Don Juan        | Loved it
+Don Juan        | Classic
+The Lost World  | Not bad
+Peter Pan       | NULL
+Robin Hood      | NULL
+```
+
+### Key idea
+
+- Every movie is shown
+- If a movie has no reviews → `NULL`
+- If a movie has many reviews → it repeats
+
+Example:
+
+- `Don Juan` appears 3 times
+- `Peter Pan` appears once with `NULL`
+
+### How to remember
+
+> LEFT JOIN = **“Keep everything on the left (Movies)”**
 
 ## RIGHT OUTER JOIN
 
-### Purpose
+### What it means
 
-- Returns **all records from the right table** (`Reviews`)
-- Returns **matching records from the left table** (`Movies`)
-- If no match exists, the left-side columns are filled with `NULL`
+“Show me **all reviews**, even if they don’t belong to a movie.”
 
-### Syntax
-
-```
-SELECT *
-FROM Movies
-RIGHT OUTER JOIN Reviews
-ON Movies.id = Reviews.movie_id;
-```
-
-### Scenario Highlight
-
-- Some `movie_id` values in `Reviews` are set to `NULL`
-- These reviews do not link to any movie
-
-### Key Observations
-
-- All reviews are included
-- Reviews without a corresponding movie show `NULL` in movie fields
-
-## Refining the RIGHT JOIN
-
-### Example Query
+### Query
 
 ```
 SELECT m.title, r.body
+
 FROM Movies m
-RIGHT OUTER JOIN Reviews r
-ON m.id = r.movie_id
-ORDER BY r.id;
+
+RIGHT JOIN Reviews r
+
+ON m.id = r.movie_id;
 ```
 
-### Result Behaviour
+### Result (conceptually)
 
-- All reviews listed
-- Reviews with no associated movie have `NULL` titles
+```
+Don Juan       | Great movie
+Don Juan       | Loved it
+Don Juan       | Classic
+The Lost World | Not bad
+NULL           | Random review
+```
 
-## LEFT vs RIGHT JOIN Summary
+### Key idea
 
-|Join Type|Includes All From|Missing Matches Show As|
+- Every review is shown
+- If a review has no movie → `NULL` in movie column
+
+Example:
+
+- The `Random review` has no movie → title is `NULL`
+
+### How to remember
+
+> RIGHT JOIN = **“Keep everything on the right (Reviews)”**
+
+## Side-by-Side
+
+| Situation             | LEFT JOIN           | RIGHT JOIN         |
+| --------------------- | ------------------- | ------------------ |
+| Movie with no reviews | shown (NULL review) | not shown          |
+| Review with no movie  | not shown           | shown (NULL movie) |
+
+
+## Quick Mental Model
+
+- LEFT JOIN → “I care about **Movies**”
+- RIGHT JOIN → “I care about **Reviews**”
+- `NULL` = missing relationship
+
+## One-Line Summary
+
+- LEFT JOIN → keep all rows from **first table**
+- RIGHT JOIN → keep all rows from **second table**
+
+
+# SQL Subqueries (Level 5)
+
+## Example Tables
+
+**Movies**
+
+|id|title|sales|duration|
+|---|---|---|---|
+|1|Batman|50000|120|
+|2|Superman|30000|110|
+|3|Iron Man|25000|105|
+|4|Robin Hood|45000|140|
+
+**Promotions**
+
+|id|movie_id|category|
 |---|---|---|
-|LEFT OUTER JOIN|Left table|NULLs in right columns|
-|RIGHT OUTER JOIN|Right table|NULLs in left columns|
+|1|1|non-cash|
+|2|2|cash|
+|3|4|non-cash|
 
-## Key Takeaways
+---
 
-- Use **LEFT JOIN** when you care about all records from the first (left) table
-- Use **RIGHT JOIN** when you care about all records from the second (right) table
-- `NULL` values indicate missing relationships
-- Ordering can affect where unmatched rows appear in results
+## What is a Subquery?
 
+- A **subquery** is a query nested inside another query.
+- The inner query runs first, and its result is used by the outer query.
 
-- Outer joins ensure you don’t lose unmatched data
-- LEFT JOIN = “Show everything from the left”
-- RIGHT JOIN = “Show everything from the right”
-- Useful for identifying missing relationships and incomplete data
+## Example 1: Using `IN` with a Subquery
 
+### Goal
+
+Find the **sum of sales** for movies with a **non-cash promotion**.
+
+```
+SELECT SUM(sales)
+
+FROM Movies
+
+WHERE id IN (
+
+    SELECT movie_id
+
+    FROM Promotions
+
+    WHERE category = 'non-cash'
+
+);
+```
+- Inner query returns: `1, 4`
+- Outer query sums: `50000 + 45000 = 95000`
+
+## Alternative: Using a Join
+
+```
+SELECT SUM(sales)
+
+FROM Movies
+
+INNER JOIN Promotions
+
+  ON Movies.id = Promotions.movie_id
+
+WHERE category = 'non-cash';
+```
+
+- Same result, different approach
+- Join is typically more performant
+
+## Subquery Variants
+
+```
+SELECT *
+
+FROM Movies
+
+WHERE id NOT IN (
+
+    SELECT movie_id
+
+    FROM Promotions
+
+);
+```
+
+- Returns movies with **no promotions**
+
+## Example 2: Aggregate Subquery
+
+### Goal
+
+Find movies with **above-average duration**
+
+```
+SELECT *
+
+FROM Movies
+
+WHERE duration > (
+
+    SELECT AVG(duration)
+
+    FROM Movies
+
+);
+```
+
+- Average duration = `(120 + 110 + 105 + 140) / 4 = 118.75`
+- Result: only **Robin Hood (140)**
+
+## Key Takeaway
+
+- Subqueries break problems into steps using intermediate results
+- Use them when aggregates or filtering logic can’t be expressed directly in `WHERE`

Career/Devops/DevOps Foundations Core Concepts and Fundamentals.md

@@ -0,0 +1,599 @@
+---
+note type:
+  - devops
+  - theory
+  - note
+date: 2026-06-20
+done:
+link: https://app.pluralsight.com/ilx/video-courses/devops-foundations-core-concepts-fundamentals/course-overview
+---
+# Introduction
+
+> **Epistemology** is the branch of philosophy that examines the **nature, origin, and limits of knowledge**, often referred to as the **theory of knowledge**.
+
+# Lean Software Development
+
+- Lean originates from **Sakichi Toyoda**, who focused on reducing wasted human effort after observing his mother’s weaving work.
+- Core idea: **optimise systems by respecting and improving human work**, not just machines.
+- Toyoda’s innovations in **automatic looms** laid the foundation for modern industrial efficiency.
+- His son, **Kiichiro Toyoda**, expanded into automobiles → founding **Toyota** (first car: 1936).
+- Post-WWII constraints forced Toyota to **avoid mass production** and focus on efficiency instead.
+- **Toyota Production System (TPS)**, led by Taichi Ohno, aimed for the **complete elimination of waste**.
+- Two key pillars:
+    - **Just-in-Time**: produce only what’s needed, when needed
+    - **Jidoka**: stop immediately when defects occur
+- **Andon system**: any worker can halt production to fix problems early (quality over speed).
+- Term **“Lean”** coined in 1990 to describe this high-efficiency, low-waste system.
+- Adapted to software (2003): focuses on **waste reduction, continuous improvement, and delivering value**, closely aligned with **Agile principles**.
+
+# Lean Software Development – Waste & Principles
+
+- Central idea in lean: **waste is the primary enemy**. Eliminating it leads to better outcomes.
+- Key insight: **optimising for speed alone creates waste**, but **eliminating waste naturally improves speed**.
+- In software, this effect is amplified. **Early problem detection drastically reduces cost and impact**.
+
+## Bug Example (Cost vs Timing)
+
+- Bug found by **customer** → highest cost (damage to trust/reputation).
+- Found **just before release** → still costly (delays).
+- Found in **code review** → limited impact (few people involved).
+- Found during **development/testing** → minimal cost.
+- Found via **test-first (TDD)** → often prevented entirely.
+
+## Core Concepts
+
+- **Shift-left thinking**: move detection and quality checks as early as possible.
+- **Test-first development (TDD)**:
+    - Write tests before code
+    - Forces clarity and simpler, more reliable designs
+    - Reduces the chance of defects early
+- Even though tests aren’t perfect, they **guide safer, more focused development**.
+
+## First Lean Principle
+
+- **Eliminate waste**:
+    - Fixing avoidable bugs = waste
+    - Rework and repetition = waste
+    - Human effort spent on low-value tasks = waste
+
+## Key Philosophy
+
+- Repetition of simple tasks by humans = **underutilising human potential**
+- Lean aims to **free people to do higher-value, creative work**
+
+# Lean Software Development – Poka‑Yoke & Defect Prevention
+
+## Inspection Types
+
+- **Inspection to find defects**:
+    - Happens **after the process**
+    - More costly and disruptive
+- **Inspection to prevent defects**:
+    - Happens **during or before the process**
+    - Cheaper and more effective
+
+## Poka‑Yoke (Error Prevention)
+
+- Japanese term meaning **“error avoidance”**
+- Focus: **design systems so mistakes are impossible or immediately caught**
+- Aim: **build quality into the process, not inspect it afterwards**
+
+## Everyday Examples
+
+- **UI validation** (e.g. input masks, dropdowns, live error messages)
+- **USB design**: only fits one way
+- **Electrical outlets (GFCI)**: shuts off power to prevent harm
+- **Manual cars**: must press clutch to start
+
+## Software & DevOps Applications
+
+- **Principle of least privilege**:
+    - Users only have permissions they need
+    - Prevents both malicious actions and accidental errors
+- **Protected branches**:
+    - No direct commits to deployment branches
+    - Changes go through **pull requests + reviews + automated tests**
+- **IDE checks**:
+    - Real-time compilation and error detection during coding
+
+## Test-First Development (TDD)
+
+- Write **tests before code**
+- Forces clarity about:
+    - Requirements
+    - Assumptions and expected behaviour
+- Helps:
+    - Prevent defects early
+    - Keep code simpler and more focused
+
+## Key Insight
+
+- Code contains **implicit assumptions** that are hard to debug later
+- Writing tests first makes these assumptions **explicit and verifiable**
+
+## Long-Term Value
+
+- Tests don’t just prevent initial bugs
+- They **guard against regressions** (future defects)
+
+# Lean Software Development – Certainty & Knowledge
+
+## Core Idea: Certainty Matters
+
+- Lean emphasises not just **knowing something**, but **being certain that it is true**.
+- Two separate statements:
+    - “This code works”
+    - “We are certain this code works”
+
+## Creating Certainty in Software
+
+- **Automated testing**:
+    - Creates knowledge (code works)
+    - Maintains certainty (re-run tests continuously)
+- At release level, certainty comes from accumulated evidence:
+    - Passing automated tests
+    - Code reviews
+    - Successful builds
+    - Load/performance tests
+
+## Limits of Human Testing
+
+- Human testing is **inherently limited** in software:
+    - Fixing one bug does **not guarantee stability elsewhere**
+    - Unlike physical systems, software behaviour is **non-linear and unpredictable**
+- Conclusion: **minimise reliance on manual testing**
+
+## Key Insight: Software is Uncertain by Nature
+
+- Software development is **“predictably unpredictable”**
+- Traditional planning often fails because:
+    - Required knowledge **doesn’t exist upfront**
+    - It only emerges **during development**
+
+## Lean / Agile Perspective
+
+- Lean, Agile, and DevOps:
+    - **Force teams to confront uncertainty honestly**
+    - Reject the illusion of perfect upfront planning
+- Agile is not idealistic—it’s based on **real-world experience of failure in rigid planning**
+
+## Software as Research
+
+- Software development is essentially **research work**:
+    - Problems are **“wicked”** (can’t be fully defined upfront)
+    - Understanding evolves through **iteration and experimentation**
+
+## Principle: Epistemic Humility
+
+- Accept that:
+    - Your knowledge is **incomplete and uncertain**
+- Plan and work accordingly:
+    - Short feedback loops
+    - Continuous validation
+    - Incremental delivery
+
+## Practical Implications
+
+- Build systems that:
+    - **Continuously generate evidence of correctness**
+    - Adapt as new knowledge emerges
+- Shift focus from:
+    - “Are we right?” → “How sure are we that we’re right?”
+
+# Lean Software Development – Defer Commitment 
+
+- **Defer commitment**: avoid making decisions too early when knowledge is still uncertain.
+- Early commitments (BDUF – Big Design Up Front) **reduce predictability** because key knowledge doesn’t exist yet.
+- **Software development creates knowledge over time**, so decisions improve as the project progresses.
+
+## Decision Types
+
+- **Irreversible decisions**:
+    - Hard or costly to undo
+    - Should be **delayed as long as possible**
+    - Allows better-informed choices
+- **Reversible decisions**:
+    - Easy to change later
+    - Can be made **early to maintain progress**
+
+## Key Strategy
+
+- Prioritise **reversible choices over irreversible ones** when possible.
+- Make early decisions that **keep options open**, not ones that lock you in.
+
+## Practical Examples
+
+- Architecture choices:
+    - Use modular systems so components can be swapped later (reversible)
+- Business analogy:
+    - Travel insurance or flexible bookings = **paying for reversibility**
+    - Easy returns (e.g. Amazon, Zappos) encourage decisions because they reduce risk
+
+## Takeaway
+
+- **Better decisions come from better information**, which takes time to develop
+- Lean encourages:
+    - **Delay irreversible commitments**
+    - **Keep options open**
+    - **Move forward with flexible, changeable decisions**
+
+# Lean Software Development – Deliver Fast 
+
+## Core Idea: Deliver Fast
+
+- “Fast” means both **early delivery** and **frequent delivery**.
+- Goal: **accelerate feedback and learning**.
+
+## Deliver Early
+
+- Release **small increments quickly** (e.g. end of each sprint).
+- Even simple outputs (e.g. a mock login page) help:
+    - Expose issues early
+    - Trigger useful discussions
+    - Force setup of delivery processes (CI/CD, version control)
+
+## Deliver Often
+
+- Shift from **large, infrequent releases (e.g. quarterly)** to **frequent ones (e.g. weekly)**.
+- Frequent releases:
+    - Reduce pressure to “get everything perfect”
+    - Allow unfinished features to be safely deferred
+
+## System-Level Impact
+
+- Frequent delivery forces teams to:
+    - **Automate repetitive tasks**
+    - **Simplify or eliminate unnecessary processes**
+
+## Key Principle
+
+- **“If it hurts, do it often”**:
+    - Increase frequency of painful processes (e.g. releases, testing)
+    - Forces improvement, automation, and efficiency
+
+## Practical Mindset
+
+- Identify risky/problematic areas (“what feels uncomfortable”)
+- **Tackle them early and repeatedly** to reduce uncertainty
+
+# Lean Software Development – Respect People 
+
+## Core Principle: Respect People
+
+- Lean (Toyota) places **people at the centre of systems**, unlike traditional approaches that treat workers like machines.
+- Origin influenced by **Sakichi Toyoda’s human-focused perspective** (seeing work through his mother’s experience).
+
+## What “Respect People” Really Means
+
+- Not just:
+    - Being polite
+    - Fairness or “HR-style” gestures
+- Instead, it means:
+    - **Empowering individuals**
+    - **Giving ownership and responsibility**
+    - **Trusting people to contribute and improve systems**
+
+## Key Idea
+
+- “People operate the system, not the other way around.”
+- Workers are **intelligent contributors**, not interchangeable parts.
+
+## Practical Implications
+
+- Give teams:
+    - Authority to **stop processes and fix issues**
+    - Opportunity to **suggest improvements**
+- Build systems where:
+    - **Anyone can contribute ideas**
+    - Collaboration is encouraged over rigid hierarchy
+
+## Business Impact
+
+- Human-centred systems:
+    - **Outperform top-down, rigid structures**
+    - Improve innovation and adaptability
+- Benefits include:
+    - Lower turnover
+    - Higher engagement
+    - Better overall results
+
+## Risks of Ignoring This Principle
+
+- Treating people like machines leads to:
+    - Poor morale
+    - Lack of innovation
+    - Long-term organisational failure
+# Lean Software Development – Optimising the Whole System
+
+## Core Idea: System Thinking Over Local Optimisation
+
+- Focusing on improving **one part of the system (e.g. development speed)** can harm the **overall system performance**.
+- True optimisation comes from improving the **entire value stream**, not individual components.
+
+## Example: Test-Driven Development (TDD)
+
+- TDD may:
+    - **Slow down development initially**
+- But it:
+    - Eliminates or reduces **manual testing effort**
+    - Prevents **regressions with automated tests**
+    - Creates a long-term **quality safety net**
+- Net effect: **faster, more efficient system overall**
+
+## The Problem: Suboptimisation
+
+- Optimising individual parts ≠ optimising the system
+- Common mistake:
+    - Managers focus on **visible metrics (e.g. dev time)**
+    - Ignore downstream effects (e.g. testing, defects, rework)
+
+## Key Insight (Elon Musk Quote)
+
+- “Optimising something that shouldn’t exist” = waste
+- Highlights:
+    - Even smart people fall into **misplaced optimisation**
+    - We often optimise **means instead of outcomes**
+
+## Evaluating System Components
+
+Every part of a system:
+
+1. **Adds value** → keep and improve
+2. **Hurts the system** → fix or remove
+3. **Adds no value** → eliminate
+
+## Lean Mindset
+
+- “The best part is no part. The best process is no process.”
+- Remove:
+    - Unnecessary steps
+    - Redundant processes
+    - Inefficient workflows
+
+## Practical Implications
+
+- Automate instead of manually repeating tasks
+- Eliminate processes that **don’t contribute to value**
+- Focus on:
+    - **Flow of work across the whole system**
+    - **End-to-end efficiency**, not local metrics
+
+## Takeaway
+
+- **Optimise the system, not individual parts**
+- TDD and automation may slow one step, but improve the **entire pipeline**
+- True lean improvement often means:
+    - **Removing what shouldn’t exist** rather than refining it
+
+# Lean Software Development – Waste (7 Types & System View)
+
+## Core Idea: What is Waste?
+
+- Waste = **any activity that does not add value to the end product**.
+- Important distinction:
+    - Something may look like waste **in hindsight (final product)**
+    - But may still be **necessary in the process (learning, iteration)**
+- Lean requires **careful identification of real waste vs necessary work**
+
+## The 7 Types of Waste (Lean Development)
+
+### 1. Partially Done Work
+
+- Work that is **not fully complete or deployed**
+- Creates:
+    - Hidden effort
+    - False sense of progress
+- Example:
+    - Code written without tests → _looks done but isn’t actually usable_
+
+### 2. Extra Features (Overproduction)
+
+- Features built **too early or unnecessarily (“just in case”)**
+- Equivalent to **unused inventory** in manufacturing
+- Root cause:
+    - Poor adaptability / fear of future change
+- Lean approach:
+    - Build **just in time**, not “just in case”
+
+### 3. Relearning
+
+- Re-acquiring knowledge that was **already known before**
+- Causes:
+    - Poor documentation
+    - Staff turnover
+    - Knowledge silos
+- Mitigation:
+    - Documentation, automation, simpler design
+    - Cross-training team members
+
+**Example**
+
+- Losing key developers → project knowledge lost → costly recovery
+
+### 4. Task Switching
+
+- Switching between tasks reduces efficiency significantly
+- Cognitive penalty ≈ **40% productivity loss**
+- Humans **cannot truly multitask**
+- Impact:
+    - Slower delivery
+    - More errors
+
+### 5. Delays
+
+- Idle time waiting for:
+    - Decisions
+    - Information
+    - Approvals
+- Causes:
+    - Centralised authority
+    - Poor communication (silos)
+- Lean solution:
+    - **Empower teams, improve access to information**
+
+### 6. Defects
+
+- Errors that require **rework and disruption**
+- Effects:
+    - Task switching
+    - Relearning context
+    - Delays and wasted effort
+- Worse when found late:
+    - May require multiple people
+    - May be harder to fix
+
+### 7. Handoffs (implicit in examples)
+
+- Knowledge transfer between people/teams
+- Risk:
+    - Miscommunication
+    - Loss of context
+- Example:
+    - Poorly transferred project → failure after staff leave
+
+## Key Insight: Wastes Are Connected
+
+- Each waste often results from **missing lean principles**:
+    - Lack of **knowledge creation** → relearning
+    - Lack of **respect for people** → delays
+    - Poor **defect prevention** → defects + rework
+
+## System-Level Impact of Defects
+
+- Defects amplify multiple wastes:
+    
+    - Trigger **task switching**
+    - Require **relearning context**
+    - Create **delays**
+    - Increase **partially done work**
+- Extreme case:
+    
+    - Original developer unavailable → severe handoff waste
+
+## Real-World Example Insight
+
+- Loss of key employees caused:
+    - Knowledge loss
+    - Project failure
+    - Major financial loss
+- Preventive actions could include:
+    - Better knowledge capture (recordings, docs)
+    - Cross-training team members
+    - Retention incentives
+
+## Takeaway
+
+- Waste is **any non-value-adding effort across the system**
+- Most waste comes from:
+    - Poor knowledge management
+    - Late feedback
+    - Weak processes and structure
+- **Defects are a critical signal**:
+    - If defect rates are high, other wastes are likely unmanaged
+
+# Lean Software Development – Practical Application (DevOps)
+
+## Core Idea: Focus on Impact (Pareto Principle)
+
+- Use **Pareto (80/20 analysis)**:
+    - **80% of problems come from 20% of causes**
+- In DevOps, biggest win:
+    - **Reduce defect rate first**
+- Focus on the **“bad neighbourhood” in code**:
+    - Areas everyone avoids → highest impact if fixed
+- Approach:
+    - **Go where the pain is (fear-driven development)**
+
+## Practical Strategy to Reduce Waste
+
+### 1. Target High-Impact Problems First
+
+- Identify:
+    - Bug hotspots
+    - Fragile systems
+- Fixing these yields **disproportionate improvements**
+
+**Example**
+
+- A single unstable module causing most outages → prioritise fixing it
+
+### 2. Improve Developer Experience (Respect People)
+
+- If developers are:
+    - Slow
+    - Frustrated
+    - Error-prone
+- That is a **system failure**, not a people problem
+
+**Example**
+
+- Inability to debug locally:
+    - Leads to slow, painful development
+    - Fixing this improves speed and quality immediately
+
+### 3. Prioritise User Pain (Internal Customers)
+
+- Treat internal teams (e.g. support engineers) as **customers**
+- Fix their tooling issues first
+
+**Example**
+
+- Improving admin tools → enables support to resolve issues faster → better customer outcomes
+
+### 4. Reduce Handoffs (Cross-Training)
+
+- Train multiple people to perform key tasks
+- Avoid reliance on individuals
+
+**Example**
+
+- Rotate responsibility for complex processes → spreads knowledge → reduces risk
+
+### 5. Simplify and Standardise Systems
+
+- Look for duplication:
+    - Similar processes/tools doing the same job
+- Consolidate into **one flexible system**
+
+**Example**
+
+- Replace multiple build pipelines with **one parameterised build system**
+
+### 6. Automate Repetition
+
+- Rule:
+    - If you do something **twice**, consider automation before the third time
+- Benefits:
+    - Reduces waste
+    - Improves consistency
+    - Frees up human effort
+
+**Example**
+
+- Automate deployments, testing, or data imports
+
+## Key Lean Connections
+
+- **Reduce defects** → biggest system-wide gain
+- **Respect people** → improve how work actually gets done
+- **Eliminate waste** → focus on high-value activities
+- **Create knowledge** → through automation, documentation, and sharing
+
+## Takeaway
+
+- Start with **where the pain is highest**, not what’s easiest
+- Improve systems by:
+    - Fixing bottlenecks
+    - Empowering people
+    - Automating repetitive work
+- Lean DevOps isn’t about doing small improvements everywhere—it’s about **making the biggest difference first**
+
+## Summary
+
+- Use **80/20 thinking** to prioritise work
+- Focus on:
+    - High-defect areas
+    - Developer productivity
+    - Tool usability
+- Standardise, automate, and cross-train to reduce waste
+- Always aim to **improve the whole system, not just individual parts**

Career/Microlise/121 - 02.07.2026.md

@@ -0,0 +1,4 @@
+- Look on SQL Execution plans
+- https://www.jetbrains.com/help/dotmemory/How_to_Find_a_Memory_Leak.html
+
+

Career/Microlise/Competencies/Synchronous vs Asynchronous Dataflow Design.md

@@ -0,0 +1,440 @@
+# Competency Evidence: Synchronous vs Asynchronous Dataflow Design
+
+**Competency:** Discerning between synchronous and asynchronous programming to improve dataflow choices within an application.
+
+**Author:** Zaine Qayyum  
+**Team:** Osmosis  
+**Date:** June 2026  
+**Primary artefact:** Dora Orchestrator Airflow pipeline (`DoraOrchestrator` DAG)
+
+---
+
+## 1. Competency Statement
+
+### Value of achieving this competency
+
+Being able to discern between using synchronous and asynchronous programming to improve dataflow choices within your application.
+
+### Scenario mapping
+
+The competency scenario describes fetching data from multiple sources to populate a dashboard, where some data can be displayed immediately and some depends on other data being available first. The Dora Orchestrator pipeline is a direct realisation of this scenario:
+
+- **Independent data domains** (builds, deployments, releases/environments) can be collected in parallel as soon as staging is prepared.
+    
+- **Dependent data domains** (repositories → pull requests → commits; work items and threads after pull requests) must wait for upstream parquet outputs before they can run.
+    
+- **A merge barrier** ensures all staging data is complete before the datamart is updated for PowerBI consumption.
+    
+
+### My role
+
+I was the primary author and maintainer of the Dora Orchestrator pipeline, including the orchestrator DAG, all collection scripts, unit tests, and the pull requests cited in this document.
+
+---
+
+## 2. Problem Context
+
+### Business need
+
+The Osmosis team needed to surface DORA (DevOps Research and Assessment) metrics via PowerBI dashboards. Metrics are derived from multiple Azure DevOps (AZDO) and SonarQube API domains: teams, systems, repositories, pull requests, commits, work items, threads, builds, deployments, releases, and environments.
+
+### Original approach
+
+Data was queried directly from AZDO APIs inside PowerBI. This was brittle, tightly coupled, and effectively serial — each query blocked the next, with no orchestration layer to manage dependencies or parallelise independent fetches.
+
+### New approach
+
+A nightly Airflow orchestrator (`DoraOrchestrator`) collects data from AZDO and SonarQube APIs, writes to staging tables in `DoraDatabase_DEV`, executes merge stored procedures, and exposes the result to PowerBI via the [Dora DataMart](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.Academy.DoraMetrics).
+
+### Performance and dataflow constraints
+
+1. **Many independent API domains** — builds, deployments, and releases do not depend on repository metadata and can run concurrently.
+    
+2. **Strict cross-domain dependencies** — commits require pull request IDs; pull requests require repository IDs; repositories require team/system enrichment.
+    
+3. **Large payloads** — tens of thousands of records per domain; inter-task data transfer must not use Airflow's metadata database.
+    
+4. **Operational reliability** — daily 1am schedule, Slack failure alerts, and 80%+ unit test coverage across all scripts.
+    
+
+---
+
+## 3. Architecture Overview
+
+The orchestrator is defined in `Dags/DAG_Osmosis_DoraOrchestrator.py`. Collection logic lives in `Dags/OSMOSIS_DoraMetrics_Scripts/`.
+
+### DAG dependency graph
+
+```mermaid
+flowchart TD
+    setup[clear_create_directory_start]
+    truncate[truncate_staging_tables]
+    builds[builds TaskGroup]
+    systemTeams[system_teams]
+    repositories[repositories]
+    pullRequests[pull_requests]
+    commits[commits]
+    workItem[work_item]
+    threads[threads]
+    deployments[deployments TaskGroup]
+    releases[releases_and_environments TaskGroup]
+    merge[merge_staging_to_data_tables]
+    cleanup[clear_create_directory_end]
+
+    setup --> truncate
+    truncate --> builds
+    truncate --> systemTeams
+    truncate --> deployments
+    truncate --> releases
+    systemTeams --> repositories --> pullRequests --> commits --> merge
+    pullRequests --> workItem --> threads --> merge
+    builds --> merge
+    deployments --> merge
+    releases --> merge
+    merge --> cleanup
+```
+
+### Orchestrator dependency declarations
+
+The parallel and serial structure is encoded explicitly in the DAG:
+
+```310:316:Dags/DAG_Osmosis_DoraOrchestrator.py
+    clear_create_directory_start >> truncate_staging_tables_task
+    truncate_staging_tables_task >> builds  >> merge_staging_to_data_tables_task
+    truncate_staging_tables_task >> system_teams >> repositories >> pull_requests >> commits >> merge_staging_to_data_tables_task
+    pull_requests >> work_item >> threads >> merge_staging_to_data_tables_task
+    truncate_staging_tables_task >> deployments >> merge_staging_to_data_tables_task
+    truncate_staging_tables_task >> releases_and_environments >> merge_staging_to_data_tables_task
+    merge_staging_to_data_tables_task >> clear_create_directory_end
+```
+
+Airflow's scheduler uses this graph to run independent branches on separate workers concurrently, while respecting serial constraints within each branch.
+
+---
+
+## 4. Design Decisions — Parallel vs Serial
+
+|   |   |   |
+|---|---|---|
+|Dataflow path|Execution model|Rationale|
+|`builds`|Parallel from truncate|Builds are fetched from a single AZDO endpoint with continuation tokens; no dependency on teams/repos parquet|
+|`deployments`|Parallel from truncate|Independent AZDO release deployment API domain|
+|`releases_and_environments`|Parallel from truncate; fan-out inside group|One fetch produces releases, artifacts, and environments parquet files; two insert tasks run in parallel after fetch|
+|`system_teams → repositories → pull_requests → commits`|Serial chain|Each step reads parquet written by the previous step (e.g. commits reads `pull_requests.parquet`)|
+|`work_item → threads`|Serial after `pull_requests`|Work items and threads are collected after the PR domain is available|
+|`merge_staging_to_data_tables`|Barrier (waits for all branches)|Staging must be complete before merge procedures update production tables|
+|`clear_create_directory_end`|Serial after merge|Cleanup only after all data is persisted|
+
+### Parallel fan-out within releases
+
+Inside the `releases_and_environments` TaskGroup, a single fetch task fans out to two independent insert tasks:
+
+```276:278:Dags/DAG_Osmosis_DoraOrchestrator.py
+        fetch_all_releases_task  >> insert_releases_staging_task 
+    
+        fetch_all_releases_task  >> insert_environments_staging_task 
+```
+
+This mirrors the dashboard scenario: once release data is fetched, releases and environments can be inserted concurrently without waiting on each other.
+
+### Serial handoff via parquet files
+
+Downstream tasks read files produced by upstream tasks. For example, commits depend on pull request data:
+
+```19:31:Dags/OSMOSIS_DoraMetrics_Scripts/Commits.py
+def fetch_all_commits(**context):
+    collection_directory = parquet_setup(COLLECTION_NAME)
+    os.makedirs(collection_directory, exist_ok=True)
+
+    pull_requests_parquet_path = os.path.join(collection_directory, "pull_requests.parquet")
+
+    pull_requests_df = pd.read_parquet(pull_requests_parquet_path)
+    repository_id = 'repository.id'
+
+    pairs_df = pull_requests_df[['pullRequestId', repository_id]].dropna()
+    pairs_df['pullRequestId'] = pairs_df['pullRequestId'].astype(int)
+    pairs_df[repository_id] = pairs_df[repository_id].astype(str)
+    pairs = list(pairs_df.itertuples(index=False, name=None))
+```
+
+This serial constraint is intentional: commits cannot be meaningfully collected until pull requests exist.
+
+---
+
+## 5. Synchronous vs Asynchronous — Honest Tradeoff Analysis
+
+This section distinguishes two layers of concurrency in the pipeline and explains why each approach was chosen.
+
+### Layer 1: Orchestration-level concurrency (used)
+
+**What it is:** Airflow runs independent TaskGroups on separate worker processes concurrently. This is workflow parallelism, not language-level `async`/`await`.
+
+**Evidence:** Four branches fan out from `truncate_staging_tables_task` (builds, system chain, deployments, releases). The Airflow scheduler determines which tasks are runnable based on the dependency graph.
+
+**Why this was the right choice:**
+
+- Dependencies between domains are explicit and visible in the DAG graph.
+    
+- Each domain is an isolated failure unit with its own retry semantics and Slack alert.
+    
+- Process-level parallelism scales with Airflow worker pool capacity without rewriting collection scripts.
+    
+
+### Layer 2: In-task I/O (synchronous — used)
+
+**What it is:** Within each task, HTTP and database calls are blocking and sequential. There is no `async`/`await`, `asyncio`, threading, or `concurrent.futures` usage anywhere in `OSMOSIS_DoraMetrics_Scripts/`.
+
+**Evidence — synchronous per-repo API loop in pull requests:**
+
+```65:68:Dags/OSMOSIS_DoraMetrics_Scripts/PullRequests.py
+    for repo_id in repo_ids:
+        pull_requests = fetch_pull_requests(repo_id) or []
+
+        repo_url = f"https://azdo.microlise.com/MicroliseCollection/_apis/git/repositories/{repo_id}/"
+```
+
+Each `fetch_pull_requests` call blocks on `HttpHook.run()` until the AZDO API responds. Nested loops also make additional blocking calls per pull request for commit dates.
+
+**Evidence — batched but sequential work item fetches:**
+
+```45:51:Dags/OSMOSIS_DoraMetrics_Scripts/WorkItems.py
+    for batch in spilt_json_into_chunks(work_item_ids, 200):
+        ids_csv = ",".join(map(str, batch))
+        endpoint = ENDPOINT_BASE + ids_csv
+
+        response = get_api_information(WEBHOOK, endpoint, data=None, headers=None, api_name="Work Items")
+        items = extract_json(response, "value")
+        all_work_items.extend(items)
+```
+
+Batches of 200 reduce request count, but each batch is fetched sequentially.
+
+**Why synchronous in-task I/O was chosen:**
+
+1. **Airflow already parallelises at the DAG layer** — adding in-process async would overlap with worker-level concurrency without simplifying the dependency graph.
+    
+2. **Predictable API rate-limit behaviour** — sequential calls per worker are easier to reason about than concurrent requests from multiple async coroutines.
+    
+3. **Simpler error handling and testability** — each callable is a plain Python function with 80%+ unit test coverage across all domains.
+    
+4. **Bounded task scope** — each task has a clear start (read parquet / call API) and end (write parquet / insert staging), which maps cleanly to Airflow's task lifecycle.
+    
+
+### Layer 3: Language-level async (not used — consciously deferred)
+
+In-process `asyncio` or thread-pool concurrency could reduce per-task runtime for I/O-bound loops (e.g. per-repo PR fetches, per-PR commit fetches). This was not implemented because:
+
+- It would not remove the need for DAG-level dependency management between domains.
+    
+- It would complicate error handling, logging, and unit testing within each script.
+    
+- API throttling risk increases with concurrent in-task requests.
+    
+- The primary performance gain was achieved by parallelising independent domains at the orchestration layer, not by parallelising API calls within a single domain.
+    
+
+This is an example of **discernment**: recognising where concurrency adds value (independent domains) and where synchronous execution is the simpler correct choice (within a single domain's collection task).
+
+### Mapping to C# async concepts
+
+The suggested training covers C# `async`/`await` and `Task`. The conceptual mapping to this Python/Airflow pipeline is:
+
+|   |   |
+|---|---|
+|C# concept|Dora Orchestrator equivalent|
+|`await` on independent I/O|Parallel Airflow TaskGroups after truncate|
+|`await` on dependent I/O|Serial TaskGroup chain with parquet handoffs|
+|`Task.WhenAll`|Multiple branches converging at merge barrier|
+|Blocking synchronous I/O|`HttpHook.run()` loops inside PythonOperator callables|
+|Choosing async vs sync|DAG dependency design + XCom vs parquet dataflow decision (PR 24206)|
+
+---
+
+## 6. Key Pull Request Evidence
+
+### PR 23806 — Merge Dora Metrics branch into Main
+
+**Link:** [PR 23806](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/23806)
+
+**Summary:** Initial delivery of the Dora Orchestrator pipeline. Moved from direct PowerBI API queries to a distributed Airflow + datamart architecture. Introduced the orchestrator DAG with TaskGroups for each AZDO domain, staging table population, merge procedures, Slack failure alerts, and unit tests exceeding 80% coverage.
+
+**Dataflow relevance:** Established the foundational dependency structure and the principle that independent AZDO domains should be orchestrated rather than queried ad hoc.
+
+**DAG structure diagram (from PR):**  
+PR 23806 DAG structure
+
+---
+
+### PR 24206 — Changing XCom to parquets (primary evidence)
+
+**Link:** [PR 24206](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24206)  
+**Work item:** #106945
+
+**Problem (from PR description):**
+
+> This pull request makes changes mainly to the way data is being transferred across different `tasks`. As we are using `task groups` to orchestrate the flow of data, we originally used `xcom` to pull and push data between different tasks; so for example:
+> 
+> `fetch_all_builds runs -> pushes to xcom -> fetch_all_work_items_per_build then runs but pulls the data pushed to xcom`.
+> 
+> Problems with this: xcom is not suitable for large amounts of data, as the docs state: "... but they are only designed for small amounts of data; do not use them to pass around large values, like dataframes". When we first ran the DAG in the DEV environment, we ran into a memory issue, which was regarding xcom not being able to handle large amounts of data.
+
+**Solution (from PR description):**
+
+> Solution: Use `parquets` for writing and reading data that are transferred between tasks. As these are meant for dataframes and for large amounts of data, it makes sense to write to storage and not the airflow metadata database.
+> 
+> The parquet folder is created in the `shared_folder` through the task `clear_create_directory` which calls the script `ClearAndCreateCollectionDirectory`.
+
+**Dataflow relevance:** This is the clearest example of discerning between two data-passing strategies. XCom is synchronous and lightweight (metadata DB) but wrong for large payloads. Parquet files on shared storage are the correct handoff mechanism for dataframe-sized datasets between tasks. All fetch tasks were updated with `do_xcom_push=False`.
+
+**Implementation:** Shared folder path resolved via `parquet_setup()` in `Osmosis_Helper.py`; directory created by `ClearAndCreateCollectionDirectory.py` at DAG start.
+
+**DAG structure diagram (from PR):**  
+PR 24206 DAG structure
+
+---
+
+### PR 24333 — Remove the cyclic task
+
+**Link:** [PR 24333](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24333)  
+**Work item:** #108242
+
+**Problem (from PR description):**
+
+> The task `clear_create_directory` was called at the start and at the end, which caused a cycle loop. This PR removes that by creating separate tasks.
+
+**Solution:** Split into `clear_create_directory_start` and `clear_create_directory_end`, preserving setup/cleanup behaviour while making the graph acyclic.
+
+**Dataflow relevance:** A directed acyclic graph (DAG) is a prerequisite for correct parallel scheduling. A cyclic dependency would prevent Airflow from determining a valid execution order, breaking the parallel branches designed in PR 24206. This fix demonstrates understanding that dataflow graph structure directly affects concurrency.
+
+---
+
+### PR 24378 — Fixing the releases DAG
+
+**Link:** [PR 24378](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24378)  
+**Work item:** #108316
+
+**Summary:** Refined the releases/environments TaskGroup. Removed unnecessary normalisation, corrected validation from `1-m` to `m-m` cardinality (one release can have multiple artifacts), and cleaned up redundant variables.
+
+**Dataflow relevance:** Corrected the fan-out pattern where a single `fetch_all_releases` task produces data consumed by parallel `insert_releases_staging` and `insert_environments_staging` tasks. Accurate cardinality modelling ensures parallel insert tasks receive correctly shaped data.
+
+---
+
+### Additional supporting PRs
+
+|   |   |   |
+|---|---|---|
+|PR|Title|Relevance|
+|[24012](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24012)|Added a schedule for the dora dag at 1 am|Operationalised daily concurrent collection window|
+|[26618](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/26618)|Adding a conversion to GUID from null repo guid values|Data integrity fix in repository enrichment chain|
+|[29337](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/29337)|Updating the SQ portfolio links - Dora DAG|Maintained serial dependency chain for system/repo enrichment|
+|[31719](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/31719)|Enriching systems with nested SQ Portfolios|Replaced hardcoded portfolio names with SonarQube API-driven nested portfolio fetch|
+
+---
+
+## 7. Outcomes and Impact
+
+|   |   |
+|---|---|
+|Outcome|Detail|
+|Parallel collection of independent domains|Builds, deployments, and releases/environments run concurrently after staging truncate, reducing wall-clock time versus a fully serial pipeline|
+|DEV memory issue resolved|PR 24206 eliminated XCom as the inter-task transport for large dataframes, fixing the memory failure encountered on first DEV run|
+|Correct DAG scheduling|PR 24333 removed the cyclic dependency, enabling Airflow to schedule parallel branches correctly|
+|Daily reliable execution|PR 24012 added 1am cron schedule for unattended nightly collection|
+|Test coverage maintained|Unit tests in `Dags/tests/run/Osmosis_DoraMetrics_Scripts_Tests/` maintained 80%+ coverage across all refactors|
+|PowerBI-ready datamart|Merge procedures atomically promote staging data to production tables for dashboard consumption|
+
+---
+
+## 8. Reflection and Future Improvements
+
+### What I would do differently with hindsight
+
+The orchestration-layer parallelism delivered the largest performance gain for the least complexity. If per-task runtime becomes a bottleneck, the highest-impact candidates for in-task concurrency are:
+
+- **PullRequests.py** — nested per-repo, per-PR API loops (I/O-bound, embarrassingly parallel across repos)
+    
+- **Commits.py** — per-PR commit fetches (I/O-bound, parallelisable across PR pairs)
+    
+
+Any move to in-process async or thread-pool concurrency would require:
+
+1. AZDO API rate-limit analysis to avoid throttling
+    
+2. Airflow worker memory profiling under concurrent requests
+    
+3. Updated unit tests for concurrent error paths
+    
+4. Clear logging to preserve observability per domain
+    
+
+### Competency demonstration summary
+
+This work evidences the competency not by using `async`/`await` everywhere, but by making deliberate dataflow choices at two levels:
+
+1. **Where to parallelise** — independent AZDO domains via Airflow TaskGroup branches
+    
+2. **Where to serialise** — dependent domains via parquet handoffs and explicit `>>` dependencies
+    
+3. **Where to stay synchronous** — blocking API calls within tasks, because orchestration-layer concurrency already addresses the primary bottleneck
+    
+4. **Where to change transport mechanism** — XCom to parquet (PR 24206), because the synchronous metadata DB was the wrong medium for large async-safe handoffs
+    
+
+---
+
+## 9. Appendix
+
+### File inventory
+
+|   |   |
+|---|---|
+|File|Purpose|
+|`Dags/DAG_Osmosis_DoraOrchestrator.py`|Main orchestrator DAG — dependency graph, TaskGroups, merge barrier|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/SystemsTeams.py`|Teams and systems collection|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Repositories.py`|Repository fetch and SonarQube enrichment|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/PullRequests.py`|Pull request collection (sync per-repo loops)|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Commits.py`|Commit collection (reads PR parquet)|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/WorkItems.py`|Work item ID fetch and batched detail fetch|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Threads.py`|Thread collection|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Builds.py`|Build and build-work-item collection|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Deployments.py`|Deployment collection|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/ReleasesAndEnvironments.py`|Release/environment fetch with parallel inserts|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/Osmosis_Helper.py`|Shared utilities — API helpers, parquet I/O, Slack alerts|
+|`Dags/OSMOSIS_DoraMetrics_Scripts/ClearAndCreateCollectionDirectory.py`|Shared folder setup/cleanup|
+|`Dags/tests/run/Osmosis_DoraMetrics_Scripts_Tests/`|Unit tests for all collection scripts|
+
+### Pull request links
+
+|   |   |
+|---|---|
+|PR|URL|
+|23806|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/23806](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/23806)|
+|24012|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24012](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24012)|
+|24206|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24206](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24206)|
+|24333|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24333](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24333)|
+|24378|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24378](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/24378)|
+|26618|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/26618](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/26618)|
+|29337|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/29337](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/29337)|
+|31719|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/31719](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Airflow/pullrequest/31719)|
+
+### Related repositories and infrastructure
+
+|   |   |
+|---|---|
+|Resource|Link|
+|Dora DataMart|[https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.Academy.DoraMetrics](https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.Academy.DoraMetrics)|
+|Airflow deployment pipeline|[https://azdo.microlise.com/MicroliseCollection/Microlise/_release?_a=releases&view=mine&definitionId=970](https://azdo.microlise.com/MicroliseCollection/Microlise/_release?_a=releases&view=mine&definitionId=970)|
+|Connection deployment pipeline|[https://azdo.microlise.com/MicroliseCollection/Microlise/_release?_a=releases&view=mine&definitionId=225](https://azdo.microlise.com/MicroliseCollection/Microlise/_release?_a=releases&view=mine&definitionId=225)|
+|Slack alerts channel|[https://microlise.slack.com/archives/C09KD6S57CJ](https://microlise.slack.com/archives/C09KD6S57CJ)|
+
+### Suggested training
+
+**Course:** Async Programming in C#  
+**Link:** [https://www.linkedin.com/learning/async-programming-in-c-sharp?u=94030082](https://www.linkedin.com/learning/async-programming-in-c-sharp?u=94030082)  
+**Duration:** 3h 35m
+
+**Conceptual mapping:** The C# course covers when to use `async`/`await` versus blocking calls, how to compose concurrent operations with `Task.WhenAll`, and how to avoid common pitfalls with shared state. These concepts map directly to the dataflow decisions in this pipeline — the mechanism differs (Airflow orchestration + synchronous Python vs C# async tasks), but the discernment skill is the same: identify independent work that can proceed concurrently, respect dependencies that require serialisation, and choose the right transport mechanism for data between steps.
+
+---
+
+_This document is standalone evidence for the Async Programming competency. It can be submitted directly to a competency portal or pasted into Confluence._

Career/Microlise/ESS ESP/Remote-Session Script.md

@@ -0,0 +1,224 @@
+---
+note type:
+  - ess
+  - note
+date: 2026-06-11
+done:
+---
+## Reference script (TMC `Session.ps1`)
+
+### 1. What is the pool key?
+
+The **server name** string (`$serverName`).
+
+```mermaid
+flowchart LR
+    serverName["serverName: APP01"]
+    pool["remoteSessionPool hashtable"]
+    session["PSSession object"]
+
+    serverName -->|"used as key"| pool
+    pool --> session
+```
+
+Same server name → same pool entry. Different server → different entry.
+
+**Caveat for your task:** if the same server could be reached with **different credentials**, keying only on server name can return the wrong session. For a first version, matching the reference (key = server name) is fine if credentials are consistent per server.
+
+---
+
+### 2. What happens on the second call for the same server?
+
+```mermaid
+sequenceDiagram
+    participant Caller
+    participant GetRemoteSession as Get-RemoteSession
+    participant Pool
+    participant NewPSSession as New-PSSession
+
+    Caller->>GetRemoteSession: serverName = APP01
+    GetRemoteSession->>Pool: lookup APP01
+    Pool-->>GetRemoteSession: session found
+    GetRemoteSession-->>Caller: existing session
+    Note over NewPSSession: not called
+```
+
+- Pool lookup hits.
+- Logs something like “Using existing remote session”.
+- Returns the cached session.
+- **`New-PSSession` is not called** — that’s the whole point of the pool.
+
+---
+
+### 3. When is `New-PSSession` called vs skipped?
+
+| Condition | Action |
+|-----------|--------|
+| Pool has an entry for `$serverName` | Skip `New-PSSession`, return cached session |
+| Pool has **no** entry for `$serverName` | Call `New-PSSession`, store result in pool, return it |
+
+```mermaid
+flowchart TD
+    start[Get-RemoteSession called]
+    lookup{Pool contains serverName?}
+    reuse[Log: using existing session]
+    create[Log: creating session]
+    newPs[New-PSSession]
+    store[Store in pool]
+    return[Return session]
+
+    start --> lookup
+    lookup -->|yes| reuse --> return
+    lookup -->|no| create --> newPs --> store --> return
+```
+
+**Important:** `Get-RemoteSession` creates/reuses the session. **`Invoke-Command -Session $session`** is a separate step the caller does afterward (e.g. `Test-Path` on the remote box).
+
+---
+
+### 4. What happens when session creation fails? Does that fit ESS?
+
+**In the reference:** if `$session` is falsy after `New-PSSession`, it logs an error and runs **`exit`** (ends the whole PowerShell process).
+
+**In ESS capabilities:** that does **not** fit. Look at [`Remove-Folder.psm1`](deploy/scripts/modules/capabilities/Remove-Folder.psm1) — failures use **`throw`**, not `exit`.
+
+| | TMC reference | ESS pattern |
+|---|---------------|-------------|
+| Failure | `exit` | `throw "message"` |
+| Why | Script runner stops everything | Pester, pipelines, and orchestrators can catch/handle it |
+
+For your module: log with `Write-Log -Level ERROR`, then **throw** so deployment fails cleanly and tests can use `Should -Throw`.
+
+---
+
+## Design decisions (for your implementation)
+
+### Parameters
+
+Match the reference, adapted to ESS style:
+
+- **`ServerName`** (mandatory) — pool key and `New-PSSession` target.
+- **`Credential`** (optional) — pass through when present; omit when using default/current-user auth.
+
+Do **not** put remote script logic (e.g. `Test-Path`) inside `Get-RemoteSession`. Callers get a session, then run `Invoke-Command` themselves.
+
+---
+
+### Pool scope — where does the hashtable live?
+
+**At module scope** in the `.psm1`, **above** the function — not inside the function body.
+
+```mermaid
+flowchart TB
+    subgraph psm1 [Get-RemoteSession.psm1]
+        poolVar["script-scoped pool hashtable"]
+        func[Get-RemoteSession function]
+        poolVar --> func
+    end
+
+    call1[First call] --> func
+    call2[Second call] --> func
+    func --> poolVar
+```
+
+If the pool is **inside** the function, it is recreated every call and caching never works.
+
+Use **`$script:`** scope so the pool belongs to the module instance, not the global session (unless you deliberately want global — you don’t for this task).
+
+---
+
+### Naming: `Get-RemoteSession` vs `New-RemoteSession`
+
+| Verb | Meaning here |
+|------|----------------|
+| `New-*` | Always creates something new |
+| `Get-*` | Retrieves existing **or** obtains one if needed |
+
+Your function is “get from pool, or create and cache” → **`Get-RemoteSession`** is the right approved verb.
+
+---
+
+### Logging
+
+Use **`Write-Log`** (from [`Core.Logging.psm1`](deploy/scripts/modules/core/Core.Logging.psm1)), not TMC’s `Logging`.
+
+Worth logging:
+
+- **INFO** — “Using existing…” vs “Creating…”
+- **INFO** — elapsed time for **creation only** (optional but matches the reference and helps debug slow WinRM)
+- **ERROR** — before you throw on failure
+
+---
+
+### Failure behavior
+
+```mermaid
+flowchart LR
+    fail[New-PSSession fails or returns nothing]
+    log[Write-Log ERROR]
+    throwNode[throw with clear message]
+
+    fail --> log --> throwNode
+```
+
+No `exit`. Callers and Pester expect exceptions, not process termination.
+
+---
+
+### Pool key edge case (same server, different credentials)
+
+Reference keys **only on server name**. Implication:
+
+```mermaid
+flowchart TD
+    callA["Call: APP01 + CredA"] --> pool1[Pool stores APP01 → sessionA]
+    callB["Call: APP01 + CredB"] --> pool2[Pool returns sessionA]
+    pool2 --> wrong[Wrong cred — sessionA reused]
+```
+
+For learning / v1: **same as reference** is OK if each server always uses one credential set. If that’s not true later, the key would need to include something credential-related — out of scope unless your manifests require it.
+
+---
+
+## How this connects to `Invoke-Command`
+
+```mermaid
+sequenceDiagram
+    participant Step as Deployment step
+    participant Get as Get-RemoteSession
+    participant IC as Invoke-Command
+
+    Step->>Get: ServerName, Credential
+    Get-->>Step: $session
+    Step->>IC: -Session $session -ScriptBlock { Test-Path ... }
+    IC-->>Step: $fileExists
+```
+
+`Get-RemoteSession` = connection management.  
+`Invoke-Command` = run code on the remote machine.
+
+---
+
+## Testing note (relevant to your attached snippet)
+
+For “fails when session cannot be created”, assert like other capabilities:
+
+- **`{ Get-RemoteSession ... } | Should -Throw`** — proves the function throws.
+- Optionally assert **`Write-Log`** was called with `-Level ERROR`.
+
+`Should -Invoke Throw` is **not** valid Pester — `throw` is a language statement, not a mockable cmdlet. Same idea as [`Remove-Folder.Tests.ps1`](deploy/tests/modules/capabilities/Remove-Folder.Tests.ps1): use `Should -Throw` on the script block, and `Should -Invoke Write-Log` for the error log line.
+
+For pool behavior tests: mock **`New-PSSession`**, call the function twice with the same server, expect **`New-PSSession` invoked once**.
+
+---
+
+## One-line summary
+
+| Question | Answer |
+|----------|--------|
+| Pool key | Server name |
+| Second call | Return cached session; no `New-PSSession` |
+| When to create | Only when key missing from pool |
+| On failure | Reference: `exit` → **you:** `Write-Log` + `throw` |
+| Pool location | Module-level `$script:` hashtable, not inside the function |
+

Career/Tech Talks/MA4 Notes.md

@@ -0,0 +1,14 @@
+---
+note type:
+  - microlise
+  - note
+date: 2026-07-02
+done: true
+---
+
+- MA4 is a Microlise data centre / infrastructure environment. The MA4 migration project involves moving VMs, servers, and services from the legacy infrastructure to the new MA4 environment. The migration has been carried out in phases (Phase 1 covering DEV and OAT environments, with pre-prod and prod to follow).
+- SOV house to close by march next year
+- Inter-Data Centre Conncectity (IDC)
+
+ ![[Pasted image 20260625102149.png]]
+- 

Career/Tech Talks/OpenShift migration Tech Talk.md

@@ -0,0 +1,23 @@
+---
+note type:
+  - microlise
+  - note
+date: 2026-07-02
+done: true
+---
+**OpenShift migration**  
+
+- PROD workloads are going to be migrated from Sovereign House (PROD) to Powergate (GEN) clusters
+- `mms. Local` is going to eventually be dropped
+- OpenShift needs to use `*.ml.local` for ALL routes, which map directly to the routes (on HELM charts)
+- `api.microlise.com` ->`apim.ml.local`
+- `api-oat.microlise.com` -> `oat.apim.ml.local` (same for `cert`/ `uat`)
+- `apim-int.ml.local` replaces `svc.cluster.local` for internal communications (`dev.apim-int.ml.local` for DEV)
+- `live.microlise.com` -> `app.ml.local` / `api.ml.local` (same for oat/cert/uat)
+- `apim` or `live` using paths - continue with paths for routes - otherwise use subdomains
+- route migration - add new route, alongside old, test, drop the old route (once everything has been updated to point to it)
+- need to use `route` `name` in the HELM charts
+- new subdomains - use the new workflow in #help-openshift-platform once it's available - automated process
+- setup the new routes on GEN/DEV first, to confirm they're all working - then repeat on PROD
+- workload migration - process TBC - spin up new PROD workloads on GEN cluster, drop from PROD cluster
+- GEN - doesn't require change control, PROD workloads on GEN - will still require it

Career/Theory/Ternary Operator.md

@@ -0,0 +1,21 @@
+---
+note type:
+  - theory
+date: 2026-06-19
+done: true
+---
+The **ternary operator**, also known as the **conditional operator**, is a concise way to write an **if-else** statement in a single line by evaluating a **condition** and returning one of two values.  It is called "ternary" because it operates on **three operands**: the condition, the expression to execute if true, and the expression to execute if false. 
+
+The standard syntax is:
+
+```
+condition ? expression_if_true : expression_if_false
+```
+
+Key characteristics include:
+
+- **Short-Circuit Evaluation**: Only the expression corresponding to the evaluated condition is executed. 
+- **Readability**: It enhances code brevity for simple logic but can reduce maintainability if nested excessively. 
+- **Language Support**: It is widely supported in C-like languages (C, C++, Java, JavaScript, C#) and has variations in others (e.g., Python uses `value_if_true if condition else value_if_false`). 
+- **Performance**: It generally has negligible performance differences compared to standard if-else blocks, as modern compilers optimize both similarly. 
+- **Association**: In most languages, the operator is **right-associative**, allowing for chained conditions similar to `if...else if...else` chains.