13

Career/Devops/DevOps Foundations Core Concepts and Fundamentals.md

@@ -0,0 +1,599 @@
+---
+note type:
+  - devops
+  - theory
+  - note
+date: 2026-06-20
+done:
+link: https://app.pluralsight.com/ilx/video-courses/devops-foundations-core-concepts-fundamentals/course-overview
+---
+# Introduction
+
+> **Epistemology** is the branch of philosophy that examines the **nature, origin, and limits of knowledge**, often referred to as the **theory of knowledge**.
+
+# Lean Software Development
+
+- Lean originates from **Sakichi Toyoda**, who focused on reducing wasted human effort after observing his mother’s weaving work.
+- Core idea: **optimise systems by respecting and improving human work**, not just machines.
+- Toyoda’s innovations in **automatic looms** laid the foundation for modern industrial efficiency.
+- His son, **Kiichiro Toyoda**, expanded into automobiles → founding **Toyota** (first car: 1936).
+- Post-WWII constraints forced Toyota to **avoid mass production** and focus on efficiency instead.
+- **Toyota Production System (TPS)**, led by Taichi Ohno, aimed for the **complete elimination of waste**.
+- Two key pillars:
+    - **Just-in-Time**: produce only what’s needed, when needed
+    - **Jidoka**: stop immediately when defects occur
+- **Andon system**: any worker can halt production to fix problems early (quality over speed).
+- Term **“Lean”** coined in 1990 to describe this high-efficiency, low-waste system.
+- Adapted to software (2003): focuses on **waste reduction, continuous improvement, and delivering value**, closely aligned with **Agile principles**.
+
+# Lean Software Development – Waste & Principles
+
+- Central idea in lean: **waste is the primary enemy**. Eliminating it leads to better outcomes.
+- Key insight: **optimising for speed alone creates waste**, but **eliminating waste naturally improves speed**.
+- In software, this effect is amplified. **Early problem detection drastically reduces cost and impact**.
+
+## Bug Example (Cost vs Timing)
+
+- Bug found by **customer** → highest cost (damage to trust/reputation).
+- Found **just before release** → still costly (delays).
+- Found in **code review** → limited impact (few people involved).
+- Found during **development/testing** → minimal cost.
+- Found via **test-first (TDD)** → often prevented entirely.
+
+## Core Concepts
+
+- **Shift-left thinking**: move detection and quality checks as early as possible.
+- **Test-first development (TDD)**:
+    - Write tests before code
+    - Forces clarity and simpler, more reliable designs
+    - Reduces the chance of defects early
+- Even though tests aren’t perfect, they **guide safer, more focused development**.
+
+## First Lean Principle
+
+- **Eliminate waste**:
+    - Fixing avoidable bugs = waste
+    - Rework and repetition = waste
+    - Human effort spent on low-value tasks = waste
+
+## Key Philosophy
+
+- Repetition of simple tasks by humans = **underutilising human potential**
+- Lean aims to **free people to do higher-value, creative work**
+
+# Lean Software Development – Poka‑Yoke & Defect Prevention
+
+## Inspection Types
+
+- **Inspection to find defects**:
+    - Happens **after the process**
+    - More costly and disruptive
+- **Inspection to prevent defects**:
+    - Happens **during or before the process**
+    - Cheaper and more effective
+
+## Poka‑Yoke (Error Prevention)
+
+- Japanese term meaning **“error avoidance”**
+- Focus: **design systems so mistakes are impossible or immediately caught**
+- Aim: **build quality into the process, not inspect it afterwards**
+
+## Everyday Examples
+
+- **UI validation** (e.g. input masks, dropdowns, live error messages)
+- **USB design**: only fits one way
+- **Electrical outlets (GFCI)**: shuts off power to prevent harm
+- **Manual cars**: must press clutch to start
+
+## Software & DevOps Applications
+
+- **Principle of least privilege**:
+    - Users only have permissions they need
+    - Prevents both malicious actions and accidental errors
+- **Protected branches**:
+    - No direct commits to deployment branches
+    - Changes go through **pull requests + reviews + automated tests**
+- **IDE checks**:
+    - Real-time compilation and error detection during coding
+
+## Test-First Development (TDD)
+
+- Write **tests before code**
+- Forces clarity about:
+    - Requirements
+    - Assumptions and expected behaviour
+- Helps:
+    - Prevent defects early
+    - Keep code simpler and more focused
+
+## Key Insight
+
+- Code contains **implicit assumptions** that are hard to debug later
+- Writing tests first makes these assumptions **explicit and verifiable**
+
+## Long-Term Value
+
+- Tests don’t just prevent initial bugs
+- They **guard against regressions** (future defects)
+
+# Lean Software Development – Certainty & Knowledge
+
+## Core Idea: Certainty Matters
+
+- Lean emphasises not just **knowing something**, but **being certain that it is true**.
+- Two separate statements:
+    - “This code works”
+    - “We are certain this code works”
+
+## Creating Certainty in Software
+
+- **Automated testing**:
+    - Creates knowledge (code works)
+    - Maintains certainty (re-run tests continuously)
+- At release level, certainty comes from accumulated evidence:
+    - Passing automated tests
+    - Code reviews
+    - Successful builds
+    - Load/performance tests
+
+## Limits of Human Testing
+
+- Human testing is **inherently limited** in software:
+    - Fixing one bug does **not guarantee stability elsewhere**
+    - Unlike physical systems, software behaviour is **non-linear and unpredictable**
+- Conclusion: **minimise reliance on manual testing**
+
+## Key Insight: Software is Uncertain by Nature
+
+- Software development is **“predictably unpredictable”**
+- Traditional planning often fails because:
+    - Required knowledge **doesn’t exist upfront**
+    - It only emerges **during development**
+
+## Lean / Agile Perspective
+
+- Lean, Agile, and DevOps:
+    - **Force teams to confront uncertainty honestly**
+    - Reject the illusion of perfect upfront planning
+- Agile is not idealistic—it’s based on **real-world experience of failure in rigid planning**
+
+## Software as Research
+
+- Software development is essentially **research work**:
+    - Problems are **“wicked”** (can’t be fully defined upfront)
+    - Understanding evolves through **iteration and experimentation**
+
+## Principle: Epistemic Humility
+
+- Accept that:
+    - Your knowledge is **incomplete and uncertain**
+- Plan and work accordingly:
+    - Short feedback loops
+    - Continuous validation
+    - Incremental delivery
+
+## Practical Implications
+
+- Build systems that:
+    - **Continuously generate evidence of correctness**
+    - Adapt as new knowledge emerges
+- Shift focus from:
+    - “Are we right?” → “How sure are we that we’re right?”
+
+# Lean Software Development – Defer Commitment 
+
+- **Defer commitment**: avoid making decisions too early when knowledge is still uncertain.
+- Early commitments (BDUF – Big Design Up Front) **reduce predictability** because key knowledge doesn’t exist yet.
+- **Software development creates knowledge over time**, so decisions improve as the project progresses.
+
+## Decision Types
+
+- **Irreversible decisions**:
+    - Hard or costly to undo
+    - Should be **delayed as long as possible**
+    - Allows better-informed choices
+- **Reversible decisions**:
+    - Easy to change later
+    - Can be made **early to maintain progress**
+
+## Key Strategy
+
+- Prioritise **reversible choices over irreversible ones** when possible.
+- Make early decisions that **keep options open**, not ones that lock you in.
+
+## Practical Examples
+
+- Architecture choices:
+    - Use modular systems so components can be swapped later (reversible)
+- Business analogy:
+    - Travel insurance or flexible bookings = **paying for reversibility**
+    - Easy returns (e.g. Amazon, Zappos) encourage decisions because they reduce risk
+
+## Takeaway
+
+- **Better decisions come from better information**, which takes time to develop
+- Lean encourages:
+    - **Delay irreversible commitments**
+    - **Keep options open**
+    - **Move forward with flexible, changeable decisions**
+
+# Lean Software Development – Deliver Fast 
+
+## Core Idea: Deliver Fast
+
+- “Fast” means both **early delivery** and **frequent delivery**.
+- Goal: **accelerate feedback and learning**.
+
+## Deliver Early
+
+- Release **small increments quickly** (e.g. end of each sprint).
+- Even simple outputs (e.g. a mock login page) help:
+    - Expose issues early
+    - Trigger useful discussions
+    - Force setup of delivery processes (CI/CD, version control)
+
+## Deliver Often
+
+- Shift from **large, infrequent releases (e.g. quarterly)** to **frequent ones (e.g. weekly)**.
+- Frequent releases:
+    - Reduce pressure to “get everything perfect”
+    - Allow unfinished features to be safely deferred
+
+## System-Level Impact
+
+- Frequent delivery forces teams to:
+    - **Automate repetitive tasks**
+    - **Simplify or eliminate unnecessary processes**
+
+## Key Principle
+
+- **“If it hurts, do it often”**:
+    - Increase frequency of painful processes (e.g. releases, testing)
+    - Forces improvement, automation, and efficiency
+
+## Practical Mindset
+
+- Identify risky/problematic areas (“what feels uncomfortable”)
+- **Tackle them early and repeatedly** to reduce uncertainty
+
+# Lean Software Development – Respect People 
+
+## Core Principle: Respect People
+
+- Lean (Toyota) places **people at the centre of systems**, unlike traditional approaches that treat workers like machines.
+- Origin influenced by **Sakichi Toyoda’s human-focused perspective** (seeing work through his mother’s experience).
+
+## What “Respect People” Really Means
+
+- Not just:
+    - Being polite
+    - Fairness or “HR-style” gestures
+- Instead, it means:
+    - **Empowering individuals**
+    - **Giving ownership and responsibility**
+    - **Trusting people to contribute and improve systems**
+
+## Key Idea
+
+- “People operate the system, not the other way around.”
+- Workers are **intelligent contributors**, not interchangeable parts.
+
+## Practical Implications
+
+- Give teams:
+    - Authority to **stop processes and fix issues**
+    - Opportunity to **suggest improvements**
+- Build systems where:
+    - **Anyone can contribute ideas**
+    - Collaboration is encouraged over rigid hierarchy
+
+## Business Impact
+
+- Human-centred systems:
+    - **Outperform top-down, rigid structures**
+    - Improve innovation and adaptability
+- Benefits include:
+    - Lower turnover
+    - Higher engagement
+    - Better overall results
+
+## Risks of Ignoring This Principle
+
+- Treating people like machines leads to:
+    - Poor morale
+    - Lack of innovation
+    - Long-term organisational failure
+# Lean Software Development – Optimising the Whole System
+
+## Core Idea: System Thinking Over Local Optimisation
+
+- Focusing on improving **one part of the system (e.g. development speed)** can harm the **overall system performance**.
+- True optimisation comes from improving the **entire value stream**, not individual components.
+
+## Example: Test-Driven Development (TDD)
+
+- TDD may:
+    - **Slow down development initially**
+- But it:
+    - Eliminates or reduces **manual testing effort**
+    - Prevents **regressions with automated tests**
+    - Creates a long-term **quality safety net**
+- Net effect: **faster, more efficient system overall**
+
+## The Problem: Suboptimisation
+
+- Optimising individual parts ≠ optimising the system
+- Common mistake:
+    - Managers focus on **visible metrics (e.g. dev time)**
+    - Ignore downstream effects (e.g. testing, defects, rework)
+
+## Key Insight (Elon Musk Quote)
+
+- “Optimising something that shouldn’t exist” = waste
+- Highlights:
+    - Even smart people fall into **misplaced optimisation**
+    - We often optimise **means instead of outcomes**
+
+## Evaluating System Components
+
+Every part of a system:
+
+1. **Adds value** → keep and improve
+2. **Hurts the system** → fix or remove
+3. **Adds no value** → eliminate
+
+## Lean Mindset
+
+- “The best part is no part. The best process is no process.”
+- Remove:
+    - Unnecessary steps
+    - Redundant processes
+    - Inefficient workflows
+
+## Practical Implications
+
+- Automate instead of manually repeating tasks
+- Eliminate processes that **don’t contribute to value**
+- Focus on:
+    - **Flow of work across the whole system**
+    - **End-to-end efficiency**, not local metrics
+
+## Takeaway
+
+- **Optimise the system, not individual parts**
+- TDD and automation may slow one step, but improve the **entire pipeline**
+- True lean improvement often means:
+    - **Removing what shouldn’t exist** rather than refining it
+
+# Lean Software Development – Waste (7 Types & System View)
+
+## Core Idea: What is Waste?
+
+- Waste = **any activity that does not add value to the end product**.
+- Important distinction:
+    - Something may look like waste **in hindsight (final product)**
+    - But may still be **necessary in the process (learning, iteration)**
+- Lean requires **careful identification of real waste vs necessary work**
+
+## The 7 Types of Waste (Lean Development)
+
+### 1. Partially Done Work
+
+- Work that is **not fully complete or deployed**
+- Creates:
+    - Hidden effort
+    - False sense of progress
+- Example:
+    - Code written without tests → _looks done but isn’t actually usable_
+
+### 2. Extra Features (Overproduction)
+
+- Features built **too early or unnecessarily (“just in case”)**
+- Equivalent to **unused inventory** in manufacturing
+- Root cause:
+    - Poor adaptability / fear of future change
+- Lean approach:
+    - Build **just in time**, not “just in case”
+
+### 3. Relearning
+
+- Re-acquiring knowledge that was **already known before**
+- Causes:
+    - Poor documentation
+    - Staff turnover
+    - Knowledge silos
+- Mitigation:
+    - Documentation, automation, simpler design
+    - Cross-training team members
+
+**Example**
+
+- Losing key developers → project knowledge lost → costly recovery
+
+### 4. Task Switching
+
+- Switching between tasks reduces efficiency significantly
+- Cognitive penalty ≈ **40% productivity loss**
+- Humans **cannot truly multitask**
+- Impact:
+    - Slower delivery
+    - More errors
+
+### 5. Delays
+
+- Idle time waiting for:
+    - Decisions
+    - Information
+    - Approvals
+- Causes:
+    - Centralised authority
+    - Poor communication (silos)
+- Lean solution:
+    - **Empower teams, improve access to information**
+
+### 6. Defects
+
+- Errors that require **rework and disruption**
+- Effects:
+    - Task switching
+    - Relearning context
+    - Delays and wasted effort
+- Worse when found late:
+    - May require multiple people
+    - May be harder to fix
+
+### 7. Handoffs (implicit in examples)
+
+- Knowledge transfer between people/teams
+- Risk:
+    - Miscommunication
+    - Loss of context
+- Example:
+    - Poorly transferred project → failure after staff leave
+
+## Key Insight: Wastes Are Connected
+
+- Each waste often results from **missing lean principles**:
+    - Lack of **knowledge creation** → relearning
+    - Lack of **respect for people** → delays
+    - Poor **defect prevention** → defects + rework
+
+## System-Level Impact of Defects
+
+- Defects amplify multiple wastes:
+    
+    - Trigger **task switching**
+    - Require **relearning context**
+    - Create **delays**
+    - Increase **partially done work**
+- Extreme case:
+    
+    - Original developer unavailable → severe handoff waste
+
+## Real-World Example Insight
+
+- Loss of key employees caused:
+    - Knowledge loss
+    - Project failure
+    - Major financial loss
+- Preventive actions could include:
+    - Better knowledge capture (recordings, docs)
+    - Cross-training team members
+    - Retention incentives
+
+## Takeaway
+
+- Waste is **any non-value-adding effort across the system**
+- Most waste comes from:
+    - Poor knowledge management
+    - Late feedback
+    - Weak processes and structure
+- **Defects are a critical signal**:
+    - If defect rates are high, other wastes are likely unmanaged
+
+# Lean Software Development – Practical Application (DevOps)
+
+## Core Idea: Focus on Impact (Pareto Principle)
+
+- Use **Pareto (80/20 analysis)**:
+    - **80% of problems come from 20% of causes**
+- In DevOps, biggest win:
+    - **Reduce defect rate first**
+- Focus on the **“bad neighbourhood” in code**:
+    - Areas everyone avoids → highest impact if fixed
+- Approach:
+    - **Go where the pain is (fear-driven development)**
+
+## Practical Strategy to Reduce Waste
+
+### 1. Target High-Impact Problems First
+
+- Identify:
+    - Bug hotspots
+    - Fragile systems
+- Fixing these yields **disproportionate improvements**
+
+**Example**
+
+- A single unstable module causing most outages → prioritise fixing it
+
+### 2. Improve Developer Experience (Respect People)
+
+- If developers are:
+    - Slow
+    - Frustrated
+    - Error-prone
+- That is a **system failure**, not a people problem
+
+**Example**
+
+- Inability to debug locally:
+    - Leads to slow, painful development
+    - Fixing this improves speed and quality immediately
+
+### 3. Prioritise User Pain (Internal Customers)
+
+- Treat internal teams (e.g. support engineers) as **customers**
+- Fix their tooling issues first
+
+**Example**
+
+- Improving admin tools → enables support to resolve issues faster → better customer outcomes
+
+### 4. Reduce Handoffs (Cross-Training)
+
+- Train multiple people to perform key tasks
+- Avoid reliance on individuals
+
+**Example**
+
+- Rotate responsibility for complex processes → spreads knowledge → reduces risk
+
+### 5. Simplify and Standardise Systems
+
+- Look for duplication:
+    - Similar processes/tools doing the same job
+- Consolidate into **one flexible system**
+
+**Example**
+
+- Replace multiple build pipelines with **one parameterised build system**
+
+### 6. Automate Repetition
+
+- Rule:
+    - If you do something **twice**, consider automation before the third time
+- Benefits:
+    - Reduces waste
+    - Improves consistency
+    - Frees up human effort
+
+**Example**
+
+- Automate deployments, testing, or data imports
+
+## Key Lean Connections
+
+- **Reduce defects** → biggest system-wide gain
+- **Respect people** → improve how work actually gets done
+- **Eliminate waste** → focus on high-value activities
+- **Create knowledge** → through automation, documentation, and sharing
+
+## Takeaway
+
+- Start with **where the pain is highest**, not what’s easiest
+- Improve systems by:
+    - Fixing bottlenecks
+    - Empowering people
+    - Automating repetitive work
+- Lean DevOps isn’t about doing small improvements everywhere—it’s about **making the biggest difference first**
+
+## Summary
+
+- Use **80/20 thinking** to prioritise work
+- Focus on:
+    - High-defect areas
+    - Developer productivity
+    - Tool usability
+- Standardise, automate, and cross-train to reduce waste
+- Always aim to **improve the whole system, not just individual parts**