vault/Career/Devops/DevOps Foundations Core Concepts and Fundamentals.md

note type, date, done, link

note type	date	done	link
devops theory note	2026-06-20		https://app.pluralsight.com/ilx/video-courses/devops-foundations-core-concepts-fundamentals/course-overview

Introduction

Epistemology is the branch of philosophy that examines the nature, origin, and limits of knowledge, often referred to as the theory of knowledge.

Lean Software Development

• Lean originates from Sakichi Toyoda, who focused on reducing wasted human effort after observing his mother’s weaving work.
• Core idea: optimise systems by respecting and improving human work, not just machines.
• Toyoda’s innovations in automatic looms laid the foundation for modern industrial efficiency.
• His son, Kiichiro Toyoda, expanded into automobiles → founding Toyota (first car: 1936).
• Post-WWII constraints forced Toyota to avoid mass production and focus on efficiency instead.
• Toyota Production System (TPS), led by Taichi Ohno, aimed for the complete elimination of waste.
• Two key pillars:
  • Just-in-Time: produce only what’s needed, when needed
  • Jidoka: stop immediately when defects occur
• Andon system: any worker can halt production to fix problems early (quality over speed).
• Term “Lean” coined in 1990 to describe this high-efficiency, low-waste system.
• Adapted to software (2003): focuses on waste reduction, continuous improvement, and delivering value, closely aligned with Agile principles.

Lean Software Development – Waste & Principles

• Central idea in lean: waste is the primary enemy. Eliminating it leads to better outcomes.
• Key insight: optimising for speed alone creates waste, but eliminating waste naturally improves speed.
• In software, this effect is amplified. Early problem detection drastically reduces cost and impact.

Bug Example (Cost vs Timing)

• Bug found by customer → highest cost (damage to trust/reputation).
• Found just before release → still costly (delays).
• Found in code review → limited impact (few people involved).
• Found during development/testing → minimal cost.
• Found via test-first (TDD) → often prevented entirely.

Core Concepts

• Shift-left thinking: move detection and quality checks as early as possible.
• Test-first development (TDD):
  • Write tests before code
  • Forces clarity and simpler, more reliable designs
  • Reduces the chance of defects early
• Even though tests aren’t perfect, they guide safer, more focused development.

First Lean Principle

• Eliminate waste:
  • Fixing avoidable bugs = waste
  • Rework and repetition = waste
  • Human effort spent on low-value tasks = waste

Key Philosophy

• Repetition of simple tasks by humans = underutilising human potential
• Lean aims to free people to do higher-value, creative work

Lean Software Development – Poka‑Yoke & Defect Prevention

Inspection Types

• Inspection to find defects:
  • Happens after the process
  • More costly and disruptive
• Inspection to prevent defects:
  • Happens during or before the process
  • Cheaper and more effective

Poka‑Yoke (Error Prevention)

• Japanese term meaning “error avoidance”
• Focus: design systems so mistakes are impossible or immediately caught
• Aim: build quality into the process, not inspect it afterwards

Everyday Examples

• UI validation (e.g. input masks, dropdowns, live error messages)
• USB design: only fits one way
• Electrical outlets (GFCI): shuts off power to prevent harm
• Manual cars: must press clutch to start

Software & DevOps Applications

• Principle of least privilege:
  • Users only have permissions they need
  • Prevents both malicious actions and accidental errors
• Protected branches:
  • No direct commits to deployment branches
  • Changes go through pull requests + reviews + automated tests
• IDE checks:
  • Real-time compilation and error detection during coding

Test-First Development (TDD)

• Write tests before code
• Forces clarity about:
  • Requirements
  • Assumptions and expected behaviour
• Helps:
  • Prevent defects early
  • Keep code simpler and more focused

Key Insight

• Code contains implicit assumptions that are hard to debug later
• Writing tests first makes these assumptions explicit and verifiable

Long-Term Value

• Tests don’t just prevent initial bugs
• They guard against regressions (future defects)

Lean Software Development – Certainty & Knowledge

Core Idea: Certainty Matters

• Lean emphasises not just knowing something, but being certain that it is true.
• Two separate statements:
  • “This code works”
  • “We are certain this code works”

Creating Certainty in Software

• Automated testing:
  • Creates knowledge (code works)
  • Maintains certainty (re-run tests continuously)
• At release level, certainty comes from accumulated evidence:
  • Passing automated tests
  • Code reviews
  • Successful builds
  • Load/performance tests

Limits of Human Testing

• Human testing is inherently limited in software:
  • Fixing one bug does not guarantee stability elsewhere
  • Unlike physical systems, software behaviour is non-linear and unpredictable
• Conclusion: minimise reliance on manual testing

Key Insight: Software is Uncertain by Nature

• Software development is “predictably unpredictable”
• Traditional planning often fails because:
  • Required knowledge doesn’t exist upfront
  • It only emerges during development

Lean / Agile Perspective

• Lean, Agile, and DevOps:
  • Force teams to confront uncertainty honestly
  • Reject the illusion of perfect upfront planning
• Agile is not idealistic—it’s based on real-world experience of failure in rigid planning

Software as Research

• Software development is essentially research work:
  • Problems are “wicked” (can’t be fully defined upfront)
  • Understanding evolves through iteration and experimentation

Principle: Epistemic Humility

• Accept that:
  • Your knowledge is incomplete and uncertain
• Plan and work accordingly:
  • Short feedback loops
  • Continuous validation
  • Incremental delivery

Practical Implications

• Build systems that:
  • Continuously generate evidence of correctness
  • Adapt as new knowledge emerges
• Shift focus from:
  • “Are we right?” → “How sure are we that we’re right?”

Lean Software Development – Defer Commitment

• Defer commitment: avoid making decisions too early when knowledge is still uncertain.
• Early commitments (BDUF – Big Design Up Front) reduce predictability because key knowledge doesn’t exist yet.
• Software development creates knowledge over time, so decisions improve as the project progresses.

Decision Types

• Irreversible decisions:
  • Hard or costly to undo
  • Should be delayed as long as possible
  • Allows better-informed choices
• Reversible decisions:
  • Easy to change later
  • Can be made early to maintain progress

Key Strategy

• Prioritise reversible choices over irreversible ones when possible.
• Make early decisions that keep options open, not ones that lock you in.

Practical Examples

• Architecture choices:
  • Use modular systems so components can be swapped later (reversible)
• Business analogy:
  • Travel insurance or flexible bookings = paying for reversibility
  • Easy returns (e.g. Amazon, Zappos) encourage decisions because they reduce risk

Takeaway

• Better decisions come from better information, which takes time to develop
• Lean encourages:
  • Delay irreversible commitments
  • Keep options open
  • Move forward with flexible, changeable decisions

Lean Software Development – Deliver Fast

Core Idea: Deliver Fast

• “Fast” means both early delivery and frequent delivery.
• Goal: accelerate feedback and learning.

Deliver Early

• Release small increments quickly (e.g. end of each sprint).
• Even simple outputs (e.g. a mock login page) help:
  • Expose issues early
  • Trigger useful discussions
  • Force setup of delivery processes (CI/CD, version control)

Deliver Often

• Shift from large, infrequent releases (e.g. quarterly) to frequent ones (e.g. weekly).
• Frequent releases:
  • Reduce pressure to “get everything perfect”
  • Allow unfinished features to be safely deferred

System-Level Impact

• Frequent delivery forces teams to:
  • Automate repetitive tasks
  • Simplify or eliminate unnecessary processes

Key Principle

• “If it hurts, do it often”:
  • Increase frequency of painful processes (e.g. releases, testing)
  • Forces improvement, automation, and efficiency

Practical Mindset

• Identify risky/problematic areas (“what feels uncomfortable”)
• Tackle them early and repeatedly to reduce uncertainty

Lean Software Development – Respect People

Core Principle: Respect People

• Lean (Toyota) places people at the centre of systems, unlike traditional approaches that treat workers like machines.
• Origin influenced by Sakichi Toyoda’s human-focused perspective (seeing work through his mother’s experience).

What “Respect People” Really Means

• Not just:
  • Being polite
  • Fairness or “HR-style” gestures
• Instead, it means:
  • Empowering individuals
  • Giving ownership and responsibility
  • Trusting people to contribute and improve systems

Key Idea

• “People operate the system, not the other way around.”
• Workers are intelligent contributors, not interchangeable parts.

Practical Implications

• Give teams:
  • Authority to stop processes and fix issues
  • Opportunity to suggest improvements
• Build systems where:
  • Anyone can contribute ideas
  • Collaboration is encouraged over rigid hierarchy

Business Impact

• Human-centred systems:
  • Outperform top-down, rigid structures
  • Improve innovation and adaptability
• Benefits include:
  • Lower turnover
  • Higher engagement
  • Better overall results

Risks of Ignoring This Principle

• Treating people like machines leads to:
  • Poor morale
  • Lack of innovation
  • Long-term organisational failure

Lean Software Development – Optimising the Whole System

Core Idea: System Thinking Over Local Optimisation

• Focusing on improving one part of the system (e.g. development speed) can harm the overall system performance.
• True optimisation comes from improving the entire value stream, not individual components.

Example: Test-Driven Development (TDD)

• TDD may:
  • Slow down development initially
• But it:
  • Eliminates or reduces manual testing effort
  • Prevents regressions with automated tests
  • Creates a long-term quality safety net
• Net effect: faster, more efficient system overall

The Problem: Suboptimisation

• Optimising individual parts ≠ optimising the system
• Common mistake:
  • Managers focus on visible metrics (e.g. dev time)
  • Ignore downstream effects (e.g. testing, defects, rework)

Key Insight (Elon Musk Quote)

• “Optimising something that shouldn’t exist” = waste
• Highlights:
  • Even smart people fall into misplaced optimisation
  • We often optimise means instead of outcomes

Evaluating System Components

Every part of a system:

1. Adds value → keep and improve
2. Hurts the system → fix or remove
3. Adds no value → eliminate

Lean Mindset

• “The best part is no part. The best process is no process.”
• Remove:
  • Unnecessary steps
  • Redundant processes
  • Inefficient workflows

Practical Implications

• Automate instead of manually repeating tasks
• Eliminate processes that don’t contribute to value
• Focus on:
  • Flow of work across the whole system
  • End-to-end efficiency, not local metrics

Takeaway

• Optimise the system, not individual parts
• TDD and automation may slow one step, but improve the entire pipeline
• True lean improvement often means:
  • Removing what shouldn’t exist rather than refining it

Lean Software Development – Waste (7 Types & System View)

Core Idea: What is Waste?

• Waste = any activity that does not add value to the end product.
• Important distinction:
  • Something may look like waste in hindsight (final product)
  • But may still be necessary in the process (learning, iteration)
• Lean requires careful identification of real waste vs necessary work

The 7 Types of Waste (Lean Development)

1. Partially Done Work

• Work that is not fully complete or deployed
• Creates:
  • Hidden effort
  • False sense of progress
• Example:
  • Code written without tests → looks done but isn’t actually usable

2. Extra Features (Overproduction)

• Features built too early or unnecessarily (“just in case”)
• Equivalent to unused inventory in manufacturing
• Root cause:
  • Poor adaptability / fear of future change
• Lean approach:
  • Build just in time, not “just in case”

3. Relearning

• Re-acquiring knowledge that was already known before
• Causes:
  • Poor documentation
  • Staff turnover
  • Knowledge silos
• Mitigation:
  • Documentation, automation, simpler design
  • Cross-training team members

Example

• Losing key developers → project knowledge lost → costly recovery

4. Task Switching

• Switching between tasks reduces efficiency significantly
• Cognitive penalty ≈ 40% productivity loss
• Humans cannot truly multitask
• Impact:
  • Slower delivery
  • More errors

5. Delays

• Idle time waiting for:
  • Decisions
  • Information
  • Approvals
• Causes:
  • Centralised authority
  • Poor communication (silos)
• Lean solution:
  • Empower teams, improve access to information

6. Defects

• Errors that require rework and disruption
• Effects:
  • Task switching
  • Relearning context
  • Delays and wasted effort
• Worse when found late:
  • May require multiple people
  • May be harder to fix

7. Handoffs (implicit in examples)

• Knowledge transfer between people/teams
• Risk:
  • Miscommunication
  • Loss of context
• Example:
  • Poorly transferred project → failure after staff leave

Key Insight: Wastes Are Connected

• Each waste often results from missing lean principles:
  • Lack of knowledge creation → relearning
  • Lack of respect for people → delays
  • Poor defect prevention → defects + rework

System-Level Impact of Defects

• Defects amplify multiple wastes:

  • Trigger task switching
  • Require relearning context
  • Create delays
  • Increase partially done work

• Extreme case:

  • Original developer unavailable → severe handoff waste

Real-World Example Insight

• Loss of key employees caused:
  • Knowledge loss
  • Project failure
  • Major financial loss
• Preventive actions could include:
  • Better knowledge capture (recordings, docs)
  • Cross-training team members
  • Retention incentives

Takeaway

• Waste is any non-value-adding effort across the system
• Most waste comes from:
  • Poor knowledge management
  • Late feedback
  • Weak processes and structure
• Defects are a critical signal:
  • If defect rates are high, other wastes are likely unmanaged

Lean Software Development – Practical Application (DevOps)

Core Idea: Focus on Impact (Pareto Principle)

• Use Pareto (80/20 analysis):
  • 80% of problems come from 20% of causes
• In DevOps, biggest win:
  • Reduce defect rate first
• Focus on the “bad neighbourhood” in code:
  • Areas everyone avoids → highest impact if fixed
• Approach:
  • Go where the pain is (fear-driven development)

Practical Strategy to Reduce Waste

1. Target High-Impact Problems First

• Identify:
  • Bug hotspots
  • Fragile systems
• Fixing these yields disproportionate improvements

Example

• A single unstable module causing most outages → prioritise fixing it

2. Improve Developer Experience (Respect People)

• If developers are:
  • Slow
  • Frustrated
  • Error-prone
• That is a system failure, not a people problem

Example

• Inability to debug locally:
  • Leads to slow, painful development
  • Fixing this improves speed and quality immediately

3. Prioritise User Pain (Internal Customers)

• Treat internal teams (e.g. support engineers) as customers
• Fix their tooling issues first

Example

• Improving admin tools → enables support to resolve issues faster → better customer outcomes

4. Reduce Handoffs (Cross-Training)

• Train multiple people to perform key tasks
• Avoid reliance on individuals

Example

• Rotate responsibility for complex processes → spreads knowledge → reduces risk

5. Simplify and Standardise Systems

• Look for duplication:
  • Similar processes/tools doing the same job
• Consolidate into one flexible system

Example

• Replace multiple build pipelines with one parameterised build system

6. Automate Repetition

• Rule:
  • If you do something twice, consider automation before the third time
• Benefits:
  • Reduces waste
  • Improves consistency
  • Frees up human effort

Example

• Automate deployments, testing, or data imports

Key Lean Connections

• Reduce defects → biggest system-wide gain
• Respect people → improve how work actually gets done
• Eliminate waste → focus on high-value activities
• Create knowledge → through automation, documentation, and sharing

Takeaway

• Start with where the pain is highest, not what’s easiest
• Improve systems by:
  • Fixing bottlenecks
  • Empowering people
  • Automating repetitive work
• Lean DevOps isn’t about doing small improvements everywhere—it’s about making the biggest difference first

Summary

• Use 80/20 thinking to prioritise work
• Focus on:
  • High-defect areas
  • Developer productivity
  • Tool usability
• Standardise, automate, and cross-train to reduce waste
• Always aim to improve the whole system, not just individual parts