org_roam/Career Concepts/20260410121927-esp_pipeline.org

:PROPERTIES:
:ID:       d66e946f-6785-4e8e-ba00-bb25a52237d1
:END:
#+DATE: 2026-04-10
#+filetags: :esp:azdo:ess:microlise:yaml:cicd:
#+STARTUP: showall
#+title: ESP — AzDO Pipeline

* What is Azure DevOps (AzDO)?
:PROPERTIES:
:RELATED: [[id:a6c345df-8db9-4538-b87e-0e72e2414905][ESP Deployment Pipeline — Index]] 
:END:

Azure DevOps is Microsoft's platform for DevOps workflows. It covers:
- *Repos* — Git source code repositories
- *Pipelines* — automated build and deployment (CI/CD)
- *Boards* — work items, epics, features, tasks
- *Artifacts* — storing build outputs (packages, DLLs, etc.)

For this project, the relevant parts are *Pipelines* and *Repos*.

* What is a YAML Pipeline?

AzDO pipelines can be defined in two ways: through a GUI (classic) or via a YAML file stored in the repo. We use *YAML pipelines* — this means the pipeline definition lives in source control alongside the code, making it versioned and auditable.

** Basic YAML Pipeline Structure
#+BEGIN_SRC yaml
trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'windows-latest'   # or a self-hosted agent

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - task: PowerShell@2
            inputs:
              filePath: 'scripts/Invoke-Deployment.ps1'
              arguments: '-Customer ROMAC -Environment DEV -Stages PREDEPLOY,DEPLOY'
#+END_SRC

** Key YAML Concepts
| Term       | Meaning                                                             |
|------------+---------------------------------------------------------------------|
| ~trigger~  | What causes the pipeline to run (e.g. a push to main)               |
| ~pool~     | The agent (machine) that runs the pipeline                          |
| ~stage~    | A high-level grouping of jobs (e.g. Build, Deploy)                  |
| ~job~      | A unit of work that runs on one agent                               |
| ~step~     | An individual action within a job (run a script, call a task, etc.) |
| ~task~     | A pre-built step from the AzDO marketplace (e.g. PowerShell@2)      |
| ~artifact~ | Output from one stage/job that can be consumed by another           |

* The ESP Pipelines

There are two existing pipelines to be aware of:

** ESS.esp Main Build
- *Purpose:* Builds the ESP applications from source code
- *Output:* Build artifacts (DLLs, binaries)
- *Known Issue:* The artifact does not separate applications cleanly - it produces a large flat folder of DLLs. See [[id:53d803aa-7e6f-44ed-8d99-a243a5ed5aab][ESP — Known Issues & Risks]] 
- *Location:* AzDO → Pipelines → Runs for ~ESS.esp Main Build~

** ESS.esp Deploy
- *Purpose:* Deploys ESP to a customer environment
- *Status:* Started by Team Ludo, incomplete
- *Location:* AzDO → Pipelines → Runs for ~ESS.esp Deploy~
- This is the primary pipeline your team is responsible for completing

* How the Pipeline Calls Our Scripts

The pipeline's job is relatively thin - it is an *orchestrator* that calls our PowerShell entry point with the correct parameters.

#+BEGIN_SRC
Pipeline (YAML)
   └── Calls: Invoke-Deployment.ps1 -Customer ROMAC -Environment DEV -Stages PREDEPLOY,DEPLOY,POSTDEPLOY
           └── Tier 1 script (loads manifest, loads modules)
                   └── Calls Tier 2 scripts (PreDeployment.ps1, Deployment.ps1, etc.)
                           └── Calls Tier 3 functions (Stop-WindowsService, Invoke-Sql, etc.)
#+END_SRC

See [[id:bf63b6c5-f32f-462e-82ad-8d4a15f7ba48][ESP — PowerShell Script Architecture]] for full detail on the script architecture.

* Pipeline Agents

An *agent* is the machine that actually runs the pipeline. There are two types:

| Type             | Description                                                 |
|------------------+-------------------------------------------------------------|
| Microsoft-hosted | Azure spins up a fresh VM for each run; ephemeral           |
| Self-hosted      | A persistent machine you manage; has access to your network |

For deploying to customer VMs (ROMAC DEV), we will almost certainly need a *self-hosted agent* — a Microsoft-hosted agent running in Azure would not have network access to the customer's internal servers.

* Pipeline Variables and Secrets

Pipelines can use variables for configuration. Sensitive values (passwords, connection strings) should be stored as *secret variables* or in *Azure Key Vault*, never hardcoded in the YAML.

#+BEGIN_SRC yaml
variables:
  - name: CustomerName
    value: ROMAC
  - name: DbPassword
    value: $(DB_PASSWORD)   # references a secret variable set in AzDO UI
#+END_SRC

* Approvals and Gates

For higher environments (PROD), AzDO supports *manual approval gates* between stages — a human must approve before the pipeline continues. This is important when the scope expands beyond DEV.

* Pipeline Run History

Each pipeline run is logged in AzDO with full step-by-step output. This is your primary debugging tool when a deployment fails.

* Relationship to Our PowerShell Scripts

A key design goal is that the PowerShell scripts should be able to run *independently of AzDO* — i.e. you could call ~Invoke-Deployment.ps1~ directly from a terminal if needed. AzDO is just a convenient trigger and logging wrapper. This means the logic must not be baked into the YAML itself.

* One other note on agents:
Almost certainly **Virtual Machines (VMs)**. Physical ("bare metal") servers are rarely used in modern infrastructure for this kind of thing. A data centre typically runs a small number of powerful physical machines, and on top of those you run many VMs — each VM behaves like its own independent server but they're all sharing the underlying physical hardware.

So the full picture is probably:

#+begin_src 
Physical machine(s) in ESS Altrincham DC
└── VM: AzDO Self-Hosted Agent
└── VM: App Server (runs the Windows Services, Citrix apps)
└── VM: SQL Server (runs EspBroker database)
... possibly more VMs

#+end_src

And the flow when you kick off a deployment in AzDO:

#+begin_src 

You click "Run Pipeline" in AzDO (cloud)
        │
        ▼
AzDO talks to the self-hosted agent VM in Altrincham
        │
        ▼
The agent picks up the job and runs the PowerShell scripts
        │
        ▼
The scripts connect (via PowerShell remoting) to the App Server VM
The scripts connect to the SQL Server VM
        │
        ▼
Deployment happens
#+end_src

The agent itself isn't the thing being deployed *to* — it's just the middleman that has network access to the other VMs in the same DC. Which also ties back to why each customer needs unique credentials — even though they share that infrastructure in DEV, each customer's VMs are in their own domain, so the agent needs the right credentials to authenticate into each one.