adding content
This commit is contained in:
157
Career Concepts/20260410121927-esp_pipeline.org
Normal file
157
Career Concepts/20260410121927-esp_pipeline.org
Normal file
@@ -0,0 +1,157 @@
|
||||
:PROPERTIES:
|
||||
:ID: d66e946f-6785-4e8e-ba00-bb25a52237d1
|
||||
:END:
|
||||
#+DATE: 2026-04-10
|
||||
#+filetags: :esp:azdo:ess:microlise:yaml:cicd:
|
||||
#+STARTUP: showall
|
||||
#+title: ESP — AzDO Pipeline
|
||||
|
||||
* What is Azure DevOps (AzDO)?
|
||||
:PROPERTIES:
|
||||
:RELATED: [[id:a6c345df-8db9-4538-b87e-0e72e2414905][ESP Deployment Pipeline — Index]]
|
||||
:END:
|
||||
|
||||
Azure DevOps is Microsoft's platform for DevOps workflows. It covers:
|
||||
- *Repos* — Git source code repositories
|
||||
- *Pipelines* — automated build and deployment (CI/CD)
|
||||
- *Boards* — work items, epics, features, tasks
|
||||
- *Artifacts* — storing build outputs (packages, DLLs, etc.)
|
||||
|
||||
For this project, the relevant parts are *Pipelines* and *Repos*.
|
||||
|
||||
* What is a YAML Pipeline?
|
||||
|
||||
AzDO pipelines can be defined in two ways: through a GUI (classic) or via a YAML file stored in the repo. We use *YAML pipelines* — this means the pipeline definition lives in source control alongside the code, making it versioned and auditable.
|
||||
|
||||
** Basic YAML Pipeline Structure
|
||||
#+BEGIN_SRC yaml
|
||||
trigger:
|
||||
branches:
|
||||
include:
|
||||
- main
|
||||
|
||||
pool:
|
||||
vmImage: 'windows-latest' # or a self-hosted agent
|
||||
|
||||
stages:
|
||||
- stage: Build
|
||||
jobs:
|
||||
- job: BuildJob
|
||||
steps:
|
||||
- task: PowerShell@2
|
||||
inputs:
|
||||
filePath: 'scripts/Invoke-Deployment.ps1'
|
||||
arguments: '-Customer ROMAC -Environment DEV -Stages PREDEPLOY,DEPLOY'
|
||||
#+END_SRC
|
||||
|
||||
** Key YAML Concepts
|
||||
| Term | Meaning |
|
||||
|------------+---------------------------------------------------------------------|
|
||||
| ~trigger~ | What causes the pipeline to run (e.g. a push to main) |
|
||||
| ~pool~ | The agent (machine) that runs the pipeline |
|
||||
| ~stage~ | A high-level grouping of jobs (e.g. Build, Deploy) |
|
||||
| ~job~ | A unit of work that runs on one agent |
|
||||
| ~step~ | An individual action within a job (run a script, call a task, etc.) |
|
||||
| ~task~ | A pre-built step from the AzDO marketplace (e.g. PowerShell@2) |
|
||||
| ~artifact~ | Output from one stage/job that can be consumed by another |
|
||||
|
||||
* The ESP Pipelines
|
||||
|
||||
There are two existing pipelines to be aware of:
|
||||
|
||||
** ESS.esp Main Build
|
||||
- *Purpose:* Builds the ESP applications from source code
|
||||
- *Output:* Build artifacts (DLLs, binaries)
|
||||
- *Known Issue:* The artifact does not separate applications cleanly - it produces a large flat folder of DLLs. See [[id:53d803aa-7e6f-44ed-8d99-a243a5ed5aab][ESP — Known Issues & Risks]]
|
||||
- *Location:* AzDO → Pipelines → Runs for ~ESS.esp Main Build~
|
||||
|
||||
** ESS.esp Deploy
|
||||
- *Purpose:* Deploys ESP to a customer environment
|
||||
- *Status:* Started by Team Ludo, incomplete
|
||||
- *Location:* AzDO → Pipelines → Runs for ~ESS.esp Deploy~
|
||||
- This is the primary pipeline your team is responsible for completing
|
||||
|
||||
* How the Pipeline Calls Our Scripts
|
||||
|
||||
The pipeline's job is relatively thin - it is an *orchestrator* that calls our PowerShell entry point with the correct parameters.
|
||||
|
||||
#+BEGIN_SRC
|
||||
Pipeline (YAML)
|
||||
└── Calls: Invoke-Deployment.ps1 -Customer ROMAC -Environment DEV -Stages PREDEPLOY,DEPLOY,POSTDEPLOY
|
||||
└── Tier 1 script (loads manifest, loads modules)
|
||||
└── Calls Tier 2 scripts (PreDeployment.ps1, Deployment.ps1, etc.)
|
||||
└── Calls Tier 3 functions (Stop-WindowsService, Invoke-Sql, etc.)
|
||||
#+END_SRC
|
||||
|
||||
See [[id:bf63b6c5-f32f-462e-82ad-8d4a15f7ba48][ESP — PowerShell Script Architecture]] for full detail on the script architecture.
|
||||
|
||||
* Pipeline Agents
|
||||
|
||||
An *agent* is the machine that actually runs the pipeline. There are two types:
|
||||
|
||||
| Type | Description |
|
||||
|------------------+-------------------------------------------------------------|
|
||||
| Microsoft-hosted | Azure spins up a fresh VM for each run; ephemeral |
|
||||
| Self-hosted | A persistent machine you manage; has access to your network |
|
||||
|
||||
For deploying to customer VMs (ROMAC DEV), we will almost certainly need a *self-hosted agent* — a Microsoft-hosted agent running in Azure would not have network access to the customer's internal servers.
|
||||
|
||||
* Pipeline Variables and Secrets
|
||||
|
||||
Pipelines can use variables for configuration. Sensitive values (passwords, connection strings) should be stored as *secret variables* or in *Azure Key Vault*, never hardcoded in the YAML.
|
||||
|
||||
#+BEGIN_SRC yaml
|
||||
variables:
|
||||
- name: CustomerName
|
||||
value: ROMAC
|
||||
- name: DbPassword
|
||||
value: $(DB_PASSWORD) # references a secret variable set in AzDO UI
|
||||
#+END_SRC
|
||||
|
||||
* Approvals and Gates
|
||||
|
||||
For higher environments (PROD), AzDO supports *manual approval gates* between stages — a human must approve before the pipeline continues. This is important when the scope expands beyond DEV.
|
||||
|
||||
* Pipeline Run History
|
||||
|
||||
Each pipeline run is logged in AzDO with full step-by-step output. This is your primary debugging tool when a deployment fails.
|
||||
|
||||
* Relationship to Our PowerShell Scripts
|
||||
|
||||
A key design goal is that the PowerShell scripts should be able to run *independently of AzDO* — i.e. you could call ~Invoke-Deployment.ps1~ directly from a terminal if needed. AzDO is just a convenient trigger and logging wrapper. This means the logic must not be baked into the YAML itself.
|
||||
|
||||
* One other note on agents:
|
||||
Almost certainly **Virtual Machines (VMs)**. Physical ("bare metal") servers are rarely used in modern infrastructure for this kind of thing. A data centre typically runs a small number of powerful physical machines, and on top of those you run many VMs — each VM behaves like its own independent server but they're all sharing the underlying physical hardware.
|
||||
|
||||
So the full picture is probably:
|
||||
|
||||
#+begin_src
|
||||
Physical machine(s) in ESS Altrincham DC
|
||||
└── VM: AzDO Self-Hosted Agent
|
||||
└── VM: App Server (runs the Windows Services, Citrix apps)
|
||||
└── VM: SQL Server (runs EspBroker database)
|
||||
... possibly more VMs
|
||||
|
||||
#+end_src
|
||||
|
||||
And the flow when you kick off a deployment in AzDO:
|
||||
|
||||
#+begin_src
|
||||
|
||||
You click "Run Pipeline" in AzDO (cloud)
|
||||
│
|
||||
▼
|
||||
AzDO talks to the self-hosted agent VM in Altrincham
|
||||
│
|
||||
▼
|
||||
The agent picks up the job and runs the PowerShell scripts
|
||||
│
|
||||
▼
|
||||
The scripts connect (via PowerShell remoting) to the App Server VM
|
||||
The scripts connect to the SQL Server VM
|
||||
│
|
||||
▼
|
||||
Deployment happens
|
||||
#+end_src
|
||||
|
||||
The agent itself isn't the thing being deployed *to* — it's just the middleman that has network access to the other VMs in the same DC. Which also ties back to why each customer needs unique credentials — even though they share that infrastructure in DEV, each customer's VMs are in their own domain, so the agent needs the right credentials to authenticate into each one.
|
||||
Reference in New Issue
Block a user