org_web/posts/career/pipelines.org

#+TITLE: Pipelines and how they work (as well as CI/CD) 
#+OPTIONS: num:nil
#+DATE: <2026-01-18 Sun 23:00> 
#+filetags: :learning:notes:
#+WIP:
#+COMMENTS: t
#+SLUG: pipelines-learning


* Summary
CI/CD stands for Continuous Integration and Continuous Delivery/Deployment. It’s a development practice that automates building, testing, and delivering software so code changes can be released quickly and reliably.

** Continuous Integration (CI)

CI means developers frequently merge their code into a shared repository. Every time code is pushed:

1. The code is built automatically.
2. Automated tests run.
3. If something breaks, the team is alerted.

Goal: catch bugs early and ensure new code works with the rest of the project.

** Continuous Delivery / Continuous Deployment (CD)
CD takes the tested code from CI and prepares it for release.

- Continuous Delivery: Code is automatically prepared for production, but a human approves the final deployment.

- Continuous Deployment: Code is automatically deployed to production with no manual approval.

Goal: make releases faster and safer.

** Pipelines
A pipeline is the automated workflow that runs these steps in sequence. Think of it as a script that defines what happens after a code change.

Typical pipeline stages:

1. Source – code pushed to repository
2. Build – compile the application
3. Test – run automated tests
4. Package – create deployable artifact (e.g., Docker image)
5. Deploy – release to staging or production

Example simplified pipeline:

~Code Push → Build → Test → Package → Deploy~

Why CI/CD is useful

- Faster development cycles

- Fewer integration bugs

- Automated testing and deployment

- More reliable releases

Common CI/CD tools

- GitHub Actions

- Jenkins

- GitLab CI/CD

- CircleCI

In short:
CI/CD uses pipelines to automatically build, test, and deploy software whenever code changes are made.


* Learning by example

During the first wave in 2026, we planned to get the netcore version of site visits out to customers. There were a bunch of tasks relating to this, so I asked AI to conceptually explain them.

* Prompt:

I want to understand builds and pipelines in more depth. we have a set of tasks to get a new upgraded project to customers. this is task 1:

Add the .net core version to the TMC release pipeline. 
Follow this guide so that the build artifacts are included in the TMC release 

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/AddingANewProjectToTMC.md&_a=preview

(This will add the artifact to the release, you will then need to test this locally, see the other task)

After that, we need to make the below changes as we did for the DriverWebAPI:

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMC_Release/pullrequest/23129?path=/ReleasePackagesConfig.csv&_a=files
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment/pullrequest/23154?_a=files
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMCBranchTool/pullrequest/23766?_a=files

this is task 2: 
Add application file and make deployable locally
See this PR from when we previously did on Drivers: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23188?path=/manifests/customers/dev/LocalDeploy/DEV/_apps.yml

Note: That some of the properties in the files are different to what you have for a local deploy, if in doubt refer to main to ensure that we have no consistencies.

Also, we don't need to alter the connection string as this has already been done.

====

This requires the previous tasks to be completed (main build and artifact added to release)

Follow this commit as an example

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/aeb2dbf1bbcef1b672aee05fdc7988959afd1a36?refName=refs/heads/main

Create a new application file for the dotnet core version of the service, swap it out in the manifests for the local deploy along with any relevant configuration

There may be some differences due to this being a web api rather than a background service, and the fact that we use the appsettings.json rather than the app.config - in which case look to Arrivals and departures and vehiclev2 if that is still around

Set up local TMC and deploy - make sure it works as expected

This is now possible on the new domain joined machines with a VM - did it myself and it works well

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/Local%20Deployment%20and%20running%20ATFs%20on%20non-domain%20joined%20machines.md&_a=preview

this is 3:
Make deployable in QA (Toblerone)
Check out this commit for the AVL where the new service was added to the templates and the deploy file, use what was learned from the local deploy to ensure this works in the same way

Check if you can just apply this to the toblerone box, the deploy_all.yml file governs what is actually deployed, this might not be possible but worth a check

Test that Toblerone deploys okay

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/59db355dbb80c784d7b7ca9f3bb30fc00ef1d13e?refName=refs/heads/main&path=/manifests

See this PR for how we did it last time for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23320

This needs to be changed for the current release, potentially the previous release if it errors when that happens as well as the ad hoc. See the last three files in the above PR. 

task 4:
Add artifact to the ad-hoc pipelines for QA / Cert
Need to complete the previous Toblerone QA task first!

We need to add a similar artifact for SiteVisits to the ad hoc QA / Cert release pipeline like so:

Image

And then give it a run and make sure it works.

See the release pipeline here: TMC.SiteVisitsWebApi - Pipelines

task 5: 
Ensure that you test each endpoint with different data, ensuring the experience is exactly the same for each endpoint (auth, request / response)

As we are planning to hotswap the instances over, they must work exactly the same

Make sure its on the same URL

task 6: 
Make deployable in the rest of QA
We have this running in Toblerone QA, once we are happy with the API in Toblerone we need to roll it out to the rest of QA.

Aaron also suggested considering doing the internal CERT environments at this stage, then we have a more live like environment tested early on too.

See this PR for how we did this for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/24329?path=/manifests/TemplateFiles/26.1/QA/deploy_tmc.yml

task 7: 
Ensure that the new Serilog logs are captured into elastic
This file needs updating with the paths for the serilogs in the sitevisitswebapi

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/ELK-Beats?path=/inventories/uat_web_servers/templates/filebeat~planBstructuredLogs.yml

However it does not look like servicename and customername parsing is working for the existing logs 

Effectively, add a line like this but for our SiteVisits API:

Image

task 8: 
Make deployable in UAT
Update UAT to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way

Use this commit for reference

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/2fce44462b699b88e4c2597c8492a1a0dad1614d?refName=refs/heads/main

See this PR for what we did last time for UAT (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml

task 9: 
Make deployable in live
Update live to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way

Use this commit for reference

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/ce61205a1f53649e7b1313fb78eb72225d99503d?refName=refs/heads/main&path=/manifests/customers/prod/CEVA_BAXTER/LIVE/_customer.yml

See this PR for what we did last time for live (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml 


* Response

You’re essentially looking at a **full CI/CD path** for introducing a **new version of a service (.NET Core API)** into a large enterprise deployment system. The tasks look complicated because they touch **multiple layers of the delivery pipeline**, not just code.

I'll explain this in a **mental model first**, then walk through **what each task is really doing in the pipeline**.

---

** 1. The Big Picture: What a Build & Release Pipeline Is

In enterprise systems like Azure DevOps, deployment is usually split into two major stages.

1. Build Pipeline (CI)

Purpose: **Turn source code into deployable artifacts**

Steps typically include:

#+begin_src 
Code → Restore packages → Compile → Run tests → Package → Publish artifact
#+end_src

Output = **artifact**

Example artifact:

#+begin_src 
SiteVisitsWebApi.zip
#+end_src

This artifact contains:

#+begin_src 
dlls
configs
dependencies
startup files
#+end_src

Think of an **artifact** as a **versioned package of your application**.


2. Release Pipeline (CD)

Purpose: **Take the artifact and deploy it to environments**

Example environments:

#+begin_src 
Local
QA
CERT
UAT
LIVE
#+end_src

Each environment might have:
- different configs
- different servers
- different manifests


*Your System Specifically*

You are working with something like:

#+begin_src 


Git Repo
   ↓
Build Pipeline
   ↓
Artifacts
   ↓
TMC Release Pipeline
   ↓
Deployment Manifests
   ↓
Servers

#+end_src


** 2. Why This Task Exists

You are **introducing a new .NET Core version of an API**.

Previously there was likely:

#+begin_src 
SiteVisitsWebApi (.NET Framework)
#+end_src

Now:

#+begin_src 
SiteVisitsWebApi (.NET Core)
#+end_src


So you must:

1. Build it
2. Package it
3. Add it to release pipeline
4. Update deployment configs
5. Deploy to environments gradually


** 3. What Each Task Actually Means

I'll translate the tasks into **plain engineering intent**.



*** Task 1: Add .NET Core Version to Build + Release

What is happening?

You add the **new project to the build pipeline** so it produces an artifact.

Before:

~Build → DriverWebAPI artifact~

After:

#+begin_src 

Build → DriverWebAPI artifact
      → SiteVisitsWebApi artifact

#+end_src


Why the PRs exist

These repositories control deployment:

#+begin_src 
TMC_Release
Deployment
TMCBranchTool
#+end_src

They likely manage:

| Repo          | Purpose                   |
| ------------- | ------------------------- |
| TMC_Release   | release packaging         |
| Deployment    | deployment scripts        |
| TMCBranchTool | release branch automation |


What ReleasePackagesConfig.csv probably does

Something like:

#+begin_src 

ServiceName,ArtifactName,DeploymentType
DriverWebApi,DriverWebApi.zip,WebApi
SiteVisitsWebApi,SiteVisitsWebApi.zip,WebApi

#+end_src

So the pipeline knows:

#+begin_src 
Include this artifact in the release bundle
#+end_src


Result of Task 1

Your pipeline will now produce:

Artifacts:

~DriverWebApi~

~SiteVisitsWebApi~


And release pipeline can access them.



*** Task 2: Make Deployable Locally

Now you test deployment **without affecting real environments**.

This uses **deployment manifests**.


What a manifest is

A manifest is basically a **deployment recipe**.

Example:

#+begin_src yaml

apps:
  - name: sitevisitswebapi
    artifact: SiteVisitsWebApi
    port: 5000
    config: appsettings.json
  
#+end_src

It tells the deployment system:

deploy this artifact
with these configs
to this environment


What you do here

Create a new **application definition**.

Example:

~_apps.yml~

Add:

~sitevisitswebapi~

Then configure:

#+begin_src 
artifact source
ports
configs
environment variables
#+end_src


Why they mention differences

Your service is:

~Web API~

instead of

~Background Windows Service~

So deployment might differ:

| Background service | Web API                 |
| ------------------ | ----------------------- |
| runs as service    | hosted in IIS / kestrel |
| app.config         | appsettings.json        |

Result of Task 2

You can now run:

local TMC deployment

and your API launches locally.

*** Task 3: Deploy to QA (Toblerone)

Now you move from:

Local → QA

"Toblerone" appears to be a **specific QA environment/server**.


You update manifests like:

deploy_all.yml

Which probably contains:

#+begin_src yaml


apps:
  - driverwebapi
  - sitevisitswebapi
#+end_src

So the deployment engine knows:

deploy this service in QA


Then test deployment.


*** Task 4: Add to Ad-hoc QA / Cert Pipelines

These are **manual release pipelines**.

Example use:

~Dev: deploy specific service to QA quickly~


Instead of full release.

You add the artifact there so people can manually deploy it.


*** Task 5: Endpoint Testing

Critical step.

You must confirm the new API behaves **exactly like the old one**.

Because they plan a **hotswap**.

Meaning:

Old API → replace with new API
same URL
same behaviour

If anything changes, clients break

So test:

Auth
Request formats
Response formats
Error codes


*** Task 6: Deploy to Remaining QA

Once Toblerone works:

Toblerone QA → All QA environments

Example:

QA1
QA2
QA3

Maybe also:

Internal CERT


*** Task 7: Logging (Serilog → Elastic)

The new API logs differently.

Probably something like:

~logs/sitevisitswebapi/*.log~


Filebeat must be told to collect them.

Example config:

#+begin_src 
- type: log
  paths:
    - /logs/sitevisitswebapi/*.log
#+end_src

Then logs go:

~Server → Filebeat → Elasticsearch → Kibana~


*** Task 8: Deploy to UAT

UAT = staging environment.

Production-like testing

Here you update the deployment manifests to use the **new API**.



*** Task 9: Deploy to Live

Final production rollout.

Same process:

manifest updates
deploy pipeline
verification


** 4. The Entire Flow (Visual)

#+begin_src 


Code (SiteVisitsWebApi)
       │
       ▼
Build Pipeline
       │
       ▼
Artifact Produced
(SiteVisitsWebApi.zip)
       │
       ▼
TMC Release Pipeline
       │
       ▼
Deployment Manifests
       │
       ▼
Local Deploy
       │
       ▼
QA (Toblerone)
       │
       ▼
All QA
       │
       ▼
UAT
       │
       ▼
LIVE
#+end_src


** 5. Why Enterprise Pipelines Feel Complicated

Because deployment is separated into **many independent systems**:

| Layer              | Responsibility          |
| ------------------ | ----------------------- |
| Build pipeline     | compile code            |
| Artifact storage   | store packages          |
| Release pipeline   | orchestrate deployments |
| Manifests          | describe environments   |
| Deployment scripts | actually install        |
| Logging            | monitor                 |

Each lives in **different repos**.


** 6. The Most Important Concept to Understand

Everything revolves around **artifacts**.

#+begin_src 
Source Code
   ↓
Build
   ↓
Artifact
   ↓
Deploy artifact everywhere
#+end_src

You **never deploy source code directly**.


** 7. What You're Really Doing in This Ticket

You are teaching the deployment system:

"Hey, there is a new service now"

Then gradually making every environment aware of it.