org_web/posts/career/pipelines.org

Pipelines and how they work (as well as CI/CD)

• Summary
  • Continuous Integration (CI)
  • Continuous Delivery / Continuous Deployment (CD)
  • Pipelines
• Learning by example
• Prompt:
• Response
  • 1. The Big Picture: What a Build & Release Pipeline Is
  • 2. Why This Task Exists
  • 3. What Each Task Actually Means
    • Task 1: Add .NET Core Version to Build + Release
    • Task 2: Make Deployable Locally
    • Task 3: Deploy to QA (Toblerone)
    • Task 4: Add to Ad-hoc QA / Cert Pipelines
    • Task 5: Endpoint Testing
    • Task 6: Deploy to Remaining QA
    • Task 7: Logging (Serilog → Elastic)
    • Task 8: Deploy to UAT
    • Task 9: Deploy to Live
  • 4. The Entire Flow (Visual)
  • 5. Why Enterprise Pipelines Feel Complicated
  • 6. The Most Important Concept to Understand
  • 7. What You're Really Doing in This Ticket

Summary

CI/CD stands for Continuous Integration and Continuous Delivery/Deployment. It’s a development practice that automates building, testing, and delivering software so code changes can be released quickly and reliably.

Continuous Integration (CI)

CI means developers frequently merge their code into a shared repository. Every time code is pushed:

1. The code is built automatically.
2. Automated tests run.
3. If something breaks, the team is alerted.

Goal: catch bugs early and ensure new code works with the rest of the project.

Continuous Delivery / Continuous Deployment (CD)

CD takes the tested code from CI and prepares it for release.

• Continuous Delivery: Code is automatically prepared for production, but a human approves the final deployment.
• Continuous Deployment: Code is automatically deployed to production with no manual approval.

Goal: make releases faster and safer.

Pipelines

A pipeline is the automated workflow that runs these steps in sequence. Think of it as a script that defines what happens after a code change.

Typical pipeline stages:

1. Source – code pushed to repository
2. Build – compile the application
3. Test – run automated tests
4. Package – create deployable artifact (e.g., Docker image)
5. Deploy – release to staging or production

Example simplified pipeline:

Code Push → Build → Test → Package → Deploy

Why CI/CD is useful

• Faster development cycles
• Fewer integration bugs
• Automated testing and deployment
• More reliable releases

Common CI/CD tools

• GitHub Actions
• Jenkins
• GitLab CI/CD
• CircleCI

In short: CI/CD uses pipelines to automatically build, test, and deploy software whenever code changes are made.

Learning by example

During the first wave in 2026, we planned to get the netcore version of site visits out to customers. There were a bunch of tasks relating to this, so I asked AI to conceptually explain them.

Prompt:

I want to understand builds and pipelines in more depth. we have a set of tasks to get a new upgraded project to customers. this is task 1:

Add the .net core version to the TMC release pipeline. Follow this guide so that the build artifacts are included in the TMC release

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/AddingANewProjectToTMC.md&_a=preview

(This will add the artifact to the release, you will then need to test this locally, see the other task)

After that, we need to make the below changes as we did for the DriverWebAPI:

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMC_Release/pullrequest/23129?path=/ReleasePackagesConfig.csv&_a=files https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment/pullrequest/23154?_a=files https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMCBranchTool/pullrequest/23766?_a=files

this is task 2: Add application file and make deployable locally See this PR from when we previously did on Drivers: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23188?path=/manifests/customers/dev/LocalDeploy/DEV/_apps.yml

Note: That some of the properties in the files are different to what you have for a local deploy, if in doubt refer to main to ensure that we have no consistencies.

Also, we don't need to alter the connection string as this has already been done.

==

This requires the previous tasks to be completed (main build and artifact added to release)

Follow this commit as an example

aeb2dbf1bb/heads/main

Create a new application file for the dotnet core version of the service, swap it out in the manifests for the local deploy along with any relevant configuration

There may be some differences due to this being a web api rather than a background service, and the fact that we use the appsettings.json rather than the app.config - in which case look to Arrivals and departures and vehiclev2 if that is still around

Set up local TMC and deploy - make sure it works as expected

This is now possible on the new domain joined machines with a VM - did it myself and it works well

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/Local%20Deployment%20and%20running%20ATFs%20on%20non-domain%20joined%20machines.md&_a=preview

this is 3: Make deployable in QA (Toblerone) Check out this commit for the AVL where the new service was added to the templates and the deploy file, use what was learned from the local deploy to ensure this works in the same way

Check if you can just apply this to the toblerone box, the deploy_all.yml file governs what is actually deployed, this might not be possible but worth a check

Test that Toblerone deploys okay

59db355dbb/heads/main&path=/manifests

See this PR for how we did it last time for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23320

This needs to be changed for the current release, potentially the previous release if it errors when that happens as well as the ad hoc. See the last three files in the above PR.

task 4: Add artifact to the ad-hoc pipelines for QA / Cert Need to complete the previous Toblerone QA task first!

We need to add a similar artifact for SiteVisits to the ad hoc QA / Cert release pipeline like so:

Image

And then give it a run and make sure it works.

See the release pipeline here: TMC.SiteVisitsWebApi - Pipelines

task 5: Ensure that you test each endpoint with different data, ensuring the experience is exactly the same for each endpoint (auth, request / response)

As we are planning to hotswap the instances over, they must work exactly the same

Make sure its on the same URL

task 6: Make deployable in the rest of QA We have this running in Toblerone QA, once we are happy with the API in Toblerone we need to roll it out to the rest of QA.

Aaron also suggested considering doing the internal CERT environments at this stage, then we have a more live like environment tested early on too.

See this PR for how we did this for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/24329?path=/manifests/TemplateFiles/26.1/QA/deploy_tmc.yml

task 7: Ensure that the new Serilog logs are captured into elastic This file needs updating with the paths for the serilogs in the sitevisitswebapi

https://azdo.microlise.com/MicroliseCollection/Microlise/_git/ELK-Beats?path=/inventories/uat_web_servers/templates/filebeat~planBstructuredLogs.yml

However it does not look like servicename and customername parsing is working for the existing logs

Effectively, add a line like this but for our SiteVisits API:

Image

task 8: Make deployable in UAT Update UAT to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way

Use this commit for reference

2fce44462b/heads/main

See this PR for what we did last time for UAT (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml

task 9: Make deployable in live Update live to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way

Use this commit for reference

ce61205a1f/heads/main&path=/manifests/customers/prod/CEVA_BAXTER/LIVE/_customer.yml

See this PR for what we did last time for live (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml

Response

You’re essentially looking at a full CI/CD path for introducing a new version of a service (.NET Core API) into a large enterprise deployment system. The tasks look complicated because they touch multiple layers of the delivery pipeline, not just code.

I'll explain this in a mental model first, then walk through what each task is really doing in the pipeline.

—

1. The Big Picture: What a Build & Release Pipeline Is

In enterprise systems like Azure DevOps, deployment is usually split into two major stages.

1. Build Pipeline (CI)

Purpose: Turn source code into deployable artifacts

Steps typically include:

Code → Restore packages → Compile → Run tests → Package → Publish artifact

Output = artifact

Example artifact:

SiteVisitsWebApi.zip

This artifact contains:

dlls
configs
dependencies
startup files

Think of an artifact as a versioned package of your application.

1. Release Pipeline (CD)

Purpose: Take the artifact and deploy it to environments

Example environments:

Local
QA
CERT
UAT
LIVE

Each environment might have:

• different configs
• different servers
• different manifests

Your System Specifically

You are working with something like:

Git Repo
   ↓
Build Pipeline
   ↓
Artifacts
   ↓
TMC Release Pipeline
   ↓
Deployment Manifests
   ↓
Servers

2. Why This Task Exists

You are introducing a new .NET Core version of an API.

Previously there was likely:

SiteVisitsWebApi (.NET Framework)

Now:

SiteVisitsWebApi (.NET Core)

So you must:

1. Build it
2. Package it
3. Add it to release pipeline
4. Update deployment configs
5. Deploy to environments gradually

3. What Each Task Actually Means

I'll translate the tasks into plain engineering intent.

Task 1: Add .NET Core Version to Build + Release

What is happening?

You add the new project to the build pipeline so it produces an artifact.

Before:

Build → DriverWebAPI artifact

After:

Build → DriverWebAPI artifact
      → SiteVisitsWebApi artifact

Why the PRs exist

These repositories control deployment:

TMC_Release
Deployment
TMCBranchTool

They likely manage:

Repo	Purpose
————-	————————-
TMC_Release	release packaging
Deployment	deployment scripts
TMCBranchTool	release branch automation

What ReleasePackagesConfig.csv probably does

Something like:

ServiceName,ArtifactName,DeploymentType
DriverWebApi,DriverWebApi.zip,WebApi
SiteVisitsWebApi,SiteVisitsWebApi.zip,WebApi

So the pipeline knows:

Include this artifact in the release bundle

Result of Task 1

Your pipeline will now produce:

Artifacts:

DriverWebApi

SiteVisitsWebApi

And release pipeline can access them.

Task 2: Make Deployable Locally

Now you test deployment without affecting real environments.

This uses deployment manifests.

What a manifest is

A manifest is basically a deployment recipe.

Example:

apps:
  - name: sitevisitswebapi
    artifact: SiteVisitsWebApi
    port: 5000
    config: appsettings.json

It tells the deployment system:

deploy this artifact with these configs to this environment

What you do here

Create a new application definition.

Example:

_apps.yml

Add:

sitevisitswebapi

Then configure:

artifact source
ports
configs
environment variables

Why they mention differences

Your service is:

Web API

instead of

Background Windows Service

So deployment might differ:

Background service	Web API
——————	———————–
runs as service	hosted in IIS / kestrel
app.config	appsettings.json

Result of Task 2

You can now run:

local TMC deployment

and your API launches locally.

Task 3: Deploy to QA (Toblerone)

Now you move from:

Local → QA

"Toblerone" appears to be a specific QA environment/server.

You update manifests like:

deploy_all.yml

Which probably contains:

apps:
  - driverwebapi
  - sitevisitswebapi

So the deployment engine knows:

deploy this service in QA

Then test deployment.

Task 4: Add to Ad-hoc QA / Cert Pipelines

These are manual release pipelines.

Example use:

Dev: deploy specific service to QA quickly

Instead of full release.

You add the artifact there so people can manually deploy it.

Task 5: Endpoint Testing

Critical step.

You must confirm the new API behaves exactly like the old one.

Because they plan a hotswap.

Meaning:

Old API → replace with new API same URL same behaviour

If anything changes, clients break

So test:

Auth Request formats Response formats Error codes

Task 6: Deploy to Remaining QA

Once Toblerone works:

Toblerone QA → All QA environments

Example:

QA1 QA2 QA3

Maybe also:

Internal CERT

Task 7: Logging (Serilog → Elastic)

The new API logs differently.

Probably something like:

logs/sitevisitswebapi/*.log

Filebeat must be told to collect them.

Example config:

- type: log
  paths:
    - /logs/sitevisitswebapi/*.log

Then logs go:

Server → Filebeat → Elasticsearch → Kibana

Task 8: Deploy to UAT

UAT = staging environment.

Production-like testing

Here you update the deployment manifests to use the new API.

Task 9: Deploy to Live

Final production rollout.

Same process:

manifest updates deploy pipeline verification

4. The Entire Flow (Visual)

Code (SiteVisitsWebApi)
       │
       ▼
Build Pipeline
       │
       ▼
Artifact Produced
(SiteVisitsWebApi.zip)
       │
       ▼
TMC Release Pipeline
       │
       ▼
Deployment Manifests
       │
       ▼
Local Deploy
       │
       ▼
QA (Toblerone)
       │
       ▼
All QA
       │
       ▼
UAT
       │
       ▼
LIVE

5. Why Enterprise Pipelines Feel Complicated

Because deployment is separated into many independent systems:

Layer	Responsibility
——————	———————–
Build pipeline	compile code
Artifact storage	store packages
Release pipeline	orchestrate deployments
Manifests	describe environments
Deployment scripts	actually install
Logging	monitor

Each lives in different repos.

6. The Most Important Concept to Understand

Everything revolves around artifacts.

Source Code
   ↓
Build
   ↓
Artifact
   ↓
Deploy artifact everywhere

You never deploy source code directly.

7. What You're Really Doing in This Ticket

You are teaching the deployment system:

"Hey, there is a new service now"

Then gradually making every environment aware of it.