Huge updates
This commit is contained in:
@@ -5,10 +5,13 @@ See the categories: @@html:<a href="../../home/categories.html">Categories</a>@@
|
||||
|
||||
See the following page for more details: @@html:<a href="./career-intro.html">Career Intro</a>@@
|
||||
|
||||
** February 2026
|
||||
- [[file:restful-api.org][Restful API]] @@html:<span class="post-date">15-02-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
|
||||
** January 2026
|
||||
- [[file:database-permissions.org][Database permissions, roles and accounts]] @@html:<span class="post-date">18-01-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
- [[file:database-permissions.org][Database Permissions, Roles, and Accounts]] @@html:<span class="post-date">18-01-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
- [[file:monitoring-and-logging.org][Monitoring and Logging]] @@html:<span class="post-date">18-01-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
- [[file:pipelines.org][Pipelines and how they work]] @@html:<span class="post-date">18-01-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
- [[file:pipelines.org][Pipelines and how they work (as well as CI/CD)]] @@html:<span class="post-date">18-01-2026 23:00</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
|
||||
** December 2025
|
||||
- [[file:probation-objectives.org][Probation Objectives:]] @@html:<span class="post-date">08-12-2025 17:55</span>@@ @@html:<a href="/tags/review.html"> <span class="post-tag">review</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
|
||||
|
||||
@@ -1,9 +1,245 @@
|
||||
#+TITLE: Database permissions, roles and accounts
|
||||
#+TITLE: Database Permissions, Roles, and Accounts
|
||||
#+OPTIONS: num:nil
|
||||
#+DATE: <2026-01-18 Sun 23:00>
|
||||
#+filetags: :learning:notes:
|
||||
#+WIP: t
|
||||
#+WIP:
|
||||
#+COMMENTS: t
|
||||
#+SLUG: database-permissions
|
||||
|
||||
* TODO
|
||||
* Introduction
|
||||
Database security is a crucial part of database administration. It ensures that only authorised users can access, modify, or manage data. Three core concepts used to control access are:
|
||||
|
||||
1. Accounts (logins/users)
|
||||
2. Roles
|
||||
3. Permissions
|
||||
|
||||
These concepts work together to create structured and secure access control inside a database system.
|
||||
|
||||
* 1. Database Accounts
|
||||
|
||||
A *database account* represents an identity that can connect to the database system. In Microsoft SQL Server this is typically split into two layers:
|
||||
|
||||
- *Login* → Authentication at the server level
|
||||
- *User* → Authorisation inside a specific database
|
||||
|
||||
** Login (Server Level)
|
||||
|
||||
A login allows someone or something to authenticate with the SQL Server instance.
|
||||
|
||||
Example: Creating a login
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
CREATE LOGIN student_user
|
||||
WITH PASSWORD = 'StrongPassword123!';
|
||||
#+END_SRC
|
||||
|
||||
You can also create a login linked to Windows authentication.
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
CREATE LOGIN [DOMAIN\Zaine] FROM WINDOWS;
|
||||
#+END_SRC
|
||||
|
||||
** Database User
|
||||
|
||||
A login must be mapped to a user inside a database before it can access that database.
|
||||
|
||||
Example:
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
USE SchoolDB;
|
||||
|
||||
CREATE USER student_user
|
||||
FOR LOGIN student_user;
|
||||
#+END_SRC
|
||||
|
||||
Now the login can access the *SchoolDB* database as the user *student_user*.
|
||||
|
||||
* 2. Permissions
|
||||
|
||||
Permissions define *what actions a user can perform*. These actions include reading data, inserting rows, modifying tables, or executing procedures.
|
||||
|
||||
Common SQL Server permissions include:
|
||||
|
||||
- SELECT → Read data
|
||||
- INSERT → Add new data
|
||||
- UPDATE → Modify data
|
||||
- DELETE → Remove data
|
||||
- EXECUTE → Run stored procedures
|
||||
- ALTER → Modify database objects
|
||||
- CONTROL → Full control over an object
|
||||
|
||||
** Granting Permissions
|
||||
|
||||
Permissions are given using the *GRANT* statement.
|
||||
|
||||
Example: Allow a user to read data from a table.
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT SELECT
|
||||
ON Students
|
||||
TO student_user;
|
||||
#+END_SRC
|
||||
|
||||
** Grant Multiple Permissions
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT SELECT, INSERT
|
||||
ON Students
|
||||
TO student_user;
|
||||
#+END_SRC
|
||||
|
||||
This allows the user to read and add new rows.
|
||||
|
||||
** Revoking Permissions
|
||||
|
||||
If a permission should be removed:
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
REVOKE INSERT
|
||||
ON Students
|
||||
FROM student_user;
|
||||
#+END_SRC
|
||||
|
||||
** Denying Permissions
|
||||
|
||||
A *DENY* explicitly blocks an action, even if another role grants it.
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
DENY DELETE
|
||||
ON Students
|
||||
TO student_user;
|
||||
#+END_SRC
|
||||
|
||||
* 3. Roles
|
||||
|
||||
Roles are collections of permissions that can be assigned to multiple users. They simplify permission management by allowing administrators to assign permissions once and reuse them.
|
||||
|
||||
Instead of granting permissions to many individual users, you grant them to a role.
|
||||
|
||||
Example scenario:
|
||||
|
||||
- Many students should be able to view course data.
|
||||
- Instead of assigning permissions to each student individually, create a role.
|
||||
|
||||
** Creating a Role
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
CREATE ROLE student_role;
|
||||
#+END_SRC
|
||||
|
||||
** Assign Permissions to the Role
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT SELECT
|
||||
ON Courses
|
||||
TO student_role;
|
||||
#+END_SRC
|
||||
|
||||
** Add Users to the Role
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
ALTER ROLE student_role
|
||||
ADD MEMBER student_user;
|
||||
#+END_SRC
|
||||
|
||||
Now *student_user* inherits all permissions from *student_role*.
|
||||
|
||||
* 4. Built-in Database Roles
|
||||
|
||||
SQL Server includes several predefined roles that already have common permission sets.
|
||||
|
||||
Examples:
|
||||
|
||||
| Role Name | Purpose |
|
||||
|---------------+-----------------------------------|
|
||||
| db_owner | Full control over the database |
|
||||
| db_datareader | Read all tables |
|
||||
| db_datawriter | Insert/update/delete all tables |
|
||||
| db_ddladmin | Create or modify database objects |
|
||||
|
||||
Example: Add a user to the read-only role.
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
ALTER ROLE db_datareader
|
||||
ADD MEMBER student_user;
|
||||
#+END_SRC
|
||||
|
||||
This allows the user to read all tables without giving modification rights.
|
||||
|
||||
* 5. Example: Simple University Database Security
|
||||
|
||||
Assume a database called *UniversityDB* with two tables:
|
||||
|
||||
- Students
|
||||
- Courses
|
||||
|
||||
Goal:
|
||||
|
||||
- Students → Read course information
|
||||
- Teachers → Modify course data
|
||||
- Admin → Full control
|
||||
|
||||
** Step 1: Create Roles
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
CREATE ROLE student_role;
|
||||
CREATE ROLE teacher_role;
|
||||
CREATE ROLE admin_role;
|
||||
#+END_SRC
|
||||
|
||||
** Step 2: Assign Permissions
|
||||
|
||||
Student role (read-only):
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT SELECT
|
||||
ON Courses
|
||||
TO student_role;
|
||||
#+END_SRC
|
||||
|
||||
Teacher role:
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT SELECT, INSERT, UPDATE
|
||||
ON Courses
|
||||
TO teacher_role;
|
||||
#+END_SRC
|
||||
|
||||
Admin role:
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
GRANT CONTROL
|
||||
ON DATABASE::UniversityDB
|
||||
TO admin_role;
|
||||
#+END_SRC
|
||||
|
||||
** Step 3: Add Users
|
||||
|
||||
#+BEGIN_SRC sql
|
||||
ALTER ROLE student_role ADD MEMBER student_user;
|
||||
ALTER ROLE teacher_role ADD MEMBER teacher_user;
|
||||
ALTER ROLE admin_role ADD MEMBER admin_user;
|
||||
#+END_SRC
|
||||
|
||||
Now permissions are organised through roles instead of assigning them individually.
|
||||
|
||||
* 6. Why Roles Are Important
|
||||
|
||||
Roles provide several benefits:
|
||||
|
||||
- *Simpler management* → Change permissions in one place
|
||||
- *Scalability* → Works well with many users
|
||||
- *Security consistency* → Reduces risk of incorrect permissions
|
||||
- *Easier auditing* → Clear structure of access control
|
||||
|
||||
Without roles, administrators would need to manually manage permissions for every individual user.
|
||||
|
||||
* Summary
|
||||
|
||||
Database access control relies on three key components:
|
||||
|
||||
- *Accounts* identify who is accessing the system (logins and users).
|
||||
- *Permissions* define what actions can be performed.
|
||||
- *Roles* group permissions together for easier management.
|
||||
|
||||
In Microsoft SQL Server, administrators typically create logins, map them to database users, assign them to roles, and grant permissions to those roles. This layered approach ensures a secure and manageable database system.
|
||||
|
||||
@@ -2,14 +2,118 @@
|
||||
#+OPTIONS: num:nil
|
||||
#+DATE: <2026-01-18 Sun 23:00>
|
||||
#+filetags: :learning:notes:
|
||||
#+WIP: t
|
||||
#+WIP:
|
||||
#+COMMENTS: t
|
||||
#+SLUG: monitoring-and-logging
|
||||
|
||||
* TODO
|
||||
* Monitoring and Logging Tools Overview
|
||||
|
||||
Talk about Kibana, Grafana, Prometheus, and others.
|
||||
- [ ] Research popular monitoring and logging tools
|
||||
- [ ] Write about their features and use cases
|
||||
- [ ] Include examples of how to set them up
|
||||
- [ ] Discuss best practices for monitoring and logging
|
||||
Modern systems, especially cloud and distributed applications, require tools to observe performance, detect issues, and analyse logs. Tools like Kibana, Grafana, and Prometheus are widely used for these purposes.
|
||||
|
||||
* Kibana
|
||||
** Purpose
|
||||
Log visualisation and analysis.
|
||||
|
||||
** How it works
|
||||
Kibana is part of the Elastic Stack (formerly ELK Stack) and is used to visualise data stored in Elasticsearch.
|
||||
|
||||
** Key Features
|
||||
- Search and analyse large log datasets
|
||||
- Interactive dashboards
|
||||
- Log filtering and querying
|
||||
- Security and anomaly detection features
|
||||
|
||||
** Common Use Cases
|
||||
- Viewing application logs
|
||||
- Debugging errors in production systems
|
||||
- Security monitoring
|
||||
|
||||
** Example Setup
|
||||
1. Install Elasticsearch
|
||||
2. Send logs using Logstash or Filebeat
|
||||
3. Use Kibana to visualise logs
|
||||
|
||||
* Prometheus
|
||||
** Purpose
|
||||
Metrics collection and monitoring.
|
||||
|
||||
Prometheus is designed to collect numeric metrics over time from systems and applications.
|
||||
|
||||
** Key Features
|
||||
- Time-series database
|
||||
- Powerful query language (PromQL)
|
||||
- Built-in alerting
|
||||
- Pull-based metrics collection
|
||||
|
||||
** Common Use Cases
|
||||
- Monitoring servers and containers
|
||||
- Tracking CPU, memory, and request latency
|
||||
- Infrastructure monitoring in Kubernetes
|
||||
|
||||
** Example Setup
|
||||
1. Install Prometheus
|
||||
2. Configure targets to scrape metrics
|
||||
3. Expose metrics via a /metrics endpoint
|
||||
4. Query metrics using PromQL
|
||||
|
||||
* Grafana
|
||||
** Purpose
|
||||
Visualisation and dashboards.
|
||||
|
||||
Grafana is commonly used with Prometheus but can connect to many different data sources.
|
||||
|
||||
** Key Features
|
||||
- Highly customisable dashboards
|
||||
- Supports many data sources (Prometheus, Elasticsearch, databases)
|
||||
- Alerting and notifications
|
||||
- Real-time visual monitoring
|
||||
|
||||
** Common Use Cases
|
||||
- Infrastructure monitoring dashboards
|
||||
- Business metrics visualisation
|
||||
- Combining logs, metrics, and traces
|
||||
|
||||
** Example Setup
|
||||
1. Install Grafana
|
||||
2. Connect a data source (Prometheus, Elasticsearch, etc.)
|
||||
3. Build dashboards using panels and queries
|
||||
|
||||
* Other Popular Tools
|
||||
|
||||
** Logstash
|
||||
- Log processing pipeline
|
||||
- Collects, transforms, and sends logs to Elasticsearch
|
||||
|
||||
** Filebeat
|
||||
- Lightweight log shipper
|
||||
- Sends logs from servers to Elasticsearch
|
||||
|
||||
** Loki
|
||||
- Log aggregation system designed by Grafana Labs
|
||||
- Integrates well with Grafana dashboards
|
||||
|
||||
* Best Practices for Monitoring and Logging
|
||||
|
||||
** Monitor Key Metrics
|
||||
- CPU usage
|
||||
- Memory usage
|
||||
- Request latency
|
||||
- Error rates
|
||||
|
||||
** Centralise Logs
|
||||
Send logs from all services to a single platform.
|
||||
|
||||
** Use Alerts
|
||||
Configure alerts to notify you of abnormal behavior.
|
||||
|
||||
** Combine Logs and Metrics
|
||||
- Metrics tell you that something is wrong
|
||||
- Logs help you understand why it is wrong
|
||||
|
||||
** Create Meaningful Dashboards
|
||||
Focus on actionable information instead of displaying excessive data.
|
||||
|
||||
* Simple Summary
|
||||
- Prometheus collects metrics
|
||||
- Grafana visualises metrics
|
||||
- Kibana analyses logs
|
||||
|
||||
@@ -1,9 +1,666 @@
|
||||
#+TITLE: Pipelines and how they work
|
||||
#+TITLE: Pipelines and how they work (as well as CI/CD)
|
||||
#+OPTIONS: num:nil
|
||||
#+DATE: <2026-01-18 Sun 23:00>
|
||||
#+filetags: :learning:notes:
|
||||
#+WIP: t
|
||||
#+WIP:
|
||||
#+COMMENTS: t
|
||||
#+SLUG: pipelines
|
||||
#+SLUG: pipelines-learning
|
||||
|
||||
|
||||
* Summary
|
||||
CI/CD stands for Continuous Integration and Continuous Delivery/Deployment. It’s a development practice that automates building, testing, and delivering software so code changes can be released quickly and reliably.
|
||||
|
||||
** Continuous Integration (CI)
|
||||
|
||||
CI means developers frequently merge their code into a shared repository. Every time code is pushed:
|
||||
|
||||
1. The code is built automatically.
|
||||
2. Automated tests run.
|
||||
3. If something breaks, the team is alerted.
|
||||
|
||||
Goal: catch bugs early and ensure new code works with the rest of the project.
|
||||
|
||||
** Continuous Delivery / Continuous Deployment (CD)
|
||||
CD takes the tested code from CI and prepares it for release.
|
||||
|
||||
- Continuous Delivery: Code is automatically prepared for production, but a human approves the final deployment.
|
||||
|
||||
- Continuous Deployment: Code is automatically deployed to production with no manual approval.
|
||||
|
||||
Goal: make releases faster and safer.
|
||||
|
||||
** Pipelines
|
||||
A pipeline is the automated workflow that runs these steps in sequence. Think of it as a script that defines what happens after a code change.
|
||||
|
||||
Typical pipeline stages:
|
||||
|
||||
1. Source – code pushed to repository
|
||||
2. Build – compile the application
|
||||
3. Test – run automated tests
|
||||
4. Package – create deployable artifact (e.g., Docker image)
|
||||
5. Deploy – release to staging or production
|
||||
|
||||
Example simplified pipeline:
|
||||
|
||||
~Code Push → Build → Test → Package → Deploy~
|
||||
|
||||
Why CI/CD is useful
|
||||
|
||||
- Faster development cycles
|
||||
|
||||
- Fewer integration bugs
|
||||
|
||||
- Automated testing and deployment
|
||||
|
||||
- More reliable releases
|
||||
|
||||
Common CI/CD tools
|
||||
|
||||
- GitHub Actions
|
||||
|
||||
- Jenkins
|
||||
|
||||
- GitLab CI/CD
|
||||
|
||||
- CircleCI
|
||||
|
||||
In short:
|
||||
CI/CD uses pipelines to automatically build, test, and deploy software whenever code changes are made.
|
||||
|
||||
|
||||
* Learning by example
|
||||
|
||||
During the first wave in 2026, we planned to get the netcore version of site visits out to customers. There were a bunch of tasks relating to this, so I asked AI to conceptually explain them.
|
||||
|
||||
* Prompt:
|
||||
|
||||
I want to understand builds and pipelines in more depth. we have a set of tasks to get a new upgraded project to customers. this is task 1:
|
||||
|
||||
Add the .net core version to the TMC release pipeline.
|
||||
Follow this guide so that the build artifacts are included in the TMC release
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/AddingANewProjectToTMC.md&_a=preview
|
||||
|
||||
(This will add the artifact to the release, you will then need to test this locally, see the other task)
|
||||
|
||||
After that, we need to make the below changes as we did for the DriverWebAPI:
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMC_Release/pullrequest/23129?path=/ReleasePackagesConfig.csv&_a=files
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment/pullrequest/23154?_a=files
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/TMCBranchTool/pullrequest/23766?_a=files
|
||||
|
||||
this is task 2:
|
||||
Add application file and make deployable locally
|
||||
See this PR from when we previously did on Drivers: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23188?path=/manifests/customers/dev/LocalDeploy/DEV/_apps.yml
|
||||
|
||||
Note: That some of the properties in the files are different to what you have for a local deploy, if in doubt refer to main to ensure that we have no consistencies.
|
||||
|
||||
Also, we don't need to alter the connection string as this has already been done.
|
||||
|
||||
====
|
||||
|
||||
This requires the previous tasks to be completed (main build and artifact added to release)
|
||||
|
||||
Follow this commit as an example
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/aeb2dbf1bbcef1b672aee05fdc7988959afd1a36?refName=refs/heads/main
|
||||
|
||||
Create a new application file for the dotnet core version of the service, swap it out in the manifests for the local deploy along with any relevant configuration
|
||||
|
||||
There may be some differences due to this being a web api rather than a background service, and the fact that we use the appsettings.json rather than the app.config - in which case look to Arrivals and departures and vehiclev2 if that is still around
|
||||
|
||||
Set up local TMC and deploy - make sure it works as expected
|
||||
|
||||
This is now possible on the new domain joined machines with a VM - did it myself and it works well
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Documents?path=/TMC/Deployment/Local%20Deployment%20and%20running%20ATFs%20on%20non-domain%20joined%20machines.md&_a=preview
|
||||
|
||||
this is 3:
|
||||
Make deployable in QA (Toblerone)
|
||||
Check out this commit for the AVL where the new service was added to the templates and the deploy file, use what was learned from the local deploy to ensure this works in the same way
|
||||
|
||||
Check if you can just apply this to the toblerone box, the deploy_all.yml file governs what is actually deployed, this might not be possible but worth a check
|
||||
|
||||
Test that Toblerone deploys okay
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/59db355dbb80c784d7b7ca9f3bb30fc00ef1d13e?refName=refs/heads/main&path=/manifests
|
||||
|
||||
See this PR for how we did it last time for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/23320
|
||||
|
||||
This needs to be changed for the current release, potentially the previous release if it errors when that happens as well as the ad hoc. See the last three files in the above PR.
|
||||
|
||||
task 4:
|
||||
Add artifact to the ad-hoc pipelines for QA / Cert
|
||||
Need to complete the previous Toblerone QA task first!
|
||||
|
||||
We need to add a similar artifact for SiteVisits to the ad hoc QA / Cert release pipeline like so:
|
||||
|
||||
Image
|
||||
|
||||
And then give it a run and make sure it works.
|
||||
|
||||
See the release pipeline here: TMC.SiteVisitsWebApi - Pipelines
|
||||
|
||||
task 5:
|
||||
Ensure that you test each endpoint with different data, ensuring the experience is exactly the same for each endpoint (auth, request / response)
|
||||
|
||||
As we are planning to hotswap the instances over, they must work exactly the same
|
||||
|
||||
Make sure its on the same URL
|
||||
|
||||
task 6:
|
||||
Make deployable in the rest of QA
|
||||
We have this running in Toblerone QA, once we are happy with the API in Toblerone we need to roll it out to the rest of QA.
|
||||
|
||||
Aaron also suggested considering doing the internal CERT environments at this stage, then we have a more live like environment tested early on too.
|
||||
|
||||
See this PR for how we did this for the Driver Web API: https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/24329?path=/manifests/TemplateFiles/26.1/QA/deploy_tmc.yml
|
||||
|
||||
task 7:
|
||||
Ensure that the new Serilog logs are captured into elastic
|
||||
This file needs updating with the paths for the serilogs in the sitevisitswebapi
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/ELK-Beats?path=/inventories/uat_web_servers/templates/filebeat~planBstructuredLogs.yml
|
||||
|
||||
However it does not look like servicename and customername parsing is working for the existing logs
|
||||
|
||||
Effectively, add a line like this but for our SiteVisits API:
|
||||
|
||||
Image
|
||||
|
||||
task 8:
|
||||
Make deployable in UAT
|
||||
Update UAT to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way
|
||||
|
||||
Use this commit for reference
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/2fce44462b699b88e4c2597c8492a1a0dad1614d?refName=refs/heads/main
|
||||
|
||||
See this PR for what we did last time for UAT (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml
|
||||
|
||||
task 9:
|
||||
Make deployable in live
|
||||
Update live to use the new dotnet core version of the API, use what was learned from the local deploy to ensure this works in the same way
|
||||
|
||||
Use this commit for reference
|
||||
|
||||
https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/commit/ce61205a1f53649e7b1313fb78eb72225d99503d?refName=refs/heads/main&path=/manifests/customers/prod/CEVA_BAXTER/LIVE/_customer.yml
|
||||
|
||||
See this PR for what we did last time for live (incorporates other environments): https://azdo.microlise.com/MicroliseCollection/Microlise/_git/Deployment.Manifests.MFD/pullrequest/25631?path=/manifests/TemplateFiles/26.2/UAT/deploy_all_zero_predeploy.yml
|
||||
|
||||
|
||||
* Response
|
||||
|
||||
You’re essentially looking at a **full CI/CD path** for introducing a **new version of a service (.NET Core API)** into a large enterprise deployment system. The tasks look complicated because they touch **multiple layers of the delivery pipeline**, not just code.
|
||||
|
||||
I'll explain this in a **mental model first**, then walk through **what each task is really doing in the pipeline**.
|
||||
|
||||
---
|
||||
|
||||
** 1. The Big Picture: What a Build & Release Pipeline Is
|
||||
|
||||
In enterprise systems like Azure DevOps, deployment is usually split into two major stages.
|
||||
|
||||
1. Build Pipeline (CI)
|
||||
|
||||
Purpose: **Turn source code into deployable artifacts**
|
||||
|
||||
Steps typically include:
|
||||
|
||||
#+begin_src
|
||||
Code → Restore packages → Compile → Run tests → Package → Publish artifact
|
||||
#+end_src
|
||||
|
||||
Output = **artifact**
|
||||
|
||||
Example artifact:
|
||||
|
||||
#+begin_src
|
||||
SiteVisitsWebApi.zip
|
||||
#+end_src
|
||||
|
||||
This artifact contains:
|
||||
|
||||
#+begin_src
|
||||
dlls
|
||||
configs
|
||||
dependencies
|
||||
startup files
|
||||
#+end_src
|
||||
|
||||
Think of an **artifact** as a **versioned package of your application**.
|
||||
|
||||
|
||||
2. Release Pipeline (CD)
|
||||
|
||||
Purpose: **Take the artifact and deploy it to environments**
|
||||
|
||||
Example environments:
|
||||
|
||||
#+begin_src
|
||||
Local
|
||||
QA
|
||||
CERT
|
||||
UAT
|
||||
LIVE
|
||||
#+end_src
|
||||
|
||||
Each environment might have:
|
||||
- different configs
|
||||
- different servers
|
||||
- different manifests
|
||||
|
||||
|
||||
*Your System Specifically*
|
||||
|
||||
You are working with something like:
|
||||
|
||||
#+begin_src
|
||||
|
||||
|
||||
Git Repo
|
||||
↓
|
||||
Build Pipeline
|
||||
↓
|
||||
Artifacts
|
||||
↓
|
||||
TMC Release Pipeline
|
||||
↓
|
||||
Deployment Manifests
|
||||
↓
|
||||
Servers
|
||||
|
||||
#+end_src
|
||||
|
||||
|
||||
** 2. Why This Task Exists
|
||||
|
||||
You are **introducing a new .NET Core version of an API**.
|
||||
|
||||
Previously there was likely:
|
||||
|
||||
#+begin_src
|
||||
SiteVisitsWebApi (.NET Framework)
|
||||
#+end_src
|
||||
|
||||
Now:
|
||||
|
||||
#+begin_src
|
||||
SiteVisitsWebApi (.NET Core)
|
||||
#+end_src
|
||||
|
||||
|
||||
So you must:
|
||||
|
||||
1. Build it
|
||||
2. Package it
|
||||
3. Add it to release pipeline
|
||||
4. Update deployment configs
|
||||
5. Deploy to environments gradually
|
||||
|
||||
|
||||
** 3. What Each Task Actually Means
|
||||
|
||||
I'll translate the tasks into **plain engineering intent**.
|
||||
|
||||
|
||||
|
||||
*** Task 1: Add .NET Core Version to Build + Release
|
||||
|
||||
What is happening?
|
||||
|
||||
You add the **new project to the build pipeline** so it produces an artifact.
|
||||
|
||||
Before:
|
||||
|
||||
~Build → DriverWebAPI artifact~
|
||||
|
||||
After:
|
||||
|
||||
#+begin_src
|
||||
|
||||
Build → DriverWebAPI artifact
|
||||
→ SiteVisitsWebApi artifact
|
||||
|
||||
#+end_src
|
||||
|
||||
|
||||
Why the PRs exist
|
||||
|
||||
These repositories control deployment:
|
||||
|
||||
#+begin_src
|
||||
TMC_Release
|
||||
Deployment
|
||||
TMCBranchTool
|
||||
#+end_src
|
||||
|
||||
They likely manage:
|
||||
|
||||
| Repo | Purpose |
|
||||
| ------------- | ------------------------- |
|
||||
| TMC_Release | release packaging |
|
||||
| Deployment | deployment scripts |
|
||||
| TMCBranchTool | release branch automation |
|
||||
|
||||
|
||||
What ReleasePackagesConfig.csv probably does
|
||||
|
||||
Something like:
|
||||
|
||||
#+begin_src
|
||||
|
||||
ServiceName,ArtifactName,DeploymentType
|
||||
DriverWebApi,DriverWebApi.zip,WebApi
|
||||
SiteVisitsWebApi,SiteVisitsWebApi.zip,WebApi
|
||||
|
||||
#+end_src
|
||||
|
||||
So the pipeline knows:
|
||||
|
||||
#+begin_src
|
||||
Include this artifact in the release bundle
|
||||
#+end_src
|
||||
|
||||
|
||||
Result of Task 1
|
||||
|
||||
Your pipeline will now produce:
|
||||
|
||||
Artifacts:
|
||||
|
||||
~DriverWebApi~
|
||||
|
||||
~SiteVisitsWebApi~
|
||||
|
||||
|
||||
And release pipeline can access them.
|
||||
|
||||
|
||||
|
||||
*** Task 2: Make Deployable Locally
|
||||
|
||||
Now you test deployment **without affecting real environments**.
|
||||
|
||||
This uses **deployment manifests**.
|
||||
|
||||
|
||||
What a manifest is
|
||||
|
||||
A manifest is basically a **deployment recipe**.
|
||||
|
||||
Example:
|
||||
|
||||
#+begin_src yaml
|
||||
|
||||
apps:
|
||||
- name: sitevisitswebapi
|
||||
artifact: SiteVisitsWebApi
|
||||
port: 5000
|
||||
config: appsettings.json
|
||||
|
||||
#+end_src
|
||||
|
||||
It tells the deployment system:
|
||||
|
||||
deploy this artifact
|
||||
with these configs
|
||||
to this environment
|
||||
|
||||
|
||||
What you do here
|
||||
|
||||
Create a new **application definition**.
|
||||
|
||||
Example:
|
||||
|
||||
~_apps.yml~
|
||||
|
||||
Add:
|
||||
|
||||
~sitevisitswebapi~
|
||||
|
||||
Then configure:
|
||||
|
||||
#+begin_src
|
||||
artifact source
|
||||
ports
|
||||
configs
|
||||
environment variables
|
||||
#+end_src
|
||||
|
||||
|
||||
Why they mention differences
|
||||
|
||||
Your service is:
|
||||
|
||||
~Web API~
|
||||
|
||||
instead of
|
||||
|
||||
~Background Windows Service~
|
||||
|
||||
So deployment might differ:
|
||||
|
||||
| Background service | Web API |
|
||||
| ------------------ | ----------------------- |
|
||||
| runs as service | hosted in IIS / kestrel |
|
||||
| app.config | appsettings.json |
|
||||
|
||||
Result of Task 2
|
||||
|
||||
You can now run:
|
||||
|
||||
local TMC deployment
|
||||
|
||||
and your API launches locally.
|
||||
|
||||
*** Task 3: Deploy to QA (Toblerone)
|
||||
|
||||
Now you move from:
|
||||
|
||||
Local → QA
|
||||
|
||||
"Toblerone" appears to be a **specific QA environment/server**.
|
||||
|
||||
|
||||
You update manifests like:
|
||||
|
||||
deploy_all.yml
|
||||
|
||||
Which probably contains:
|
||||
|
||||
#+begin_src yaml
|
||||
|
||||
|
||||
apps:
|
||||
- driverwebapi
|
||||
- sitevisitswebapi
|
||||
#+end_src
|
||||
|
||||
So the deployment engine knows:
|
||||
|
||||
deploy this service in QA
|
||||
|
||||
|
||||
Then test deployment.
|
||||
|
||||
|
||||
*** Task 4: Add to Ad-hoc QA / Cert Pipelines
|
||||
|
||||
These are **manual release pipelines**.
|
||||
|
||||
Example use:
|
||||
|
||||
~Dev: deploy specific service to QA quickly~
|
||||
|
||||
|
||||
Instead of full release.
|
||||
|
||||
You add the artifact there so people can manually deploy it.
|
||||
|
||||
|
||||
*** Task 5: Endpoint Testing
|
||||
|
||||
Critical step.
|
||||
|
||||
You must confirm the new API behaves **exactly like the old one**.
|
||||
|
||||
Because they plan a **hotswap**.
|
||||
|
||||
Meaning:
|
||||
|
||||
Old API → replace with new API
|
||||
same URL
|
||||
same behaviour
|
||||
|
||||
If anything changes, clients break
|
||||
|
||||
So test:
|
||||
|
||||
Auth
|
||||
Request formats
|
||||
Response formats
|
||||
Error codes
|
||||
|
||||
|
||||
*** Task 6: Deploy to Remaining QA
|
||||
|
||||
Once Toblerone works:
|
||||
|
||||
Toblerone QA → All QA environments
|
||||
|
||||
Example:
|
||||
|
||||
QA1
|
||||
QA2
|
||||
QA3
|
||||
|
||||
Maybe also:
|
||||
|
||||
Internal CERT
|
||||
|
||||
|
||||
*** Task 7: Logging (Serilog → Elastic)
|
||||
|
||||
The new API logs differently.
|
||||
|
||||
Probably something like:
|
||||
|
||||
~logs/sitevisitswebapi/*.log~
|
||||
|
||||
|
||||
Filebeat must be told to collect them.
|
||||
|
||||
Example config:
|
||||
|
||||
#+begin_src
|
||||
- type: log
|
||||
paths:
|
||||
- /logs/sitevisitswebapi/*.log
|
||||
#+end_src
|
||||
|
||||
Then logs go:
|
||||
|
||||
~Server → Filebeat → Elasticsearch → Kibana~
|
||||
|
||||
|
||||
*** Task 8: Deploy to UAT
|
||||
|
||||
UAT = staging environment.
|
||||
|
||||
Production-like testing
|
||||
|
||||
Here you update the deployment manifests to use the **new API**.
|
||||
|
||||
|
||||
|
||||
*** Task 9: Deploy to Live
|
||||
|
||||
Final production rollout.
|
||||
|
||||
Same process:
|
||||
|
||||
manifest updates
|
||||
deploy pipeline
|
||||
verification
|
||||
|
||||
|
||||
** 4. The Entire Flow (Visual)
|
||||
|
||||
#+begin_src
|
||||
|
||||
|
||||
Code (SiteVisitsWebApi)
|
||||
│
|
||||
▼
|
||||
Build Pipeline
|
||||
│
|
||||
▼
|
||||
Artifact Produced
|
||||
(SiteVisitsWebApi.zip)
|
||||
│
|
||||
▼
|
||||
TMC Release Pipeline
|
||||
│
|
||||
▼
|
||||
Deployment Manifests
|
||||
│
|
||||
▼
|
||||
Local Deploy
|
||||
│
|
||||
▼
|
||||
QA (Toblerone)
|
||||
│
|
||||
▼
|
||||
All QA
|
||||
│
|
||||
▼
|
||||
UAT
|
||||
│
|
||||
▼
|
||||
LIVE
|
||||
#+end_src
|
||||
|
||||
|
||||
** 5. Why Enterprise Pipelines Feel Complicated
|
||||
|
||||
Because deployment is separated into **many independent systems**:
|
||||
|
||||
| Layer | Responsibility |
|
||||
| ------------------ | ----------------------- |
|
||||
| Build pipeline | compile code |
|
||||
| Artifact storage | store packages |
|
||||
| Release pipeline | orchestrate deployments |
|
||||
| Manifests | describe environments |
|
||||
| Deployment scripts | actually install |
|
||||
| Logging | monitor |
|
||||
|
||||
Each lives in **different repos**.
|
||||
|
||||
|
||||
** 6. The Most Important Concept to Understand
|
||||
|
||||
Everything revolves around **artifacts**.
|
||||
|
||||
#+begin_src
|
||||
Source Code
|
||||
↓
|
||||
Build
|
||||
↓
|
||||
Artifact
|
||||
↓
|
||||
Deploy artifact everywhere
|
||||
#+end_src
|
||||
|
||||
You **never deploy source code directly**.
|
||||
|
||||
|
||||
** 7. What You're Really Doing in This Ticket
|
||||
|
||||
You are teaching the deployment system:
|
||||
|
||||
"Hey, there is a new service now"
|
||||
|
||||
Then gradually making every environment aware of it.
|
||||
|
||||
* TODO
|
||||
|
||||
323
posts/career/restful-api.org
Normal file
323
posts/career/restful-api.org
Normal file
@@ -0,0 +1,323 @@
|
||||
#+TITLE: Restful API
|
||||
#+OPTIONS: num:nil
|
||||
#+DATE: <2026-02-15 Sun 23:00>
|
||||
#+filetags: :learning:notes:
|
||||
#+WIP: t
|
||||
#+COMMENTS: t
|
||||
#+SLUG: restful-api
|
||||
|
||||
[[../../assets/images/career/05-03-26-api-protocols.png]]
|
||||
Note: REST and Webhooks are styles, not protocols
|
||||
|
||||
* What is a RESTful API?
|
||||
|
||||
A *RESTful API* is a web service that follows the principles of *REST (Representational State Transfer)*. REST is an architectural style used for designing networked applications.
|
||||
|
||||
In a RESTful system:
|
||||
|
||||
- Everything is treated as a *resource*
|
||||
- Resources are identified using *URLs*
|
||||
- Standard *HTTP methods* are used to interact with resources
|
||||
- Communication is usually done using *JSON*
|
||||
|
||||
Example resource:
|
||||
|
||||
#+begin_src
|
||||
/users
|
||||
/users/1
|
||||
/users/1/orders
|
||||
#+end_src
|
||||
|
||||
These represent data stored on the server.
|
||||
|
||||
* Core HTTP Methods
|
||||
|
||||
REST APIs rely heavily on HTTP verbs.
|
||||
|
||||
| Method | Purpose | Example Endpoint |
|
||||
|--------+-----------------------------+------------------|
|
||||
| GET | Retrieve data | GET /users |
|
||||
| POST | Create a new resource | POST /users |
|
||||
| PUT | Update an existing resource | PUT /users/1 |
|
||||
| DELETE | Remove a resource | DELETE /users/1 |
|
||||
|
||||
* Example Resource: User
|
||||
|
||||
Assume we have a simple *User* resource:
|
||||
|
||||
#+begin_src json
|
||||
{
|
||||
"id": 1,
|
||||
"name": "Alice",
|
||||
"email": "alice@email.com"
|
||||
}
|
||||
#+end_src
|
||||
|
||||
The API allows clients to create, read, update, and delete users.
|
||||
|
||||
* Creating a REST API in C# (ASP.NET Core)
|
||||
|
||||
In C#, REST APIs are commonly built using *ASP.NET Core Web API*.
|
||||
|
||||
Example project creation:
|
||||
|
||||
#+begin_src bash
|
||||
dotnet new webapi -n UserApi
|
||||
cd UserApi
|
||||
dotnet run
|
||||
#+end_src
|
||||
|
||||
This creates a ready-to-run REST API project.
|
||||
|
||||
* Defining a Model
|
||||
|
||||
First, define the resource model.
|
||||
|
||||
File: Models/User.cs
|
||||
|
||||
#+begin_src csharp
|
||||
namespace UserApi.Models
|
||||
{
|
||||
public class User
|
||||
{
|
||||
public int Id { get; set; }
|
||||
|
||||
public string Name { get; set; }
|
||||
|
||||
public string Email { get; set; }
|
||||
}
|
||||
}
|
||||
#+end_src
|
||||
|
||||
This represents the data stored and returned by the API.
|
||||
|
||||
* Creating a Controller
|
||||
|
||||
Controllers handle HTTP requests.
|
||||
|
||||
File: Controllers/UserController.cs
|
||||
|
||||
#+begin_src csharp
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using UserApi.Models;
|
||||
|
||||
namespace UserApi.Controllers
|
||||
{
|
||||
[ApiController]
|
||||
[Route("api/[controller]")]
|
||||
public class UserController : ControllerBase
|
||||
{
|
||||
private static List<User> users = new List<User>()
|
||||
{
|
||||
new User { Id = 1, Name = "Alice", Email = "alice@email.com" },
|
||||
new User { Id = 2, Name = "Bob", Email = "bob@email.com" }
|
||||
};
|
||||
|
||||
[HttpGet]
|
||||
public ActionResult<List<User>> GetUsers()
|
||||
{
|
||||
return Ok(users);
|
||||
}
|
||||
}
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Endpoint created:
|
||||
|
||||
#+begin_src
|
||||
GET /api/user
|
||||
#+end_src
|
||||
|
||||
Response:
|
||||
|
||||
#+begin_src json
|
||||
[
|
||||
{ "id": 1, "name": "Alice", "email": "alice@email.com" },
|
||||
{ "id": 2, "name": "Bob", "email": "bob@email.com" }
|
||||
]
|
||||
#+end_src
|
||||
|
||||
* Getting a Single Resource
|
||||
|
||||
Add an endpoint to retrieve a specific user.
|
||||
|
||||
#+begin_src csharp
|
||||
[HttpGet("{id}")]
|
||||
public ActionResult<User> GetUser(int id)
|
||||
{
|
||||
var user = users.FirstOrDefault(u => u.Id == id);
|
||||
|
||||
if (user == null)
|
||||
{
|
||||
return NotFound();
|
||||
}
|
||||
|
||||
return Ok(user);
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Endpoint:
|
||||
|
||||
#+begin_src
|
||||
GET /api/user/1
|
||||
#+end_src
|
||||
|
||||
* Creating a Resource (POST)
|
||||
|
||||
Clients send JSON data to create a new user.
|
||||
|
||||
#+begin_src csharp
|
||||
[HttpPost]
|
||||
public ActionResult<User> CreateUser(User newUser)
|
||||
{
|
||||
newUser.Id = users.Max(u => u.Id) + 1;
|
||||
users.Add(newUser);
|
||||
|
||||
return CreatedAtAction(nameof(GetUser), new { id = newUser.Id }, newUser);
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Example request:
|
||||
|
||||
#+begin_src json
|
||||
POST /api/user
|
||||
|
||||
{
|
||||
"name": "Charlie",
|
||||
"email": "charlie@email.com"
|
||||
}
|
||||
#+end_src
|
||||
|
||||
* Updating a Resource (PUT)
|
||||
|
||||
#+begin_src csharp
|
||||
[HttpPut("{id}")]
|
||||
public IActionResult UpdateUser(int id, User updatedUser)
|
||||
{
|
||||
var user = users.FirstOrDefault(u => u.Id == id);
|
||||
|
||||
if (user == null)
|
||||
{
|
||||
return NotFound();
|
||||
}
|
||||
|
||||
user.Name = updatedUser.Name;
|
||||
user.Email = updatedUser.Email;
|
||||
|
||||
return NoContent();
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Endpoint:
|
||||
|
||||
#+begin_src
|
||||
PUT /api/user/1
|
||||
#+end_src
|
||||
|
||||
* Deleting a Resource
|
||||
|
||||
#+begin_src csharp
|
||||
[HttpDelete("{id}")]
|
||||
public IActionResult DeleteUser(int id)
|
||||
{
|
||||
var user = users.FirstOrDefault(u => u.Id == id);
|
||||
|
||||
if (user == null)
|
||||
{
|
||||
return NotFound();
|
||||
}
|
||||
|
||||
users.Remove(user);
|
||||
|
||||
return NoContent();
|
||||
}
|
||||
#+end_src
|
||||
|
||||
Endpoint:
|
||||
|
||||
#+begin_src
|
||||
DELETE /api/user/1
|
||||
#+end_src
|
||||
|
||||
* REST Principles
|
||||
|
||||
A good REST API should follow these key ideas:
|
||||
|
||||
** 1. Statelessness
|
||||
|
||||
Each request contains all information needed.
|
||||
|
||||
The server does *not store client session state*.
|
||||
|
||||
** 2. Resource-Based URLs
|
||||
|
||||
Endpoints should represent *nouns*, not verbs.
|
||||
|
||||
Good:
|
||||
|
||||
#+begin_src
|
||||
GET /users
|
||||
POST /users
|
||||
GET /users/1
|
||||
#+end_src
|
||||
|
||||
Bad:
|
||||
|
||||
#+begin_src
|
||||
GET /getUsers
|
||||
POST /createUser
|
||||
#+end_src
|
||||
|
||||
** 3. Standard HTTP Status Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------+--------------|
|
||||
| 200 | OK |
|
||||
| 201 | Created |
|
||||
| 204 | No Content |
|
||||
| 400 | Bad Request |
|
||||
| 404 | Not Found |
|
||||
| 500 | Server Error |
|
||||
|
||||
* Example Full API Structure
|
||||
|
||||
#+begin_src
|
||||
UserApi/
|
||||
├── Controllers/
|
||||
│ └── UserController.cs
|
||||
├── Models/
|
||||
│ └── User.cs
|
||||
├── Program.cs
|
||||
└── appsettings.json
|
||||
#+end_src
|
||||
|
||||
* Testing the API
|
||||
|
||||
You can test APIs using tools like:
|
||||
|
||||
- curl
|
||||
- Postman
|
||||
- Swagger UI (included with ASP.NET)
|
||||
|
||||
Example curl request:
|
||||
|
||||
#+begin_src bash
|
||||
curl http://localhost:5000/api/user
|
||||
#+end_src
|
||||
|
||||
* Summary
|
||||
|
||||
A RESTful API:
|
||||
|
||||
- Exposes *resources via URLs*
|
||||
- Uses *HTTP methods (GET, POST, PUT, DELETE)*
|
||||
- Communicates typically using *JSON*
|
||||
- Is *stateless*
|
||||
- Returns *standard HTTP status codes*
|
||||
|
||||
In C#, *ASP.NET Core Web API* makes building REST APIs straightforward using:
|
||||
|
||||
- Models
|
||||
- Controllers
|
||||
- Routing
|
||||
- Built-in JSON serialization
|
||||
Reference in New Issue
Block a user