added makefile and cleaned backups

This commit is contained in:
2025-12-30 18:33:47 +00:00
parent 04f2f7b845
commit 78cfe20d64
56 changed files with 103 additions and 11 deletions

View File

@@ -1,63 +0,0 @@
#+TITLE: Datamarts, Airflow and DAG's
#+OPTIONS: num:nil
#+DATE:<2025-11-15 Sat 18:37>
#+filetags: :learning:notes:
* What they are:
Summary from Filippo:
- A DataMart it's simply a database that is made ad-hoc to contain a certain subset of data (Specific tables with specific columns).
- Once that is created Airflow is used to put data in that database. So it extract data from other sources (APIS, databases) and loads it into the datamart we have created.
- Once you get Airflow locally working and a new Datamart (deployed locally) you can then write a dag to load data into the Mart.
A data warehouse is where *all* the data is stored (sales, marketing, HR etc.). A data mart is just what one group needs, so for example, a finance data mart would contain invoices, payments and budgets data.
** Apache Airflow
It is a tool that lets you automate and schedule data tasks. It:
- Runs jobs on a schedule (e.g. every night at 1am).
- Knows what needs to run before what.
- Shows you a nice UI so you can see: what succeeded, what failed and where its stuck.
So: Airflow = a smart scheduler and organiser for data workflows.
** DAG
DAG stands for directed acyclic graph. In simple terms, it is just a workflow made of tasks in a certain order. For example, a DAG can be: ~get_data~, then ~clean_data~, then ~load_data_into_datamart~, and finally ~send_email~. DAG is the whole workflow, the tasks are the individual steps inside the DAG.
*** ELT (Extract → Load → Transform)
Do the cleaning after loading into the warehouse. Flow:
- Extract from source systems
- Load raw data directly into the warehouse
- Transform inside the warehouse (usually with SQL)
Here is a diagram that explains this process:
[[../../assets/sketchnotes/elt-yed.svg]]
** Diagram
*Sam has written the following and made this diagram*
[[../../../assets/images/diagrams/airflow.png]]
1. AZDO repo for our datamart to go in. Clone the data mart example template
2. Datamart, can use one big one that has different tables for everything that we need.
3. Local airflow that calls api and gets data. continuation tokens can be used. there is a good example
4. Upload everything to microlise servers where it is run automatically
5. Power bi, change data sources to be references to our sql. Can have all tables in power bi. Use filters
** Summary that I wrote in slack:
- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.SQLServer: very useful link explaining what datamarts are, ELT (extract, load, transform) orchestration and how to set this up
- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.DataEngineering.InsightReadiness: an example of what a datamart repo looks like. in the template repo, the two important files are tsqlt and DataMartTemplate (in this instance its called InsightReadiness )
- tsqlt coverage should be above 60%
- https://microliseuk.sharepoint.com/sites/msteams_bb7636: This link is the CoP (community of practice) for the airflow. You'll find useful information regarding what apache airflow is and how to set things up.
- https://microliseuk.sharepoint.com/sites/msteams_bb7636/SitePages/Getting-started-with-Airflow.aspx: This link tells you how to set airflow up locally. Once you're done, this will be published to an airflow scheduler platform. Keep in mind that when you want to publish this, document that it succeeds in the PR.
- http://airmastqa01.mms.local:8080/home: this link shows all the DAGs that are currently running in a testing environment, and when they are scheduled. Login using MSUP (edited)
- http://airmastprd01.mms.local:8080/home: This link is the prod version
- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DevDocs?path=/Airflow/HowTo/PowerBiReportRefresh.md&version=GBmain&_a=preview: Might prove to be useful when we do the powerbi refresh thing...

View File

@@ -1,7 +0,0 @@
#+TITLE: Career Introduction
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-11-06 Thu 21:29>
#+FILETAGS: :introduction:
#+WIP:
Here lies posts that relate to the technicalities and intricacies of my career. They will include information regarding my competencies, career growth, technical components and many other things. The goal is to have a single designated area where I can read, learn and relearn different things.

View File

@@ -1,6 +0,0 @@
#+title: Invest Principles
#+date: <2025-11-06 Thu 21:08>
#+OPTIONS: num:nil tags:t toc:t
#+filetags: :learning:notes:
* TODO

View File

@@ -1,10 +0,0 @@
#+title: Lean
#+date: <2025-11-05 wed 20:46>
#+OPTIONS: num:nil tags:t toc:t
#+filetags: :learning:notes:
#+WIP: t
* What Lean aims to do
Deliver more value with less waste by shortening feedback loops, improving flow, and continually learning.

View File

@@ -1,18 +0,0 @@
#+TITLE: Management of self training
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-11-06 Thu 22:01>
#+FILETAGS: :learning:notes:
#+WIP:
Management of self in the workplace refers to how individuals regulate their behavior, emotions, and productivity to perform effectively and professionally. Here's a brief overview:
- *Time Management*: Prioritising tasks, meeting deadlines, and avoiding procrastination.
- *Emotional Regulation*: Staying calm under pressure, handling stress, and maintaining a positive attitude.
- *Accountability*: Taking responsibility for actions and outcomes, owning mistakes, and learning from them.
- *Adaptability*: Being flexible with changes, open to feedback, and willing to improve.
- *Professionalism*: Maintaining ethical standards, being respectful, and demonstrating integrity.
- *Goal Setting*: Defining clear objectives and staying focused on achieving them.
It is important as it helps build trust, improve performance and creates a healthy work environment.
[[../../assets/images/stress-in-workplace.png]]

View File

@@ -1,11 +0,0 @@
#+TITLE: Benefits of Normalisation
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-11-10 Mon 18:04>
#+FILETAGS: :learning:notes:
#+WIP:
For some reason it didn't include one of the bullet points, here it is:
- It simplifies maintenance, so its easier to update, delete or insert data without anomalies.
[[../../assets/images/competencies/Aware-of-the-Benefits-of-Normalisation.png]]

View File

@@ -1,84 +0,0 @@
#+TITLE: OWASP Top Ten
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-10-19 Sun 13:21>
#+FILETAGS: :learning:notes:
* Competency notes:
[[../../assets/images/owasp/Screenshot 2025-10-16 134403.png]]
[[../../assets/images/owasp/Screenshot 2025-10-16 134438.png]]
[[../../assets/images/owasp/Screenshot 2025-10-16 134457.png]]
[[../../assets/images/owasp/Screenshot 2025-10-16 134403.png]]
[[../../assets/images/owasp/Screenshot 2025-10-16 134512.png]]
* OWASP Top 10 (2021)
** A01: Broken Access Control
- Users can act outside their permissions (e.g., IDOR, privilege escalation).
- Mitigations:
- Enforce server-side authorization on every request.
- Deny by default; allow only explicit permissions.
- Test with role and privilege matrices.
** A02: Cryptographic Failures
- Sensitive data is exposed due to weak or misused cryptography.
- Mitigations:
- Use modern TLS everywhere.
- Encrypt sensitive data in transit and at rest.
- Manage keys securely and avoid homegrown crypto.
** A03: Injection
- Untrusted input alters queries or commands (SQLi, XSS, command injection).
- Mitigations:
- Use parameterized queries and prepared statements.
- Sanitize and validate input.
- Escape output appropriately for each context.
** A04: Insecure Design
- Security flaws in system design or architecture.
- Mitigations:
- Perform threat modeling early.
- Use secure design patterns and reference architectures.
- Enforce defense-in-depth and least privilege.
** A05: Security Misconfiguration
- Insecure defaults, unnecessary features, verbose errors, misconfigured services.
- Mitigations:
- Harden configurations and use secure defaults.
- Automate configuration management.
- Disable unused features and accounts.
** A06: Vulnerable and Outdated Components
- Using outdated software or libraries with known vulnerabilities.
- Mitigations:
- Maintain an inventory (SBOM) of components.
- Monitor CVEs and patch frequently.
- Use trusted package sources.
** A07: Identification and Authentication Failures
- Broken login, session management, or weak authentication.
- Mitigations:
- Use strong authentication (MFA recommended).
- Protect session tokens; implement logout and timeouts.
- Avoid exposing sensitive session IDs in URLs.
** A08: Software and Data Integrity Failures
- Relying on untrusted software or pipelines without integrity verification.
- Mitigations:
- Sign and verify code and dependencies.
- Secure CI/CD pipelines.
- Use integrity checks for data and updates.
** A09: Security Logging and Monitoring Failures
- Inadequate logging or monitoring prevents detection of attacks.
- Mitigations:
- Log critical events (auth, access, errors).
- Centralize and monitor logs.
- Establish alerting and response procedures.
** A10: Server-Side Request Forgery (SSRF)
- The application fetches URLs from untrusted sources on behalf of users.
- Mitigations:
- Deny outbound network access by default.
- Use allowlists for external requests.
- Protect metadata and internal services.

View File

@@ -36,11 +36,11 @@
** Books
- +The Toyota Way+
- +The Clean Coder+
- +Clean Code+
- Agile Testing (in progress)
- Headfirst into Design Patterns (in progress)
- Code Complete 2
- Clean Code (in progress)
- Working Effectively with Legacy Code
- Headfirst into Design Patterns
** Public Code Review (Demonstrate development best practice)
- Training on appropriate languages (C#, PHP Laravel, JavaScript, SQL, PowerShell, NUnit, PHPUnit, Jest, tSQLt, pester)
@@ -49,7 +49,7 @@
- Daily Katas and regular Dojos
- Present software developed by the team in a Public Code Review
** Deployment Pipeline (Integrate a build and deployment pipeline)
** +Deployment Pipeline (Integrate a build and deployment pipeline)+
- Implement a TFS build and deployment pipeline with pester tests
- Implement and maintain CI/CD
- Continuously monitor environments daily
@@ -66,7 +66,7 @@ Rotations start on: <2026-01-14 Wed>
* Probation meetings:
1. <2025-11-06 Thu>
2. <2025-12-11 Thu>
2. <2025-12-11 Thu> -> Got pushed to <2025-12-15 Mon>
3.
* Document:

View File

@@ -1,77 +0,0 @@
#+TITLE: Probation Objectives:
#+OPTIONS: num:nil
#+DATE: <2025-12-08 Mon 17:55>
#+filetags: :review:notes:
#+WIP: t
* Checklists:
** Customer Service Report
- Introduction
- Service Desk Experience
- Customer Experience
- Lessons Learnt
- Conclusion
** Personal Development Plan
- Create a SWOT
- Create a Personal Development Plan with SMART objectives
- Include strengths and weaknesses of your team role
- Define short-, medium-, and long-term objectives
- Keep traction on the competency tasks and dashboard
** Process Best Practice Report
- Introduction
- Lean Overview
- Software Development Overview
- Lessons Learnt
- Conclusion
** Competencies
- +20% End of Nov+
- 20% End of Dec
- 20% End of Jan
** Books
- +The Toyota Way+
- +The Clean Coder+
- Agile Testing (in progress)
- Code Complete 2
- Clean Code (in progress)
- Working Effectively with Legacy Code
- Headfirst into Design Patterns
** Public Code Review (Demonstrate development best practice)
- Training on appropriate languages (C#, PHP Laravel, JavaScript, SQL, PowerShell, NUnit, PHPUnit, Jest, tSQLt, pester)
- Read books and do training on development best practices (SOLID, Design Patterns, Continuous Monitoring, TDD, BDD, CI/CD)
- Training around Requirements (Feature writing, User Story writing, Task writing, User Story Mapping, Low Fidelity UI Designs)
- Daily Katas and regular Dojos
- Present software developed by the team in a Public Code Review
** Deployment Pipeline (Integrate a build and deployment pipeline)
- Implement a TFS build and deployment pipeline with pester tests
- Implement and maintain CI/CD
- Continuously monitor environments daily
- Build and/or maintain an environment using PowerShell, Vagrant, Ansible, and/or Terraform
** Misc
- DevOps Skills for Developers with VS and TFS
- Getting Started with Ansible
- Introduction to Versioning Environments with Vagrant
- Terraform Getting Started
* Rotations
Rotations start on: <2026-01-14 Wed>
* Probation meetings:
1. <2025-11-06 Thu>
2. <2025-12-11 Thu>
3.
* Document:
[[../../assets/images/reviews/probation-objectives/probation-objectives-1.png]]
[[../../assets/images/reviews/probation-objectives/probation-objectives-2.png]]
[[../../assets/images/reviews/probation-objectives/probation-objectives-3.png]]
[[../../assets/images/reviews/probation-objectives/probation-objectives-4.png]]
[[../../assets/images/reviews/probation-objectives/probation-objectives-5.png]]

View File

@@ -1,75 +0,0 @@
#+TITLE: Requirements, features, user stories, tasks, walking skeletons
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-11-06 Thu 22:01>
#+FILETAGS: :learning:notes:
#+WIP:
** Requirements
- Definition: High-level descriptions of what the system must do or how it should behave.
- Types:
- *Functional*: Specific behaviors (e.g., “The system must allow users to reset passwords”).
- *Non-functional*: Quality attributes (e.g., performance, security, usability).
- Agile Context: Requirements are often evolving and are captured incrementally through user stories and feedback loops.
** Epic
An epic is a high-level objective or initiative thats too big to complete in a single sprint. It represents a major business goal or capability and is broken down into smaller deliverables:
- Purpose: Organise related work under a common theme.
- Scope: Broad and strategic; spans multiple sprints.
- Example: “Enable secure user access” could be an epic that includes login, signup, password reset, and multi-factor authentication.
“An epic serves to manage tasks. It's a defined body of work that is segmented into specific tasks (called stories) based on the needs/requests of customers or end-users"
** Features
- Definition: Distinct pieces of functionality that deliver value to users.
- Scope: Larger than user stories but smaller than epics.
- Example: “User profile management” could be a feature encompassing login, profile editing, and avatar upload.
** User Stories
- Definition: Short, simple descriptions of a feature told from the perspective of the user.
- Format: “As a [user], I want [goal] so that [reason].”
- Purpose: Helps teams focus on user needs and prioritize work.
- Agile Role: Central to backlog grooming, sprint planning, and iterative delivery.
** Tasks
- Definition: Technical or operational steps needed to implement a user story.
- Granularity: Smallest unit of work, often assigned to individual team members.
- Example: For a story like “User can upload avatar,” tasks might include:
- Create upload UI
- Validate image format
- Store image in database
** Walking Skeleton
- Definition: A minimal, end-to-end version of the system that includes the core architecture and basic functionality.
- Purpose: Validates the systems structure early and allows incremental development.
- Agile Benefit: Enables early integration, testing, and feedback while avoiding big-bang delivery.
** Hierarchy Overview
| Level | Description | Agile Artifact |
|----------------+----------------------------+------------------|
| Vision | Broad business goals | Epics |
| Feature | Functional chunks of value | Features |
| User Need | Specific user goals | User Stories |
| Implementation | Technical execution steps | Tasks |
| Skeleton | Minimal working system | Walking Skeleton |
Here is a summary of the agile hierarchy:
| Level | Example | Description |
|----------------+--------------------------------------+----------------------------------------|
| **Epic** | Enable secure user access | Broad goal spanning multiple features |
| **Feature** | User Login | Functional unit delivering user value |
| **User Story** | “Log in with email/password” | Specific user goal within the feature |
| **Task** | Build login form, connect to backend | Technical steps to implement the story |
* Example from work:
<2025-10-09 Thu>
The user story creation we did for a particular made up project (not the proudest but team voted on this sadly).
[[../../assets/images/user_stories.jpg]]

View File

@@ -1,54 +0,0 @@
#+title: Retrospectives
#+date: <2025-11-05 wed 20:46>
#+OPTIONS: num:nil tags:t toc:t
#+filetags: :learning:notes:
#+WIP:
* Dates:
- First retrospective meeting held at the end of sprint 1: <2025-11-04 Tue>
- Second retrospective (and subsequent biweekly ones): <2025-11-18 Tue>
** Retros from <2025-12-02 Tue>
[[../../assets/images/reviews/02-12-25-retros.jpg]]
* Introduction:
Retrospectives are part of the agile methodology of working, and is a way for the team to discuss and reflect on the way the sprint went.
** Purpose of Sprint Retrospectives
- Continuous improvement: Teams inspect their work and processes to enhance quality and effectiveness.
- Reflection and learning: Members discuss what went well, what didnt, and why.
- Actionable change: The goal is to identify specific improvements and implement them in the next sprint.
** Key Activities During a Retrospective
- Review the sprint: Examine interactions, tools, processes, and outcomes.
- Identify successes and challenges: Celebrate wins and analyse problems.
- Generate insights: Explore root causes of issues and assumptions that led to missteps.
- Plan improvements: Decide on changes to try in the next sprint, often adding them to the backlog.
** Who Participates
- Entire Scrum Team: Product Owner, Scrum Master, and Developers.
- Optional stakeholders if their input is valuable, but the focus remains on the teams internal dynamics.
** Common Retrospective Techniques
- Start, Stop, Continue: What to begin, cease, or keep doing.
- Mad, Sad, Glad: Emotional reflection on the sprint.
- 4Ls (Liked, Learned, Lacked, Longed for): A deeper dive into team sentiment and learning.
** Timing and Format
- Held at the end of each sprint, typically lasting 6090 minutes.
- Can be in-person, remote, or hybrid, using tools like sticky notes, voting widgets, and templates.
** Best Practices
- Keep it constructive and focused.
- Ensure psychological safety so team members feel comfortable sharing.
- Follow through on action items to avoid stagnation and disillusionment.
* Frameworks used
Boat model:
[[../../assets/images/retrospective_boat_model.png]]

View File

@@ -1,19 +0,0 @@
#+TITLE: SOLID Principles
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-10-18 Sat 19:14>
#+FILETAGS: :learning:notes:
The SOLID principles are five design guidelines for writing maintainable, flexible, and scalable object-oriented code. Here's a quick summary:
1. Single Responsibility Principle (SRP): A class should have only one reason to change, meaning it should have a single responsibility or purpose.
2. Open/Closed Principle (OCP): Classes should be open for extension (e.g., adding new functionality) but closed for modification (no need to change existing code).
3. Liskov Substitution Principle (LSP): Subclasses should be substitutable for their base classes without breaking the program's behavior.
4. Interface Segregation Principle (ISP): Clients should not be forced to depend on interfaces they don't use, favoring specific, small interfaces over large, general ones.
5. Dependency Inversion Principle (DIP): High-level modules should not depend on low-level modules; both should depend on abstractions (e.g., interfaces), and abstractions should not depend on details.
These principles promote cleaner, more modular, and testable code.

View File

@@ -1,51 +0,0 @@
#+TITLE: Wireframe Designs
#+OPTIONS: num:nil tags:t toc:t
#+DATE: <2025-11-06 Thu 22:01>
#+FILETAGS: :learning:notes:
#+WIP:
* What are wireframes?
Wireframes are basic visual representations of a website, app, or digital product that focus on structure and layout rather than design details like colors, fonts, or images. They typically include elements such as:
- Navigation menus
- Buttons
- Content blocks
- Headers and footers
Wireframes range from low-fidelity sketches to high-fidelity {{{sidenote(1, Low-fidelity (lo-fi) refers to a basic\, quick prototype that focuses on core concepts with minimal detail\, while high-fidelity (hi-fi) refers to a realistic\, detailed prototype that closely mimics the final product's look\, feel\, and functionality)}}} mockups, depending on the stage of the design process.
* Why wireframes are useful
Wireframes are essential tools in UX and UI design for several reasons:
- Clarify layout and functionality: They help designers and stakeholders visualise how users will interact with the product
- Facilitate collaboration: Wireframes serve as a communication tool between designers, developers, and clients, aligning everyone on structure and goals (close the feedback loop)
- Identify usability issues early: By focusing on user flow and interface logic, wireframes allow teams to spot and fix problems before investing in detailed design
- Support iterative design: Wireframes are easy to update and adapt, making them ideal for testing and refining ideas
* When to use wireframes?
Wireframes are typically created in the early stages of product development, before visual design and coding begin. Theyre especially helpful during:
- Initial concept development
- Stakeholder presentations
- User testing and feedback sessions
* Competency example:
[[../../assets/images/wireframes/Screenshot 2025-10-31 112139.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 112206.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 112427.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 115621.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 115949.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 120038.png]]
[[../../assets/images/wireframes/Screenshot 2025-10-31 120203.png]]

View File

@@ -4,7 +4,7 @@
See the categories: @@html:<a href="../categories.html">Categories</a>@@
* Posts:
- [[file:career/career-list.org][Career List]] @@html:<span class="post-date">28-12-2025 22:00</span>@@
- [[file:career/career-list.org][Career List]] @@html:<span class="post-date">30-12-2025 18:24</span>@@
- [[file:career/probation-objectives.org][Probation Objectives:]] @@html:<span class="post-date">08-12-2025 17:55</span>@@ @@html:<a href="/tags/review.html"> <span class="post-tag">review</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
- [[file:career/airflow.org][Datamarts, Airflow and DAG's]] @@html:<span class="post-date">15-11-2025 18:37</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@
- [[file:career/normalisation.org][Benefits of Normalisation]] @@html:<span class="post-date">10-11-2025 18:04</span>@@ @@html:<a href="/tags/learning.html"> <span class="post-tag">learning</span> </a>@@ @@html:<a href="/tags/notes.html"> <span class="post-tag">notes</span> </a>@@