org_web/blogs/2025/11-november/16-11-week-review.org

[16-11-2025] - Weekly Review

• Previous Week List:
• Weekly review:
• Other
  • Airflow/Datamart
    • Diagram
    • Summary that I wrote in slack:
• Next week:

#+DATE:<2025-11-10 Mon 17:44>

Previous Week List:

• Competencies to do:

  • Aware of the Benefits of Normalisation
  • Aware of PowerShell, Python, Bash and Pester
  • Aware of Lean, Agile & extreme programming Principles
  • Understands Server Operating Systems (might be tough)
  • Understanding of TCP/IP
  • Understands responsive design
  • Understands Network design

Weekly review:

• <2025-11-10 Mon>:

  • Got assigned to create a database that can help the whole company. Value added work! The database will have system names linked to the team names. In other words, which teams are working under which systems? Also it will have the repo names and which teams own which repos.

• <2025-11-11 Tue>

  • Worked on getting the DB for the system-team-repo. Met with Jez and he said to get the IPT names and use them instead. [reminder to attatch the links for this].
  • Created a powershell script to load all the data into the SQL server (local)

• <2025-11-12 Wed>

  • Met with the whole team monopoly (+ others). -> First time being in a group huddle
  • Was interesting to see how yworks is used

• <2025-11-13 Thu>

  • Angular training. Was taught about the new Angular 19 changes and how to use them. Things like the the @ decorator was discussed.

• <2025-11-14 Fri>

  • Training on datamarts and airflow. Learnt a lot about how they can be used. I wrote a few slack messages explaining to the team what it is, and also got clarification from Filipo. Also sent a request to create a new repository for this project.

Was able to complete 4/5 competencies this week. The reason for not completing them is that Friday was used mainly for meetings and other admin tasks. In hindsight, I could have completed them, as there were times when I chose to not do value added work.

Daily retrospectives:

One thing that really helped this week is guac. I was able to use this whilst at work, very handy when wanting to refer to some notes/files that is only located at home.

Other

Airflow/Datamart

Airflow Post

Diagram

A work colleague has written the following and made this diagram

1. AZDO repo for our datamart to go in. Clone the data mart example template
2. Datamart, can use one big one that has different tables for everything that we need.
3. Local airflow that calls api and gets data. continuation tokens can be used. there is a good example
4. Upload everything to microlise servers where it is run automatically
5. Power bi, change data sources to be references to our sql. Can have all tables in power bi. Use filters

Summary that I wrote in slack:

• https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.SQLServer: very useful link explaining what datamarts are, ELT (extract, load, transform) orchestration and how to set this up
• https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.DataEngineering.InsightReadiness: an example of what a datamart repo looks like. in the template repo, the two important files are tsqlt and DataMartTemplate (in this instance its called InsightReadiness )
• tsqlt coverage should be above 60%
• https://microliseuk.sharepoint.com/sites/msteams_bb7636: This link is the CoP (community of practice) for the airflow. You'll find useful information regarding what apache airflow is and how to set things up.
• https://microliseuk.sharepoint.com/sites/msteams_bb7636/SitePages/Getting-started-with-Airflow.aspx: This link tells you how to set airflow up locally. Once you're done, this will be published to an airflow scheduler platform. Keep in mind that when you want to publish this, document that it succeeds in the PR.

• Just a summary from Filippo:

  • A DataMart it's simply a database that is made ad-hoc to contain a certain subset of data (Specific tables with specific columns).
  • Once that is created Airflow is used to put data in that database. So it extract data from other sources (APIS, databases) and loads it into the datamart we have created.
  • Once you get Airflow locally working and a new Datamart (deployed locally) you can then write a dag to load data into the Mart.
• http://airmastqa01.mms.local:8080/home: this link shows all the DAGs that are currently running in a testing environment, and when they are scheduled. Login using MSUP (edited)
• http://airmastprd01.mms.local:8080/home: This link is the prod version
• https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DevDocs?path=/Airflow/HowTo/PowerBiReportRefresh.md&version=GBmain&_a=preview: Might prove to be useful when we do the powerbi refresh thing…

Next week:

• Understands Server Operating Systems (might be tough)
• Understanding of TCP/IP
• Aware of PowerShell, Python, Bash and Pester