fixing gitignore

posts/career/airflow.org

@@ -0,0 +1,65 @@
+#+TITLE: Datamarts, Airflow and DAG's
+#+OPTIONS: num:nil
+#+DATE:<2025-11-15 Sat 18:37> 
+#+filetags: :learning:notes:
+#+COMMENTS: t
+#+SLUG: airflow
+
+* What they are:
+Summary from Filippo:
+- A DataMart it's simply a database that is made ad-hoc to contain a certain subset of data (Specific tables with specific columns).
+- Once that is created Airflow is used to put data in that database. So it extract data from other sources (APIS, databases) and loads it into the datamart we have created.
+- Once you get Airflow locally working and a new Datamart (deployed locally) you can then write a dag to load data into the Mart.
+
+A data warehouse is where *all* the data is stored (sales, marketing, HR etc.). A data mart is just what one group needs, so for example, a finance data mart would contain invoices, payments and budgets data.
+
+** Apache Airflow
+It is a tool that lets you automate and schedule data tasks. It:
+- Runs jobs on a schedule (e.g. every night at 1am).
+- Knows what needs to run before what.
+- Shows you a nice UI so you can see: what succeeded, what failed and where it’s stuck.
+
+So: Airflow = a smart scheduler and organiser for data workflows.
+
+** DAG
+DAG stands for directed acyclic graph. In simple terms, it is just a workflow made of tasks in a certain order. For example, a DAG can be: ~get_data~, then ~clean_data~, then ~load_data_into_datamart~, and finally ~send_email~. DAG is the whole workflow, the tasks are the individual steps inside the DAG.
+
+*** ELT (Extract → Load → Transform)
+
+Do the cleaning after loading into the warehouse. Flow:
+- Extract from source systems
+- Load raw data directly into the warehouse
+- Transform inside the warehouse (usually with SQL)
+
+Here is a diagram that explains this process:
+
+[[../../assets/sketchnotes/elt-yed.svg]]
+
+** Diagram
+*Sam has written the following and made this diagram*
+
+[[../../../assets/images/diagrams/airflow.png]]
+
+1. AZDO repo for our datamart to go in. Clone the data mart example template
+2. Datamart, can use one big one that has different tables for everything that we need.
+3. Local airflow that calls api and gets data. continuation tokens can be used. there is a good example
+4. Upload everything to microlise servers where it is run automatically
+5. Power bi, change data sources to be references to our sql. Can have all tables in power bi. Use filters
+
+** Summary that I wrote in slack:
+
+- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.SQLServer: very useful link explaining what datamarts are, ELT (extract, load, transform) orchestration and how to set this up
+
+- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DataMart.DataEngineering.InsightReadiness: an example of what a datamart repo looks like. in the template repo, the two important files are tsqlt and DataMartTemplate (in this instance its called InsightReadiness )
+
+- tsqlt coverage should be above 60%
+
+- https://microliseuk.sharepoint.com/sites/msteams_bb7636: This link is the CoP (community of practice) for the airflow. You'll find useful information regarding what apache airflow is and how to set things up.
+
+- https://microliseuk.sharepoint.com/sites/msteams_bb7636/SitePages/Getting-started-with-Airflow.aspx: This link tells you how to set airflow up locally. Once you're done, this will be published to an airflow scheduler platform. Keep in mind that when you want to publish this, document that it succeeds in the PR.
+
+- http://airmastqa01.mms.local:8080/home: this link shows all the DAGs that are currently running in a testing environment, and when they are scheduled. Login using MSUP (edited) 
+
+- http://airmastprd01.mms.local:8080/home: This link is the prod version
+
+- https://azdo.microlise.com/MicroliseCollection/Microlise/_git/DevDocs?path=/Airflow/HowTo/PowerBiReportRefresh.md&version=GBmain&_a=preview: Might prove to be useful when we do the powerbi refresh thing...