:PROPERTIES:
:ID:       c6cf8f7a-778f-4e83-b607-753ef8dbb3f1
:END:
#+title: Airflow
#+filetags: :database:docs:

* xcom

XComs stands for cross-communications, and as the name suggests, it allows [[id:1ea42d73-3cd1-4cac-a6ee-028de21d08d5][Airflow Tasks]] to talk to each other (because by default, tasks are entirely isolated and can run on seperate machines).

See [[https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html][this]]

* parquets
*Key Takeaways*

- What is Parquet: An open-source columnar storage file format for efficient analytics.
- Core Benefits: Superior compression, faster query performance (via column pruning and predicate pushdown), and schema evolution support.
- Common Alternatives: Compared to row-based (CSV, Avro), columnar (ORC), and table formats (Iceberg, Delta Lake).
- DuckDB & MotherDuck: Parquet integrates seamlessly with DuckDB for high-performance SQL queries directly on Parquet files.

*What is Parquet?*

Parquet is a columnar storage file format. When data engineers ask 'what is a Parquet file?', the simple answer is that it's a file that stores data in columns, not rows. This Parquet data format is designed for efficient data processing, particularly in the context of big data applications. Developed as part of the Apache Hadoop ecosystem, Parquet has gained widespread adoption due to its ability to optimize storage and query performance.

[[https://motherduck.com/learn-more/why-choose-parquet-table-file-format/][See this]]
