Course Outline
Overview of Apache Airflow
- The role of workflow orchestration.
- Principal features and advantages of Apache Airflow.
- Enhancements in Airflow 2.x and an overview of the ecosystem.
Architecture and Fundamental Concepts
- Scheduler, web server, and worker processes.
- DAGs, tasks, and operators.
- Executors and backend options (Local, Celery, Kubernetes).
Deployment and Configuration
- Installing Airflow in local and cloud-based settings.
- Configuring Airflow with various executor types.
- Establishing metadata databases and external connections.
Utilizing the Airflow Interface and Command Line
- Navigating the Airflow web dashboard.
- Tracking DAG executions, task status, and logs.
- Administering Airflow via the CLI.
Development and Management of DAGs
- Building DAGs using the TaskFlow API.
- Implementing operators, sensors, and hooks.
- Managing dependencies and scheduling frequencies.
Integrating Airflow with Data and Cloud Platforms
- Linking databases, APIs, and message queues.
- Executing ETL pipelines through Airflow.
- Cloud-specific integrations: AWS, GCP, and Azure operators.
Monitoring and Observability
- Task logging and real-time tracking.
- Metric collection using Prometheus and Grafana.
- Configuring alerts and notifications via email or Slack.
Hardening Apache Airflow
- Role-Based Access Control (RBAC).
- Authentication methods including LDAP, OAuth, and SSO.
- Secrets management via Vault and cloud-based secret stores.
Scaling Apache Airflow
- Managing parallelism, concurrency, and task queues.
- Utilizing CeleryExecutor and KubernetesExecutor.
- Deploying Airflow on Kubernetes using Helm.
Production Best Practices
- Version control and CI/CD integration for DAGs.
- Testing strategies and debugging techniques.
- Ensuring reliability and performance at scale.
Troubleshooting and Performance Tuning
- Diagnosing failed DAGs and tasks.
- Improving DAG execution efficiency.
- Identifying and avoiding common pitfalls.
Recap and Future Directions
Requirements
- Proficiency in Python programming.
- Knowledge of data engineering or DevOps principles.
- Familiarity with ETL processes or workflow orchestration.
Target Audience
- Data scientists.
- Data engineers.
- DevOps and infrastructure engineers.
- Software developers.
Testimonials (7)
The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.
Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH
Course - Apache Airflow: Building and Managing Data Pipelines
The training was spot on. Very useful theory and exercices.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
