How to Monitor Apache Airflow with Prometheus and Grafana: A Step-by-Step Guide

Apache Airflow is a key tool in data engineering and MLOps. It helps manage and schedule complex workflows efficiently. However, as your DAGs (Directed Acyclic Graphs) grow in size and complexity, monitoring them becomes increasingly challenging.

That’s where Prometheus and Grafana come in. These tools provide real-time visibility into Airflow’s performance, helping you track metrics and identify issues before they impact your pipelines.

Real-time monitoring ensures your workflows remain efficient, prevents bottlenecks, and optimizes resource utilization, critical for scaling and maintaining reliability in production environments.

This guide shows how to connect Airflow, Prometheus, and Grafana using StatsD Exporter. By the end, you’ll be able to track task duration, DAG success rate, scheduler delay, and more. 

Why Monitor Airflow? 

Monitoring keeps your DAGs running smoothly. It helps you spot issues early and use resources better. 

 With Prometheus and Grafana, you can: 

• See how long your DAGs take to run. 
• Track success and failure rates. 
• Watch scheduler delays and worker load. 
• Set alerts for slow or failed tasks. 
• Improve Airflow’s performance as it scales. 

Main Components 

• Apache Airflow – Orchestrates and runs workflows. 
• StatsD – Listens for metrics like counters and timers. 
• StatsD Exporter – Converts StatsD data into Prometheus format. 
• Prometheus – Collects and stores metrics. 
• Grafana – Displays the data in charts and dashboards. 

How It Works

•  Airflow sends metrics to StatsD. 
• StatsD Exporter exposes these at /metrics. 
• Prometheus collects them regularly. 
• Grafana reads the data and shows it visually. 

Step-by-Step Guide

Step 1: Configure Airflow 

Airflow supports StatsD out of the box. 

Edit your airflow.cfg file: 

[metrics] 
statsd_on = True 
statsd_host = statsd-exporter 
statsd_port = 9125 
statsd_prefix = airflow 

Or set these in docker-compose.yaml: 

environment: 
  - AIRFLOW__METRICS__STATSD_ON=True 
  - AIRFLOW__METRICS__STATSD_HOST=statsd-exporter 
  - AIRFLOW__METRICS__STATSD_PORT=9125 
  - AIRFLOW__METRICS__STATSD_PREFIX=airflow 

Step 2: Writing statsd exporter service and statsd config mapping

In docker-compose.yaml, write service for statsd-exporter 

service: 
statsd-exporter: 
image: prom/statsd-exporter:v0.21.0 
volumes: 
- ./configs/statsd.yaml:/home/statsd-mapping-configs.yaml 
entrypoint: ["/bin/sh", "-c", "--"] 
command: ["statsd_exporter --log.level debug --statsd.mapping-config=/home/statsd-mapping-configs.yaml"] 
ports: 
- 9102:9102 # scrape port 
- 9125:9125 # ingest port 
restart: always 
   

In statsd.yaml file: 

mappings: 
 
# Counts how many times the Airflow scheduler sends heartbeat signals. 
- match: "*.scheduler_heartbeat" 
match_metric_type: counter 
name: "af_agg_scheduler_heartbeat" 
labels: 
airflow_id: "$1" 
 
# Total number of DAGs loaded into the Airflow DagBag. 
- match: "*.dagbag_size" 
match_metric_type: gauge 
name: "af_agg_dagbag_size" 
labels: 
airflow_id: "$1" 
 
# Total time (in seconds) Airflow spent parsing all DAG files. 
- match: "*.dag_processing.total_parse_time" 
match_metric_type: gauge 
name: "af_agg_dag_processing_total_parse_time" 
labels: 
airflow_id: "$1" 
 
# Number of tasks currently running in the Airflow executor. 
- match: "*.executor.running_tasks" 
match_metric_type: gauge 
name: "af_agg_executor_running_tasks" 
labels: 
airflow_id: "$1" 
 
 

Step 3: Add Prometheus service and Configure 

In docker-compose.yaml, expose Prometheus service 

Service: 
prometheus: 
image: prom/prometheus:v2.26.0 
extra_hosts: 
- "host.docker.internal:host-gateway" 
volumes: 
- ./configs/prometheus.yaml:/etc/prometheus/prometheus.yaml 
- prometheus_data:/prometheus 
command: 
- '--config.file=/etc/prometheus/prometheus.yaml' 
- '--storage.tsdb.path=/prometheus' 
- '--web.console.libraries=/usr/share/prometheus/console_libraries' 
- '--web.console.templates=/usr/share/prometheus/consoles' 
ports: 
- 9092:9090 
restart: always 

Add this to prometheus.yml: 

global: 
scrape_interval: 5s 
evaluation_interval: 15s 
 
scrape_configs: 
- job_name: airflow 
scheme: http 
metrics_path: metrics 
static_configs: 
- targets: ['host.docker.internal:9102'] 
labels: 
airflow_id: 'airflow' 
 

In Prometheus webUI, execute some expression (for eg. af_agg_scheduler_heartbeat).

Step 4: Write Grafana service 

In docker-compose.yaml, expose grafana service 

service: 
grafana: 
image: grafana/grafana:6.7.2 
container_name: grafana 
volumes: 
- grafana_data:/var/lib/grafana 
environment: 
- GF_SECURITY_ADMIN_USER=admin 
- GF_SECURITY_ADMIN_PASSWORD=grafana 
- GF_USERS_ALLOW_SIGN_UP=false 
restart: always 
ports: 
- 3000:3000 

Now Run everything: 

docker compose up -d 

Step 5: Configure Grafana data source and create a new Dashboard 

Go to http://localhost:3000 and log in. Add a data source. Mention the HTTP URL of Prometheus (http://localhost:9092) and set Access as Browser. Save and test the data source. 

Then, create a new dashboard using a JSON model 

Conclusion 

Monitoring Airflow with Prometheus and Grafana makes your workflow easier to manage. You can track queue sizes, scheduler delays, retries, and failures. It helps detect issues before they grow. This setup makes your pipelines reliable, observable, and ready to scale. Using open-source tools keeps it simple and cost-effective.