Apache Airflow Interview Questions and Answers (2026)

Orchestration Interview Prep

Apache Airflow Interview Questions & Answers (2026)

The most-asked Airflow questions for data engineering roles — covering DAG design, scheduling, operators, sensors, XComs, and real-world workflow patterns.

📅 Updated April 2026  |  ⏱ 12 min read  |  🎯 All Levels

📋 Topics Covered

1. DAG & Core Concepts
2. Operators & Sensors
3. XComs & Task Communication
4. Scheduling & Execution
5. Production Best Practices

1. DAG & Core Concepts

Q1What is a DAG in Airflow? What makes it "acyclic"?

A DAG (Directed Acyclic Graph) is the core object in Airflow — a Python file that defines a workflow as a collection of tasks with dependencies.

Directed — Tasks have a defined order (task A must finish before task B).
Acyclic — No circular dependencies. Task A cannot eventually depend on itself. Airflow will reject a DAG with cycles.

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG(
    dag_id='daily_etl_pipeline',
    start_date=datetime(2026, 1, 1),
    schedule_interval='0 6 * * *',   # every day at 6am
    catchup=False,
    tags=['production', 'etl']
) as dag:

    extract = PythonOperator(task_id='extract', python_callable=extract_data)
    transform = PythonOperator(task_id='transform', python_callable=transform_data)
    load = PythonOperator(task_id='load', python_callable=load_data)

    extract >> transform >> load  # dependency chain

Q2What is the Airflow Executor and what are the types?

The Executor determines how and where tasks are run.

Executor	Best For	Notes
SequentialExecutor	Local dev only	One task at a time, SQLite backend
LocalExecutor	Small teams	Parallel tasks on one machine via subprocesses
CeleryExecutor	Multi-worker production	Distributed workers via Celery + Redis/RabbitMQ
KubernetesExecutor	Cloud-native	Each task runs in its own pod — scales to zero

Production recommendation: KubernetesExecutor (Airflow 2.x on GKE/EKS) or CeleryExecutor for on-prem/VM setups.

2. Operators & Sensors

Q3What is the difference between an Operator and a Sensor?

Operator — Defines what a task does: run Python code, submit a Spark job, call an API, execute SQL.
Sensor — A special operator that waits for a condition to be met (polls at intervals). Blocks downstream tasks until the condition is satisfied.

# Operator example — does work
from airflow.operators.python import PythonOperator
run_transform = PythonOperator(task_id='transform', python_callable=run_transform_fn)

# Sensor example — waits for a condition
from airflow.sensors.s3_key_sensor import S3KeySensor
wait_for_file = S3KeySensor(
    task_id='wait_for_input_file',
    bucket_name='my-data-lake',
    bucket_key='input/{{ ds }}/data.csv',  # uses Jinja templating
    aws_conn_id='aws_default',
    poke_interval=300,   # check every 5 minutes
    timeout=3600         # fail after 1 hour
)

wait_for_file >> run_transform  # only transform after file arrives

⚠️ Sensor Mode Default sensors use "poke" mode which occupies a worker slot while waiting. Use mode='reschedule' to free the worker between checks — critical for long-wait sensors in production.

Q4What are the most commonly used Airflow operators?

Operator	Use Case
PythonOperator	Run any Python function
BashOperator	Run shell commands
SparkSubmitOperator	Submit a Spark job to a cluster
PostgresOperator / BigQueryOperator	Execute SQL queries
S3ToGCSOperator	Cross-cloud data transfer
DummyOperator	Placeholder for branching/grouping
BranchPythonOperator	Conditional path selection
TriggerDagRunOperator	Trigger another DAG

✅ Modern Approach In Airflow 2.x, prefer the @task decorator (TaskFlow API) over PythonOperator. It's cleaner, handles XComs automatically, and reduces boilerplate significantly.

3. XComs & Task Communication

Q5What are XComs and what are their limitations?

XComs (Cross-Communications) allow tasks to pass small values to each other through Airflow's metadata database.

# TaskFlow API — XComs happen automatically
from airflow.decorators import dag, task

@dag(schedule_interval='@daily', start_date=datetime(2026,1,1), catchup=False)
def etl_pipeline():
    @task
    def extract():
        return {"record_count": 5000, "source_file": "s3://bucket/data.csv"}

    @task
    def transform(extracted_data):
        count = extracted_data["record_count"]
        print(f"Transforming {count} records")
        return count * 2

    @task
    def load(transformed_count):
        print(f"Loading {transformed_count} records")

    data = extract()
    count = transform(data)
    load(count)

etl_pipeline()

Limitations (critical to know for interviews):
• Stored in Airflow's metadata DB (SQLite/MySQL/PostgreSQL) — NOT designed for large data.
• Maximum practical size: a few KB. Passing a DataFrame via XCom will crash your metadata DB.
• Correct pattern: save large data to S3/GCS/database and pass only the path/ID via XCom.

4. Scheduling & Execution

Q6What is catchup in Airflow and when would you disable it?

catchup=True (default) — When a DAG is turned on, Airflow will backfill and run all missed DAG runs between start_date and today. Useful for historical data ingestion.

catchup=False — Only run from the current schedule interval forward. Use this for real-time pipelines where processing old dates makes no sense (e.g., a daily report DAG).

Rule: Set catchup=False for 90% of pipelines unless you specifically need backfilling.

Q7What does execution_date mean in Airflow? Why is it confusing?

execution_date (Airflow 1.x) / data_interval_start (Airflow 2.x) is the start of the scheduling interval, NOT the actual time the task ran.

For a DAG with schedule_interval='@daily':
• The run for 2026-04-01 (execution_date = 2026-04-01 00:00:00) will actually trigger on 2026-04-02 (after the interval closes).

This "off by one" behavior trips up many engineers. Always use {{ ds }} (the date string of the execution) in your SQL/file paths to reference the correct data date.

5. Production Best Practices

Q8How would you handle a DAG that runs for too long or has too many retries?

with DAG(
    dag_id='monitored_pipeline',
    dagrun_timeout=timedelta(hours=4),  # fail entire DAG after 4 hours
    default_args={
        'retries': 2,
        'retry_delay': timedelta(minutes=5),
        'retry_exponential_backoff': True,  # 5m, 10m, 20m delays
        'execution_timeout': timedelta(hours=1),  # per-task timeout
        'on_failure_callback': alert_slack,  # notify on failure
        'sla': timedelta(hours=2),  # SLA miss alert
    }
) as dag:
    ...

Additional strategies:
• Break large monolithic DAGs into smaller sub-DAGs or modular DAGs connected via TriggerDagRunOperator.
• Use max_active_runs=1 to prevent concurrent runs from overlapping.
• Set depends_on_past=True for tasks that must not run until the previous day's run succeeded.

🎯 Master More Data Engineering Topics

Practice free quizzes on SQL, Spark, PySpark, Hadoop and Networking — and get the 300Q PDF bundle for focused offline prep.

Visit the Blog → Get the PDF Bundle