Getting Started with Apache Airflow Using Docker Compose

Apache Airflow is a workflow orchestration platform. You describe a workflow as a DAG, or directed acyclic graph, and Airflow schedules the tasks, tracks their state, retries failures, and gives you a UI for inspecting what happened.

This walkthrough sets up a local Airflow 3.2.2 environment with Docker Compose and then builds three small Python DAGs. The goal is not to make a production deployment. The goal is to make the core Airflow ideas tangible: DAGs, tasks, schedules, dependencies, logs, XComs, and branching.

Follow along by cloning this repo locally.

What We Are Building

The local stack uses:

airflow-apiserver for the web UI and API.
airflow-scheduler to decide when tasks should run.
airflow-dag-processor to parse DAG files.
airflow-worker to execute tasks through Celery.
airflow-triggerer for deferrable tasks.
postgres for Airflow metadata.
redis as the Celery message broker.

The repository also includes three DAGs:

hello_airflow: the first DAG, using classic operators.
taskflow_sales_summary: a Pythonic TaskFlow DAG.
branching_quality_check: a branching DAG that skips one path at runtime.

The setup is based on the official Apache Airflow Docker Compose quick start: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

Project Layout

.
|-- docker-compose.yaml
|-- dags/
|   |-- hello_airflow.py
|   |-- taskflow_sales_summary.py
|   `-- branching_quality_check.py
|-- data/
|-- logs/
|-- plugins/
|-- config/
|-- Makefile
`-- README.md

The important directory is dags/. Airflow scans this folder and imports each Python file to discover DAG definitions.

The data/ directory is a shared volume for tutorial output. One of the DAGs writes a JSON summary there so you can see a task create an artifact.

Step 1: Start the Airflow Environment

Make a copy of .env.example to .env

cp .env.example .env

From the project root, initialize the database and create the default admin user:

docker compose up airflow-init

Then start the services:

docker compose up -d

Open the UI at http://localhost:8080 and sign in with:

username: airflow
password: airflow

Check the containers:

docker compose ps

The API server, scheduler, DAG processor, worker, triggerer, Postgres, and Redis services should be running or healthy.

Step 2: Understand the First DAG

Open dags/hello_airflow.py. A simplified version of the structure looks like this:

with DAG(
    dag_id="hello_airflow",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
) as dag:
    start = EmptyOperator(task_id="start")
    hello_python = PythonOperator(
        task_id="hello_python",
        python_callable=greet_airflow,
    )
    print_runtime_context = BashOperator(
        task_id="print_runtime_context",
        bash_command="echo 'Logical date: {{ ds }}'",
    )
    finish = EmptyOperator(task_id="finish")

    start >> hello_python >> print_runtime_context >> finish

The DAG object describes the workflow. The operators create tasks. The >> operator defines the dependency order.

In this example:

start is a marker task.
hello_python calls a Python function.
print_runtime_context prints templated Airflow values.
finish marks the end of the workflow.

The expression {{ ds }} is a Jinja template. Airflow renders it at runtime with the logical date for the DAG run.

Run the DAG from the command line:

docker compose run --rm airflow-cli dags test hello_airflow 2026-01-01

Then open the UI, select hello_airflow, and inspect the graph and task logs.

Step 3: Use TaskFlow for Python Pipelines

Airflow also has the TaskFlow API, where decorated Python functions become tasks. Open dags/taskflow_sales_summary.py.

The DAG has three steps:

orders = extract_orders()
summary = summarize_orders(orders)
write_summary(summary)

Each function is decorated with @task. Airflow turns the function calls into task dependencies and stores returned values as XComs. That means summarize_orders receives the output from extract_orders, and write_summary receives the output from summarize_orders.

Run it:

docker compose run --rm airflow-cli dags test taskflow_sales_summary 2026-01-01

After it runs, check the shared output directory:

ls data

You should see a file like:

sales_summary_2026-01-01.json

This is still a tiny example, but it maps directly to real-world extract, transform, and load pipelines.

Step 4: Add Runtime Branching

Open dags/branching_quality_check.py. This DAG uses BranchPythonOperator to choose one downstream task:

def choose_quality_path(**context):
    day_of_month = int(context["ds_nodash"][-2:])
    if day_of_month % 2 == 0:
        return "publish_quality_report"
    return "open_review_ticket"

If the logical date falls on an even day, Airflow follows publish_quality_report. If it falls on an odd day, Airflow follows open_review_ticket. The branch that is not selected gets marked as skipped.

Run both paths:

docker compose run --rm airflow-cli dags test branching_quality_check 2026-01-01
docker compose run --rm airflow-cli dags test branching_quality_check 2026-01-02

In the UI, compare the graph views for each run. Branching is one of the fastest ways to see that Airflow is more than a cron replacement. It understands task state and downstream dependencies.

Step 5: Iterate Like a DAG Author

The normal development loop is:

Edit a file in dags/.
Wait for the DAG processor to parse it.
Open the Airflow UI and confirm the DAG imports cleanly.
Trigger a test run.
Read task logs and XCom values.

Useful commands:

docker compose run --rm airflow-cli dags list
docker compose run --rm airflow-cli dags list-import-errors
docker compose logs -f airflow-dag-processor airflow-scheduler

If a DAG does not show up, start with import errors. Most beginner DAG problems are normal Python errors: a bad import, a syntax error, or code that does too much work while Airflow is parsing the file.

Cleanup

Stop the environment:

docker compose down --volumes --remove-orphans

The --volumes flag removes the local Postgres metadata volume. Use it when you want a fresh Airflow database.

What to Try Next

Add a fourth task to taskflow_sales_summary that filters orders above a threshold.
Change the schedule on hello_airflow from @daily to @hourly.
Add a failed task intentionally, then observe retry behavior in the UI.
Move reusable Python logic into a separate module under plugins/.
Add a connection in the UI and read it from a Python task.

Once these pieces feel comfortable, the next Airflow topics to learn are sensors, datasets or asset-aware scheduling, custom operators, and deployment patterns for production.

Getting Started with Apache Airflow Using Docker Compose#

What We Are Building#

Project Layout#

Step 1: Start the Airflow Environment#

Step 2: Understand the First DAG#

Step 3: Use TaskFlow for Python Pipelines#

Step 4: Add Runtime Branching#

Step 5: Iterate Like a DAG Author#

Cleanup#

What to Try Next#