canonical: https://jentic.com/apis/apache.org/airflow

# Apache Airflow API (Stable)

Apache Airflow is the open-source workflow orchestration platform used to schedule and monitor data pipelines, ML training runs, and ETL jobs. The stable REST API exposes DAGs, DAG runs, task instances, connections, variables, datasets, pools, and XComs so external systems can trigger runs, inspect state, and manage Airflow's metadata without going through the web UI. Authentication options include HTTP basic, Google OpenID Connect, and Kerberos, configured per Airflow deployment. The base URL is the `/api/v1` path on whichever host runs the Airflow webserver.

## For AI agents

Trigger DAG runs, inspect task instances, and manage Airflow connections, variables, and datasets across an Apache Airflow deployment.

## Scope

Does not handle DAG authoring, code deployment, or the Airflow UI - use for triggering and inspecting runs, task instances, and Airflow metadata only.

## Capabilities

- Trigger a new DAG run with a custom configuration payload and capture the run id
- Inspect the state of a DAG run and its task instances, including logs and try numbers
- Pause, unpause, and update DAGs, including retrieving the parsed source
- Manage Airflow connections, variables, and pools used by tasks at runtime
- List datasets and their associated DAGs to understand data lineage across pipelines
- Clear or set the state of task instances to recover failed runs without UI access

## Use cases

### Trigger a pipeline from an external system

Kick off an Airflow DAG run from a service outside Airflow, for example a webhook from a SaaS app or an event from a message broker. The agent posts to `/dags/{dag_id}/dagRuns` with a config payload, then polls `/dags/{dag_id}/dagRuns/{dag_run_id}` for state until it reaches success or failure. This avoids hard-wiring schedules and keeps the trigger event observable in Airflow's UI.

Example prompt: POST to `/dags/sales_etl/dagRuns` with {"conf": {"date": "2026-06-09"}} and then GET `/dags/sales_etl/dagRuns/{dag_run_id}` every 30 seconds until state is success or failed

### Operational dashboard over Airflow state

Build a custom dashboard that surfaces DAG and task health beyond what the Airflow UI offers, for example aggregated SLA misses or per-team DAG ownership. The agent calls /dags to list DAGs, `/dags/{dag_id}/dagRuns` for run history, and `/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances` for fine-grained task state.

Example prompt: GET /dags, then for each dag GET `/dags/{dag_id}/dagRuns`?limit=20 and aggregate failed-run counts per team for a Slack digest

### Recover failed task instances

When a downstream service was briefly down, recover failed task instances by clearing them so Airflow reschedules. The agent identifies failed task instances via `/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances` and POSTs to `/dags/{dag_id}/clearTaskInstances` to reset them, optionally including downstream tasks.

Example prompt: GET `/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances` filtered to state=failed, then POST to `/dags/{dag_id}/clearTaskInstances` with the failed task ids and include_downstream=true

### Manage connections and variables programmatically

Manage Airflow connections (database URLs, API tokens, cloud credentials) and variables from automation rather than the web UI. Useful for environment promotion or rotating credentials. The agent uses /connections and /variables endpoints to create, update, and test entries, and POSTs `/connections/test` to validate before saving.

Example prompt: POST `/connections/test` with the new connection payload, then PATCH `/connections/{connection_id}` once the test returns success

### AI agent integration via Jentic

An agent that needs to operate an Airflow deployment from natural language can search Jentic for 'trigger an Airflow DAG' and Jentic returns the schema for `/dags/{dag_id}/dagRuns.` Jentic stores the deployment's auth (basic, OIDC, or Kerberos) and base URL so the agent does not handle credentials or hostnames.

Example prompt: Search Jentic for 'trigger an Airflow DAG', load the `/dags/{dag_id}/dagRuns` schema, and execute it for the user-named DAG

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/dags/{dag_id}/dagRuns` | Trigger a new DAG run |
| GET | `/dags/{dag_id}/dagRuns/{dag_run_id}` | Get a DAG run by id |
| GET | `/dags` | List all DAGs registered in the deployment |
| GET | `/dags/{dag_id}` | Fetch a single DAG's metadata |
| POST | `/dags/{dag_id}/clearTaskInstances` | Clear task instances to reschedule failed runs |
| GET | `/connections` | List Airflow connections |
| POST | `/connections/test` | Test a connection definition before saving |
| GET | `/variables` | List Airflow variables |

## Key resources

- **DAG** — List, fetch, pause, unpause, and update DAGs and retrieve their parsed source
- **DAGRun** — Trigger, list, fetch, and clear DAG runs with state filtering
- **TaskInstance** — Inspect task instances, set state, fetch logs, and resolve dependencies
- **Connection** — Create, update, list, delete, and test Airflow connections
- **Variable** — CRUD on Airflow variables used by DAG code at runtime
- **Dataset** — List datasets and their producing and consuming DAGs
- **Pool** — Manage execution pools that bound task concurrency
- **XCom** — Read XCom values exchanged between tasks in a DAG run

## Why Jentic

- **Setup:** Wiring the Airflow stable API by hand means picking among basic, OpenID Connect, and HTTP auth, pointing at your own Airflow deployment, and mapping dozens of run, task, and metadata routes yourself. Through Jentic you install once, import Airflow from the API Directory, store the credential once, and your agent calls it.
- **Permission scoping:** Airflow puts the DAG id in the URL path (`/dags/{dag_id}/dagRuns`), so a rule can pin your agent to one DAG: it can trigger and inspect runs for that DAG and nothing else. You choose the operations it may call, so destructive ones like clearing task instances are not included unless you add them.
- **Credential handling:** Your Airflow credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'trigger an Airflow DAG' or 'list failed task instances', and Jentic returns the matching Airflow operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Prefect API** — Workflow orchestration with a hosted control plane and Python-first flow definitions.
- **Temporal API** — Durable workflow execution with code-defined workflows and signals.
- **GitHub API** — Source the DAGs from a GitHub repo and react to commits.

## FAQ

### What authentication does the Airflow API use?

The spec declares three options: HTTP Basic, Google OpenID Connect, and Kerberos (negotiate). Which one is active depends on the deployment's auth backend in airflow.cfg. Through Jentic the chosen credential is stored in the encrypted Jentic One instance and attached server-side, so the agent never sees the raw secret.

### Can I trigger a DAG run with a custom config through the API?

Yes. POST to `/dags/{dag_id}/dagRuns` with a JSON body including a conf object that the DAG code reads via context['dag_run'].conf. The response returns the new dag_run_id which can be polled via GET `/dags/{dag_id}/dagRuns/{dag_run_id}` for state.

### How do I trigger an Airflow DAG through Jentic?

Search Jentic for 'trigger an Airflow DAG', load the schema for POST `/dags/{dag_id}/dagRuns`, and execute it with the dag id and the conf payload. Jentic injects the configured auth and base URL for your Airflow deployment.

### What are the rate limits for the Airflow API?

Apache Airflow itself does not enforce a global rate limit; throughput is bounded by the webserver's worker pool and the underlying metadata database. In practice deployments sit behind a reverse proxy that may add request limits.

### Can I clear failed task instances to retry them?

Yes. POST to `/dags/{dag_id}/clearTaskInstances` with a list of task ids and optional include_downstream and reset_dag_runs flags, and Airflow re-queues the cleared tasks. This is the API equivalent of the UI's clear button.

### Is the Airflow API free to use?

Apache Airflow is open source under the Apache 2.0 license, so the API itself is free. Cost is the infrastructure that runs the webserver, scheduler, and workers - or a managed offering such as Astronomer or AWS MWAA.

### Can I limit what my agent is allowed to do with the Airflow API?

Yes. Because you run Jentic One yourself, your own rules decide which Airflow operations and credential the agent can use. Airflow puts the DAG id in the URL path, such as `/dags/{dag_id}/dagRuns`, so a rule can pin the agent to a single DAG and let it trigger and inspect runs for that DAG only. You also choose the exact operations it may call, so destructive ones like clearing task instances via `/dags/{dag_id}/clearTaskInstances` stay off unless you add them.