canonical: https://jentic.com/apis/ibm.com/datastage-api

# IBM DataStage API

IBM DataStage on IBM Cloud is a managed ETL and data integration service used to design, compile, and run data flows that move and transform data between sources and targets. The REST API exposes data integration flows and DataStage jobs so external tools can list, clone, compile, run, and monitor them programmatically. It is aimed at platform teams who need to embed DataStage runs inside larger data pipelines or trigger them from orchestrators and AI agents.

## For AI agents

List, compile, run, and monitor IBM DataStage flows and jobs on IBM Cloud - automate ETL execution and inspect run logs without using the DataStage UI.

## Scope

Does not handle data warehouse storage, BI reporting, or low-level Spark execution - use for managing and running DataStage flows and jobs only.

## Capabilities

- List and inspect data integration flows via `/v3/data_intg_flows`
- Clone an existing flow into a new flow via `/v3/data_intg_flows/{flow_id}/clone`
- Compile a flow before execution via `/v3/data_intg_flows/{flow_id}/compile`
- Trigger a DataStage job run via `/v3/ds_jobs/{job_id}/run`
- Inspect job run history and status via `/v3/ds_jobs/{job_id}/runs` and `/runs/{run_id}`
- Pull execution logs for a specific run via `/v3/ds_jobs/{job_id}/runs/{run_id}/logs`
- Encrypt or decrypt migration payloads via `/v3/migration/encrypt` and `/v3/migration/decrypt`

## Use cases

### Pipeline Orchestration

Trigger DataStage jobs from an external orchestrator such as Airflow or a Jentic agent so that ETL execution sits alongside the rest of the data pipeline rather than running on a DataStage-only schedule. The API provides `/v3/ds_jobs/{job_id}/run` to start a job and `/v3/ds_jobs/{job_id}/runs/{run_id}` to poll for completion, which is enough to wire DataStage into a dependency-aware DAG.

Example prompt: Call `/v3/ds_jobs/{job_id}/run` for the daily-customer-load job, then poll `/v3/ds_jobs/{job_id}/runs/{run_id}` every 30 seconds until status is Completed.

### Run Diagnostics

Pull logs and run metadata for failed DataStage executions so an on-call engineer or agent can diagnose without opening the DataStage UI. The `/v3/ds_jobs/{job_id}/runs` endpoint returns the recent run history, and `/v3/ds_jobs/{job_id}/runs/{run_id}/logs` returns the per-run log payload that contains row counts, warnings, and stack traces.

Example prompt: When a DataStage job run ends in Failed, fetch `/v3/ds_jobs/{job_id}/runs/{run_id}/logs` and post the last 50 log lines to the on-call channel.

### Flow Cloning for Environments

Clone a production data integration flow into a new flow ID so it can be edited safely in a non-production project. The `/v3/data_intg_flows/{flow_id}/clone` endpoint copies the flow definition without copying credentials, which makes it suitable for promoting changes through dev, test, and prod environments.

Example prompt: Clone the data integration flow with id daily-orders-prod into a new flow named daily-orders-staging in the staging project.

### AI Agent Data Pipeline Operator

An AI agent monitors DataStage runs, retries transient failures, and surfaces persistent ones to a human owner. Through Jentic, the agent searches by intent, loads the run and logs schemas, and executes calls without holding the raw IBM Cloud IAM token. Typical agent integration is under one hour because only a handful of operations are needed to cover the run-and-monitor surface.

Example prompt: Search Jentic for 'run a DataStage job', load the `/v3/ds_jobs/{job_id}/run` schema, run the job, and on Failed status fetch the logs and summarise the error.

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| GET | `/v3/data_intg_flows` | List data integration flows |
| POST | `/v3/data_intg_flows/{flow_id}/clone` | Clone a flow |
| POST | `/v3/data_intg_flows/{flow_id}/compile` | Compile a flow |
| POST | `/v3/ds_jobs/{job_id}/run` | Run a DataStage job |
| GET | `/v3/ds_jobs/{job_id}/runs` | List runs for a job |
| GET | `/v3/ds_jobs/{job_id}/runs/{run_id}` | Get a specific run |
| GET | `/v3/ds_jobs/{job_id}/runs/{run_id}/logs` | Fetch run logs |

## Key resources

- **Flows** — Manage data integration flow definitions
- **Jobs** — Run, list, and monitor DataStage jobs
- **Job Runs** — Inspect run history, status, and logs
- **Migration** — Encrypt and decrypt migration payloads

## Why Jentic

- **Setup:** Wiring the IBM DataStage API by hand means handling its bearer-token auth against the cloud data_intg endpoint and mapping its flow and job run routes yourself. Through Jentic you install once, import the IBM DataStage API from the API Directory, store the token once, and your agent calls it.
- **Permission scoping:** DataStage puts the flow id and job id in the URL path (`/data_intg_flows/{flow_id}`, `/ds_jobs/{job_id}`), so a rule can pin your agent to reading flows and run logs for one job. You choose the operations it may call, so actions like cloning or compiling a flow or triggering a job run are not included unless you add them.
- **Credential handling:** Your DataStage token is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'run a DataStage job' or 'check a job run's logs', and Jentic returns the matching operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **IBM webMethods Integration API** — webMethods covers iPaaS and SaaS connectors; DataStage covers heavy ETL and data warehouse loads.
- **Azure Logic Apps Management API** — Azure Logic Apps is a managed integration and workflow service; DataStage is a managed ETL service.
- **IBM API Connect Management API** — API Connect manages the API lifecycle in front of services like DataStage.

## FAQ

### What authentication does the IBM DataStage API use?

The DataStage REST API uses HTTP bearer authentication with an IBM Cloud IAM token. Through Jentic, the IAM credentials are stored in your Jentic One instance and the agent calls DataStage via a scoped Jentic credential, never seeing the raw IAM token.

### Can I run a DataStage job from this API?

Yes. POST to `/v3/ds_jobs/{job_id}/run` to start a run, then poll `/v3/ds_jobs/{job_id}/runs/{run_id}` for status. The run record returns success, failure, and timing fields that an external scheduler can use to gate downstream steps.

### How do I retrieve logs for a failed DataStage run through Jentic?

Search Jentic for 'fetch DataStage run logs', load the `/v3/ds_jobs/{job_id}/runs/{run_id}/logs` schema, and execute with the job ID and run ID. Install the SDK with pip install jentic and use the async search, load, execute pattern.

### Does the DataStage API support cloning flows between projects?

Yes. POST to `/v3/data_intg_flows/{flow_id}/clone` with the target project ID to copy a flow definition. The clone does not carry credentials, which is what makes it safe for promoting flows from production to a non-production project.

### What are the rate limits for the IBM DataStage API?

IBM Cloud applies account-level rate limits across IAM-protected services rather than a fixed DataStage limit. Treat 429 responses as backpressure, retry with exponential backoff, and avoid polling run status more often than every few seconds for the same run.

### How do I compile a flow before running it?

POST to `/v3/data_intg_flows/{flow_id}/compile` to compile a flow into an executable job. A flow must compile successfully before `/v3/ds_jobs/{job_id}/run` will produce a useful run, so this endpoint is part of the standard deploy-then-run sequence.

### Can I limit what my agent is allowed to do with the IBM DataStage API?

Yes. Jentic One is self-hosted by you, so your own rules decide which DataStage operations and credentials the agent may use. Because DataStage puts the flow id and job id in the URL path (`/data_intg_flows/{flow_id}`, `/ds_jobs/{job_id}`), you can pin the agent to read-only calls like listing flows and pulling run logs for a single job. Higher-impact operations such as cloning a flow, compiling a flow, or triggering a job run stay off limits unless you explicitly grant them.