canonical: https://jentic.com/apis/amazonaws.com/aws-datapipeline # AWS Data Pipeline Jentic publishes the only available OpenAPI specification for AWS Data Pipeline, keeping it validated and agent-ready. AWS Data Pipeline configures and runs data-driven workflows that move and transform data between AWS services and on-premises sources. The API manages pipeline definitions, activations, task runners, and validation. AWS announced Data Pipeline as legacy in 2023 and recommends Step Functions, Glue Workflows, or MWAA for new work, but the API is still available for existing pipelines. ## For AI agents Manage AWS Data Pipeline definitions, activate and deactivate pipelines, and coordinate task runners. ## Scope Does not handle modern serverless orchestration or Spark-based ETL - use AWS Data Pipeline only for managing existing legacy pipelines; use Step Functions or Glue for new work. ## Capabilities - Create pipelines and put their definitions with PutPipelineDefinition - Activate, deactivate, and delete pipelines to control execution - Validate a pipeline definition before activation - Poll for and report on tasks executed by task runners - List pipelines and describe the status of objects within them - Add tags and query objects with arbitrary expressions ## Use cases ### Maintaining a Legacy ETL Pipeline Many organizations still operate AWS Data Pipeline workflows that move data between S3, RDS, Redshift, and on-premises sources on a schedule. The API allows operators to put updated definitions, activate or deactivate runs, and inspect status without using the deprecated console. Example prompt: Call PutPipelineDefinition with the updated PipelineObjects, then ActivatePipeline to start the next run. ### Pipeline Validation Before Promotion Before promoting a Data Pipeline definition from staging to production, teams validate that all referenced fields, schedules, and resources are well-formed. ValidatePipelineDefinition returns errors and warnings without activating the pipeline, which prevents broken changes from running. Example prompt: Call ValidatePipelineDefinition with the candidate PipelineObjects and return errors[] and warnings[] for review. ### Custom Task Runner Coordination Some teams run Data Pipeline activities on their own EC2 instances or on-premises hosts via the task runner protocol. PollForTask, ReportTaskProgress, and SetTaskStatus let custom task runners pull work, report heartbeat progress, and report success or failure. Example prompt: Call PollForTask with a Worker Group, execute the returned activity, then call SetTaskStatus with FINISHED. ### Agent-Assisted Migration Off Data Pipeline An AI agent helping a team migrate off Data Pipeline can list every pipeline, fetch each definition for review, and deactivate ones that have been replaced by Step Functions or Glue Workflows. Through Jentic, the agent issues each operation as one structured call. Example prompt: Call ListPipelines, then for each pipelineId call GetPipelineDefinition and DescribePipelines, and deactivate any pipeline tagged 'replaced=true'. ## Key endpoints | Method | Path | Description | | --- | --- | --- | | POST | /#X-Amz-Target=DataPipeline.CreatePipeline | Create a new pipeline | | POST | /#X-Amz-Target=DataPipeline.PutPipelineDefinition | Set the pipeline definition | | POST | /#X-Amz-Target=DataPipeline.ValidatePipelineDefinition | Validate a pipeline definition | | POST | /#X-Amz-Target=DataPipeline.ActivatePipeline | Activate a pipeline | | POST | /#X-Amz-Target=DataPipeline.DeactivatePipeline | Deactivate a pipeline | | POST | /#X-Amz-Target=DataPipeline.DescribePipelines | Describe pipeline state | | POST | /#X-Amz-Target=DataPipeline.PollForTask | Poll for the next task as a task runner | ## Key resources - **Pipeline** — Create, list, describe, activate, deactivate, and delete pipelines. - **PipelineDefinition** — Put, get, and validate pipeline definitions. - **PipelineObject** — Describe and query pipeline objects with QueryObjects and DescribeObjects. - **TaskRunner** — PollForTask, ReportTaskProgress, ReportTaskRunnerHeartbeat, and SetTaskStatus for custom task runners. ## Why Jentic - **Setup:** Wiring AWS Data Pipeline by hand means implementing AWS Signature v4 request signing, resolving the regional datapipeline host, and setting the X-Amz-Target header to route each JSON action to the right operation. Through Jentic you install once, import AWS Data Pipeline from the API Directory, store the AWS credentials once, and your agent calls it. - **Permission scoping:** This API routes every call through a single endpoint with the action named in the request, covering operations like CreatePipeline, ValidatePipelineDefinition, and ActivatePipeline. Limit the agent to the operations it needs, such as DescribePipelines and ValidatePipelineDefinition, so DeactivatePipeline is not included unless you add it. - **Credential handling:** Your AWS credentials are stored once, encrypted, by your own Jentic One instance and injected at execution time. They never enter the agent's prompt, logs, or context. - **Discovery method:** Agents search Jentic by intent such as 'validate and activate a Data Pipeline' or 'describe pipeline state', and Jentic returns the matching AWS Data Pipeline operation with its input schema so the agent calls the right endpoint without browsing the AWS service reference. ## Related APIs - **AWS Step Functions** — Modern serverless workflow orchestration that AWS recommends over Data Pipeline. - **AWS Step Functions** — Modern serverless workflow orchestration that AWS recommends over Data Pipeline. - **Amazon Kinesis Analytics** — SQL and Apache Flink analytics on streaming AWS data. ## FAQ ### Why is there no official OpenAPI spec for AWS Data Pipeline? AWS does not publish an OpenAPI specification. Jentic generates and maintains this spec so that AI agents and developers can call AWS Data Pipeline via structured tooling. It is validated against the live API and kept up to date. Get started with Jentic One, the self-hosted execution layer. ### What authentication does the AWS Data Pipeline API use? AWS Signature v4 (HMAC) with IAM permissions on datapipeline:* actions plus pass-through permissions for the AWS resources the pipeline uses (S3, EC2, EMR, RDS, Redshift). Jentic stores the AWS credentials in its vault and signs each request. ### Can I activate and deactivate pipelines with the API? Yes. ActivatePipeline starts the next scheduled execution and DeactivatePipeline halts further runs without deleting the pipeline. DescribePipelines reports current state. ### What are the rate limits for the AWS Data Pipeline API? Per-account, per-region TPS limits apply, and PollForTask is the rate-limiting operation for custom task runner fleets - back off when no task is returned. AWS treats Data Pipeline as legacy, so capacity may be lower than newer services. ### How do I validate a pipeline definition through Jentic? Search Jentic for 'validate a Data Pipeline definition before activating it', load the ValidatePipelineDefinition schema, and execute it with the pipelineId and PipelineObjects. Errors and warnings come back in structured arrays so you can fail closed before activation. ### Should I use AWS Data Pipeline for new workloads? AWS announced Data Pipeline as legacy in 2023 and recommends AWS Step Functions, AWS Glue Workflows, or Amazon MWAA (managed Airflow) for new work. The API remains supported for existing pipelines. ### Can I limit what my agent is allowed to do with the AWS Data Pipeline API? Yes. Because you run Jentic One yourself, your own rules decide which AWS Data Pipeline operations the agent may call, even though every request routes through a single endpoint with the action named inside it. You can allow read-and-check operations like DescribePipelines and ValidatePipelineDefinition while withholding state-changing ones such as ActivatePipeline, DeactivatePipeline, or CreatePipeline. The agent only ever reaches the operations you have explicitly granted, using the AWS credentials your instance holds.