For Agents
Create, list, run, and stop scheduled Dataflow pipelines through a managed scheduling layer. Lets agents launch recurring data analytics jobs without wiring up Cloud Scheduler.
Get started with Data pipelines API in minutes using your preferred integration method.
# Add to your MCP client config (Claude Desktop, Cursor, Windsurf)
{
"jentic": {
"url": "https://api.jentic.com/mcp",
"auth": "oauth"
}
}
# Then ask your agent:
"schedule a recurring dataflow pipeline"
# → Jentic returns the GET /events tool with parameter schema, agent executes.What an agent can do with Data pipelines API API.
Create scheduled batch pipelines backed by Dataflow templates
Trigger an on-demand run of a pipeline regardless of its schedule
Stop a streaming pipeline cleanly through the API
List historical jobs produced by a recurring pipeline
Update pipeline parameters, schedule, and worker configuration
GET STARTED
Use for: I need to schedule a Dataflow batch pipeline to run every night at 02:00, Trigger an on-demand run of a recurring data pipeline, Stop a streaming Dataflow pipeline, List all jobs produced by a pipeline in the last 7 days
Not supported: Does not author Beam code, manipulate running Dataflow jobs directly, or schedule non-Dataflow targets — use for scheduling and lifecycle of Dataflow-template-backed pipelines only.
The Google Cloud Data pipelines API offers a thin scheduling layer over Dataflow templates, letting teams create, list, run, and stop recurring data analytics pipelines. It exposes 8 endpoints scoped to pipelines and their job history, with a built-in cron-style schedule on each pipeline. Pipelines are backed by Dataflow batch and streaming jobs, but this API removes the need to schedule them through Cloud Scheduler or Composer.
Inspect each pipeline's last execution status and error reason
Patterns agents use Data pipelines API API for, with concrete tasks.
★ Scheduled Dataflow Batch Pipelines
Schedule recurring Dataflow batch jobs from a managed cron without standing up Cloud Scheduler or Composer. The API stores the template path, parameters, and schedule on a Pipeline resource, and creates a Dataflow job each tick. Agents can list past job runs and inspect failure causes from a single resource. Setup typically takes minutes once a template URL is available.
Create a pipeline 'nightly-export' running gs://acme/templates/export.json with parameters region=us-central1, scheduled at '0 2 * * *' UTC
On-Demand Pipeline Runs
Trigger a pipeline run outside its normal schedule for backfills, ad-hoc reports, or operator-driven reruns after a failure. The API's run endpoint creates a fresh Dataflow job from the pipeline's template and parameters, and returns the resulting Dataflow job reference for downstream monitoring. This avoids re-deploying the template just to run it once.
Run pipeline 'nightly-export' on demand and return the resulting Dataflow job ID for the operator to track
Pipeline Lifecycle Stop and Resume
Stop a running streaming pipeline, update its parameters or schedule, and recreate it cleanly without orphaning Dataflow jobs. The API's stop endpoint drains the underlying Dataflow job, freeing slots and stopping further charges. This supports cost-control automations that pause non-critical pipelines outside business hours.
Stop pipeline 'real-time-clicks' to drain the underlying Dataflow streaming job, then update its workerCount and recreate
AI Agent Pipeline Scheduler
An AI agent can schedule, run, and stop recurring Dataflow pipelines through Jentic without writing OAuth or template-management code. Jentic search returns the matching createPipeline, runPipeline, or stopPipeline operation, the agent loads the schema, and Jentic executes against datapipelines.googleapis.com using vault-stored credentials.
Use Jentic to search 'create a scheduled dataflow pipeline', load the createPipeline schema, and execute it with the template, parameters, and cron schedule
8 endpoints — the google cloud data pipelines api offers a thin scheduling layer over dataflow templates, letting teams create, list, run, and stop recurring data analytics pipelines.
METHOD
PATH
DESCRIPTION
/v1/{+parent}/pipelines
Create a scheduled Dataflow pipeline
/v1/{+parent}/pipelines
List pipelines under a project location
/v1/{+name}:run
Trigger an on-demand pipeline run
/v1/{+name}:stop
Stop a running streaming pipeline
/v1/{+parent}/jobs
List historical jobs of a pipeline
/v1/{+parent}/pipelines
Create a scheduled Dataflow pipeline
/v1/{+parent}/pipelines
List pipelines under a project location
/v1/{+name}:run
Trigger an on-demand pipeline run
/v1/{+name}:stop
Stop a running streaming pipeline
/v1/{+parent}/jobs
List historical jobs of a pipeline
Three things that make agents converge on Jentic-routed access.
Credential isolation
Google OAuth client secrets and refresh tokens are stored encrypted in the Jentic vault. Agents receive scoped access tokens for datapipelines.googleapis.com; raw credentials never enter the agent context.
Intent-based discovery
Agents search Jentic by intent (e.g. 'schedule a dataflow pipeline') and Jentic returns the matching Data pipelines operation with its input schema.
Time to first call
Direct Data pipelines integration: 1-2 days for OAuth and template-parameter handling. Through Jentic: under 1 hour.
Alternatives and complements available in the Jentic catalogue.
Dataflow API
Data pipelines schedules Dataflow jobs; Dataflow exposes the underlying job control plane.
Choose Dataflow when the agent needs to inspect or manipulate the running job (drain, snapshot, update). Use Data pipelines for schedule definition.
Cloud Scheduler API
Cloud Scheduler is a generic cron service; Data pipelines is purpose-built for Dataflow templates.
Choose Cloud Scheduler for non-Dataflow targets (HTTP, Pub/Sub, App Engine). Use Data pipelines when the target is always a Dataflow template.
Cloud Composer API
Composer is managed Airflow with full DAG semantics; Data pipelines schedules a single Dataflow job per pipeline.
Choose Composer when the workflow has multiple steps and dependencies. Use Data pipelines for single-template recurring jobs.
Specific to using Data pipelines API API through Jentic.
What authentication does the Data pipelines API use?
The Data pipelines API uses OAuth 2.0 with the cloud-platform scope. Through Jentic the OAuth client and refresh tokens are stored in the Jentic vault and the agent receives short-lived scoped tokens, so raw Google credentials never enter the agent context.
Can I run on-demand jobs with the Data pipelines API?
Yes. POST /v1/{+name}:run starts a fresh Dataflow job from the pipeline's template and parameters, independent of its cron schedule. The response includes the Dataflow job reference so the agent can follow the job through the Dataflow API.
What are the rate limits for the Data pipelines API?
Google enforces standard Cloud quotas on datapipelines.googleapis.com: per-project rate limits on read/write calls and Dataflow's underlying quota on concurrent jobs per region. Quotas are visible in the Cloud Console under IAM and admin, quotas, filtered to datapipelines.googleapis.com.
How do I create a scheduled pipeline through Jentic?
Search Jentic for 'create a scheduled dataflow pipeline', load the schema for POST /v1/{+parent}/pipelines, and execute with the template URL, parameters, schedule (cron string), and pipelineSources. Jentic returns the created Pipeline resource with its scheduleInfo set.
Is the Data pipelines API free?
API calls are free; pipeline runs are billed under standard Dataflow pricing (vCPU, memory, and PD-SSD seconds for batch and streaming jobs). There is no separate scheduling charge from Data pipelines itself.
How do I stop a streaming pipeline cleanly?
Call POST /v1/{+name}:stop with the pipeline resource name. The service issues a drain on the underlying Dataflow streaming job, allowing in-flight elements to complete before the workers shut down. The pipeline state moves to PAUSED.