canonical: https://jentic.com/apis/googleapis.com/dataproc

# Google Cloud Dataproc API

Cloud Dataproc is Google's managed Hadoop and Spark service for running batch, streaming, and interactive analytics workloads. The API exposes 65 endpoints covering clusters, jobs, workflow templates, autoscaling policies, batches (serverless Spark), and sessions (interactive Jupyter). It supports cluster create-start-stop-repair lifecycle plus job submission for Spark, PySpark, Hive, Pig, Presto, and SparkR engines.

## For AI agents

Provision and operate Hadoop and Spark clusters, submit jobs, and run serverless Spark batches in Google Cloud. Lets agents drive analytics workloads without managing infrastructure by hand.

## Scope

Does not run BigQuery SQL, manage non-Dataproc compute, or schedule cross-service workflows - use for Dataproc cluster lifecycle, job submission, serverless batches, and workflow templates only.

## Capabilities

- Provision Dataproc clusters with custom machine types, autoscaling, and initialization actions
- Submit Spark, PySpark, Hive, Pig, Presto, and SparkR jobs to a cluster
- Run serverless Spark batches without standing up a cluster
- Define workflow templates that orchestrate multi-step Dataproc jobs
- Diagnose, repair, start, and stop clusters through the API
- Manage interactive Jupyter sessions for exploratory analytics

## Use cases

### On-Demand Spark Cluster Workloads

Spin up a Dataproc cluster, submit Spark or PySpark jobs against Cloud Storage data, and tear the cluster down once the job finishes. The API exposes clusters create, jobs submit, and clusters delete in a tight loop suitable for ephemeral analytics. Cluster startup is typically 90 seconds; deleting the cluster after the job completes keeps cost bounded to job runtime.

Example prompt: Create cluster 'etl-2026-06-10', submit a PySpark job with mainPythonFileUri gs://acme/jobs/etl.py, wait for state DONE, and delete the cluster

### Serverless Spark Batches

Run Spark workloads without managing cluster infrastructure using Dataproc Batches. POST /v1/{+parent}/batches accepts a Spark, PySpark, SparkR, or SparkSQL payload plus runtime config and a Cloud Storage staging bucket. The service provisions transient infrastructure for the run and bills only for the batch's runtime.

Example prompt: POST /v1/{+parent}/batches with a PySpark batch pointing at gs://acme/jobs/score.py and runtimeConfig version 2.2, then poll the batch resource until state is SUCCEEDED

### Multi-Step Workflow Templates

Define a Dataproc workflow template that creates a managed cluster, runs a sequence of jobs (with dependencies), and deletes the cluster on completion. Workflow templates support parameterised execution so the same template runs against different inputs. Use the instantiate or instantiateInline endpoints to start a run from CI or an agent.

Example prompt: Instantiate workflow template 'nightly-etl' with parameter input_date=2026-06-09 and watch the resulting Operation until it reaches DONE

### AI Agent Analytics Workload Operator

An AI agent can run Spark workloads on demand through Jentic without operator-written cluster code. Jentic search returns the matching cluster, job, batch, or workflow operation, the agent loads the schema, and Jentic executes against dataproc.googleapis.com using vault-stored credentials. This collapses the multi-day setup of Dataproc OAuth and operation polling into a single agent run.

Example prompt: Use Jentic to search 'submit a pyspark job to dataproc', load the submit schema, and execute it against the named cluster with the provided main file URI

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | /v1/projects/{projectId}/regions/{region}/clusters | Create a Dataproc cluster |
| POST | /v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:diagnose | Diagnose a cluster and produce a diagnostic tarball |
| POST | /v1/projects/{projectId}/regions/{region}/jobs | Submit a job to a cluster |
| POST | /v1/{+parent}/batches | Run a serverless Spark batch |
| POST | /v1/{+parent}/workflowTemplates | Create a workflow template |
| POST | /v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:stop | Stop a running cluster |

## Key resources

- **Clusters** — Provision, repair, start, stop, and delete Dataproc clusters
- **Jobs** — Submit Spark, PySpark, Hive, Pig, Presto, and SparkR jobs
- **Batches** — Serverless Spark batches that run without a long-lived cluster
- **WorkflowTemplates** — Multi-step orchestrated Dataproc workflows
- **AutoscalingPolicies** — Reusable autoscaling rules attached to clusters
- **Sessions** — Interactive Jupyter sessions for exploratory analytics

## Why Jentic

- **Setup:** Wiring the Cloud Dataproc API by hand means setting up Google OAuth, refreshing short-lived scoped tokens instead of shipping a JSON key, and building project and region paths against dataproc.googleapis.com. Through Jentic you install once, import the Cloud Dataproc API from the API Directory, store the Google credential once, and your agent calls it.
- **Permission scoping:** The API puts the project, region, and cluster name in the URL path (/v1/projects/{projectId}/regions/{region}/clusters/{clusterName}), so a rule can pin your agent to one project, region, or cluster. You choose the operations it may call, so it can create clusters and submit jobs while stopping a cluster is not included unless you add it.
- **Credential handling:** Your Google OAuth credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'submit a PySpark job to Dataproc', and Jentic returns the matching Cloud Dataproc operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Dataflow API** — Dataflow runs Apache Beam pipelines; Dataproc runs Spark and Hadoop. Beam vs Spark choice drives the pick.
- **BigQuery API** — BigQuery is the typical sink for Dataproc-derived datasets and the source for SparkSQL reads.
- **Cloud Storage API** — Cloud Storage holds inputs, outputs, and staging artifacts for Dataproc jobs and batches.

## FAQ

### What authentication does the Cloud Dataproc API use?

The Cloud Dataproc API uses OAuth 2.0 with the cloud-platform scope. Through Jentic the OAuth client and refresh tokens are stored in your Jentic One instance and the agent receives short-lived scoped access tokens, so raw Google credentials never enter the agent context.

### Can I run serverless Spark with the Cloud Dataproc API?

Yes. POST /v1/{+parent}/batches submits a PySpark, Spark, SparkR, or SparkSQL workload with runtime and environment configuration; Dataproc provisions transient infrastructure for the duration of the run and tears it down automatically.

### What are the rate limits for the Cloud Dataproc API?

Google enforces standard Cloud quotas on dataproc.googleapis.com: per-project rate limits on read/write calls plus quotas on concurrent clusters, jobs, and batches per region. Quotas are visible in the Cloud Console under IAM and admin, quotas, filtered to dataproc.googleapis.com.

### How do I submit a PySpark job through Jentic?

Search Jentic for 'submit a pyspark job to dataproc', load the schema for POST /v1/projects/{projectId}/regions/{region}/jobs, and execute with placement.clusterName and pysparkJob.mainPythonFileUri set. Jentic returns the Job resource with its job ID for status polling.

### Is the Cloud Dataproc API free?

API calls are free; clusters, batches, and sessions are billed by underlying Compute Engine vCPU and memory plus a per-vCPU-hour Dataproc premium. Batches and Sessions are billed per runtime second only, with no charge while idle.

### How do I diagnose a failing cluster?

Call POST /v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:diagnose to produce a diagnostic tarball in Cloud Storage. The response contains the gs:// path of the tarball, which holds master and worker logs plus a YARN dump for offline analysis.

### Can I limit what my agent is allowed to do with the Cloud Dataproc API?

Yes. Because you self-host Jentic One, your own rules decide which Dataproc operations and credentials the agent may use. Since the API carries the project, region, and cluster name in the URL path (/v1/projects/{projectId}/regions/{region}/clusters/{clusterName}), you can pin the agent to a single project, region, or cluster. You also pick the exact operations it may call, so it can create clusters and submit jobs while stopping or deleting a cluster stays off limits unless you add those operations.