canonical: https://jentic.com/apis/googleapis.com/google-dataproc-metastore

# Google Dataproc Metastore API

Dataproc Metastore is Google Cloud's managed Apache Hive metastore service that provides a centralised metadata catalog for Dataproc, BigQuery federation, and other open-source data engines. The API covers metastore service lifecycle, backups, metadata import, federations across multiple metastores, and IAM policy management on individual services. Long-running operations are returned for state-changing calls and most resources are scoped to a project location.

## For AI agents

Provision and manage Dataproc Metastore services that act as a managed Hive metastore for Dataproc and BigQuery: create services, run backups, import metadata, and federate metastores.

## Scope

Does not handle Hive query execution, table data files, or cluster compute - use for metastore service lifecycle, backups, metadata imports, federations, and IAM policy only.

## Capabilities

- Provision and delete managed Hive metastore services in a Google Cloud project location
- Create and list backups of metastore services for disaster recovery
- Import existing Hive metadata from Cloud Storage into a metastore service
- Create and manage federations that combine multiple metastores under one endpoint
- Track migration executions when moving from a self-managed Hive metastore
- Manage IAM policy and test IAM permissions on individual metastore services

## Use cases

### Managed Hive metastore for Dataproc

Provision a Dataproc Metastore service so that ephemeral Dataproc clusters share a persistent Hive metadata catalog rather than each cluster maintaining its own. This lets jobs run against the same tables and partitions across cluster lifetimes and integrates cleanly with BigQuery federation queries.

Example prompt: Create a metastore service named lake-meta under projects/acme/locations/europe-west1 with hive metastore version 3.1.2 and tier DEVELOPER, then poll the returned operation

### Metastore backup and restore

Run scheduled backups of a metastore service so that table definitions, partitions, and database metadata can be restored after an accidental drop or schema corruption. The backups resource is per-service and pairs with the import flow for restore-style workflows.

Example prompt: Create a backup of metastore service projects/acme/locations/europe-west1/services/lake-meta and confirm the operation completes

### Metastore federation across regions

Combine multiple Dataproc Metastore services under a single federation endpoint so that data engines see one logical metastore for queries spanning regions or business units. Federations are useful for organisations that keep separate metastores per data domain but want unified access.

Example prompt: Create a federation named global-lake under projects/acme/locations/global with backing services from europe-west1 and us-central1

### Agent integration via Jentic

AI agents managing data lake metadata can use the Dataproc Metastore API through Jentic to provision services, run scheduled backups, or import metadata from Cloud Storage URIs without holding the underlying OAuth refresh token in their prompt.

Example prompt: List metadataImports for service lake-meta and start a new import from gs://acme-lake/metadata/2026-06.dump if no import has run in the last 7 days

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| GET | /v1/{+parent}/services | List metastore services |
| DELETE | /v1/{+name} | Delete a metastore service or other resource |
| GET | /v1/{+parent}/backups | List backups for a service |
| GET | /v1/{+parent}/federations | List federations in a location |
| GET | /v1/{+parent}/metadataImports | List metadata import runs |
| GET | /v1/{+parent}/migrationExecutions | List migration executions |
| GET | /v1/{+resource}:getIamPolicy | Read IAM policy on a service |
| POST | /v1/{+name}:cancel | Cancel a long-running operation |

## Key resources

- **services** — Provision, list, update, and delete Dataproc Metastore services
- **backups** — Create and list backups of metastore services
- **metadataImports** — Import Hive metadata from Cloud Storage into a metastore service
- **federations** — Combine multiple metastores under a single federation endpoint
- **migrationExecutions** — Track migration runs from self-managed Hive metastores
- **iamPolicy** — Manage IAM policy and test permissions on metastore services

## Why Jentic

- **Setup:** Wiring the Dataproc Metastore API by hand means setting up Google OAuth 2.0 with the cloud-platform scope, refreshing bearer tokens, and polling long-running service operations yourself against the metastore.googleapis.com host. Through Jentic you install once, import the Dataproc Metastore API from the API Directory, store the OAuth credential once, and your agent calls it.
- **Permission scoping:** The Metastore API puts the service and resource name in the URL path (/v1/{+parent}/... and /v1/{+name}), so a rule can pin your agent to one metastore service: it can list backups, federations, and metadata imports there and nothing else. You choose the operations it may call, so destructive ones like service deletion are not included unless you add them.
- **Credential handling:** Your Google OAuth credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'provision a metastore service' or 'import Hive metadata', and Jentic returns the matching Dataproc Metastore operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Cloud Dataproc API** — Managed Spark and Hadoop clusters that consume metadata from Dataproc Metastore
- **BigQuery API** — Queries Hive tables registered in a metastore via BigQuery federation
- **Cloud Storage API** — Holds the underlying data files referenced by Hive tables and the metadata import dumps

## FAQ

### What authentication does the Dataproc Metastore API use?

The API uses Google OAuth 2.0 with the https://www.googleapis.com/auth/cloud-platform scope. Through Jentic, the OAuth credentials are held in the encrypted vault and the agent receives only a scoped bearer token; the refresh token never enters the agent's context.

### Can I federate multiple metastore services through this API?

Yes, the federations endpoints under a project location let you combine multiple metastore services under a single federation endpoint, useful when teams want one logical catalog backed by per-region metastores.

### What are the rate limits for the Dataproc Metastore API?

Quotas are managed per project under the Dataproc Metastore service in the Cloud Console quotas page. Long-running operations have concurrency caps in addition to per-minute request limits, so bulk lifecycle scripts should respect the operation queue.

### How do I import existing Hive metadata through Jentic?

Search Jentic for import hive metadata to dataproc metastore, load the metadataImports create operation, and execute with the gs:// URI of the metadata dump and the target service name. Jentic returns the long-running operation handle for polling.

### Is the Dataproc Metastore API free?

The API itself is not separately billed; you pay for the metastore service tier (DEVELOPER or ENTERPRISE) per hour and any associated storage. See the Dataproc Metastore pricing page for current rates.

### Can I run Hive queries through this API?

No, this API manages metastore services themselves. To run queries, point a Dataproc cluster, BigQuery federation, or Spark job at the metastore endpoint and execute Hive or Spark SQL there.

### Can I limit what my agent is allowed to do with the Dataproc Metastore API?

Yes. Because your Jentic One instance is self-hosted, you write the rules that decide which operations and credentials your agent may use. Since this API puts the service and resource name in the URL path, you can pin the agent to a single metastore service so it only lists backups, federations, and metadata imports there and touches nothing else. You also choose the exact operations it may call, so destructive ones like service deletion stay out of reach unless you explicitly add them.