canonical: https://jentic.com/apis/googleapis.com/dataform

# Google Dataform API

The Dataform API lets you develop, version-control, and operationalize SQL pipelines that run inside BigQuery. It manages repositories, workspaces, release configurations, compilation results, and workflow invocations so teams can ship reproducible SQL transformations through Git-backed workflows. The API exposes 43 endpoints covering Git operations (commit, push, pull, fetch history), workspace file editing, and scheduled workflow execution against BigQuery.

## For AI agents

Programmatically manage Dataform repositories, workspaces, and SQL workflow invocations in BigQuery. Lets agents author, compile, and trigger SQL pipelines without touching the Dataform UI.

## Scope

Does not run SQL queries directly, manage BigQuery tables, or schedule non-SQL tasks - use for Git-backed Dataform repository, workspace, and workflow-invocation management only.

## Capabilities

- Create and manage Dataform repositories backed by Git remotes
- Edit SQLX files inside workspaces and commit changes through the API
- Compile workflow definitions into validated BigQuery execution graphs
- Trigger workflow invocations on demand or on a release schedule
- Inspect Git status, ahead/behind counts, and file history for a workspace
- Manage release configurations that pin compilations to a Git ref

## Use cases

### Scheduled BigQuery Transformation Pipelines

Operationalize SQL transformations in BigQuery by defining release configurations that compile a Git ref and run on a schedule. Dataform handles dependency resolution between models, incremental table builds, and assertions, so analytics teams ship production SQL pipelines without standing up Airflow. A typical setup completes in a few hours by pointing the API at an existing GitHub repository.

Example prompt: Create a release configuration on repository 'analytics-prod' that compiles the main branch and triggers a workflow invocation every night at 02:00 UTC

### Git-Backed SQL Development Workflow

Edit SQLX models inside an isolated workspace, commit changes, push to a Git remote, and merge through pull requests before promoting to production. The API exposes fetchFileGitStatuses, commit, push, and pull operations so agents and CI bots can drive the full development loop without the Dataform console. Most repository-and-workspace setups take under an hour.

Example prompt: Create a workspace 'feature-revenue-model' from repository 'analytics-prod', write a new SQLX file, commit it with message 'add daily revenue model', and push to origin

### Pre-Production Compilation Validation

Before a SQL change reaches production, compile the workflow against BigQuery to catch reference errors, circular dependencies, and assertion failures. The API returns a CompilationResult with the resolved DAG and any compilation errors, so CI pipelines can fail fast on broken SQL. This adds compile-time safety to a workflow that would otherwise only catch errors at runtime in BigQuery.

Example prompt: Create a compilation result for the pull-request commit SHA on repository 'analytics-prod' and report any compilation errors back to the PR

### AI Agent SQL Pipeline Operator

An AI agent can drive Dataform end-to-end through Jentic: search for the right operation by intent, load the input schema, and execute repository, workspace, compilation, and workflow-invocation calls without browsing reference docs. Credentials stay in your Jentic One instance, so the agent receives only scoped access tokens. Setup through Jentic takes under an hour versus several days for a direct integration with Google's OAuth flow.

Example prompt: Use Jentic to search 'trigger a dataform workflow', load the createWorkflowInvocation schema, and execute it against repository 'analytics-prod' with the latest compilation result

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | /v1beta1/{+parent}/repositories | Create a Dataform repository |
| POST | /v1beta1/{+parent}/workspaces | Create a workspace inside a repository |
| POST | /v1beta1/{+name}:commit | Commit changes in a workspace |
| POST | /v1beta1/{+parent}/compilationResults | Compile a Git ref into an execution graph |
| POST | /v1beta1/{+parent}/workflowInvocations | Trigger a workflow invocation against BigQuery |
| GET | /v1beta1/{+name}:fetchFileGitStatuses | Fetch Git status of files in a workspace |

## Key resources

- **Repositories** — Create, list, update, and delete Dataform repositories with Git remote configuration
- **Workspaces** — Isolated editing environments for SQLX files with Git status and file CRUD operations
- **CompilationResults** — Compile a Git ref into a validated BigQuery execution graph
- **WorkflowInvocations** — Trigger and monitor execution of compiled workflows against BigQuery
- **ReleaseConfigs** — Define schedules and Git refs that drive recurring compilations

## Why Jentic

- **Setup:** Wiring the Dataform API by hand means setting up Google OAuth, refreshing short-lived scoped tokens rather than holding client secrets in code, and addressing repository and workspace resources on dataform.googleapis.com. Through Jentic you install once, import the Dataform API from the API Directory, store the Google credential once, and your agent calls it.
- **Permission scoping:** The API carries the parent and resource name in the URL path (/v1beta1/{parent}/repositories and /v1beta1/{name}:commit), so a rule can pin your agent to one repository or workspace. You choose the operations it may call, so it can create workspaces and trigger workflow invocations while anything you leave out stays unavailable.
- **Credential handling:** Your Google OAuth credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'trigger a Dataform workflow', and Jentic returns the matching Dataform operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **BigQuery API** — Dataform compiles and runs SQL against BigQuery; BigQuery is the execution engine and storage layer.
- **Dataflow API** — Dataflow runs Apache Beam pipelines for batch and streaming ETL; Dataform is SQL-only inside BigQuery.
- **Cloud Composer API** — Composer (managed Airflow) can orchestrate Dataform workflow invocations alongside other tasks.

## FAQ

### What authentication does the Dataform API use?

The Dataform API uses OAuth 2.0 with Google scopes (cloud-platform). Through Jentic the OAuth client and refresh tokens are stored in your Jentic One instance and the agent receives a short-lived scoped access token, so raw Google credentials never enter the agent context.

### Can I trigger BigQuery SQL pipelines with the Dataform API?

Yes. POST /v1beta1/{+parent}/workflowInvocations triggers a compiled workflow against BigQuery, and POST /v1beta1/{+parent}/compilationResults produces the compilation that the invocation runs from. Together they give you full programmatic execution of a Dataform release.

### What are the rate limits for the Dataform API?

Google enforces standard Cloud quotas on Dataform: per-project read/write quotas on repositories, workspaces, and workflow invocations, plus the underlying BigQuery quotas on jobs and slot usage. Quotas are visible in the Cloud Console under IAM and admin, quotas, filtered to dataform.googleapis.com.

### How do I commit a SQLX file to a Dataform workspace through Jentic?

Search Jentic for 'commit changes to a dataform workspace', load the schema for POST /v1beta1/{+name}:commit, and execute with the workspace name, commit message, and file actions array. Jentic returns the request body shape and handles the OAuth token exchange.

### Is the Dataform API free?

Dataform itself has no per-call charge, but the BigQuery jobs that workflow invocations run are billed under standard BigQuery on-demand or reservation pricing. Storage of repositories and workspaces is also free; you pay for the compute the SQL consumes.

### How do I check the Git status of a workspace?

Call GET /v1beta1/{+name}:fetchFileGitStatuses with the workspace resource name. It returns the per-file status (added, modified, deleted, conflicted) so a CI bot or agent can decide whether to commit, reset, or surface conflicts before pushing.

### Can I limit what my agent is allowed to do with the Dataform API?

Yes. Because Jentic One is self-hosted, you set the rules that decide which Dataform operations and credentials your agent may use. Since the API carries the parent and resource name in the URL path, such as /v1beta1/{parent}/repositories and /v1beta1/{name}:commit, you can pin the agent to a single repository or workspace. You also choose the exact operations it may call, so it can create workspaces and trigger workflow invocations while any operation you leave out stays unavailable.