HDInsightJobManagementClient

★ Only Publicly Available OpenAPI DocumentDeveloper ToolsCi Cdoauth210 EndpointsREST

For Agents

Submit Hive, Pig, MapReduce, and Sqoop jobs to a running HDInsight cluster, list active jobs, fetch job state, and kill jobs.

Connect the HDInsightJobManagementClient to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the HDInsightJobManagementClient, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Install Jentic One Beta (Self-Hosted)

Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with HDInsightJobManagementClient API.

Submit a Hive job to an HDInsight cluster

Submit a MapReduce or MapReduce streaming job and track its job ID

Submit Pig and Sqoop jobs to the same cluster surface

List jobs running on the cluster, including those started after a given job ID

GET STARTED

Start building with HDInsightJobManagementClient API

Explore with Jentic

View OpenAPI Document

Jentic publishes the only available OpenAPI document for HDInsightJobManagementClient, keeping it validated and agent-ready.

Jentic publishes the only available OpenAPI specification for HDInsightJobManagementClient, keeping it validated and agent-ready. The HDInsight Job Management API submits and tracks Hadoop jobs against a running HDInsight cluster via the cluster's WebHCat (Templeton) endpoint. It supports Hive, Pig, Sqoop, MapReduce, and MapReduce streaming job submissions, plus job listing and kill operations. The API runs against the cluster DNS name (e.g. clustername.azurehdinsight.net) and is authenticated with cluster gateway credentials over OAuth2.

Use Cases

Patterns agents use HDInsightJobManagementClient API for, with concrete tasks.

★ Submit Hive jobs from a data pipeline

Submit Hive queries to an HDInsight cluster from an orchestration step via POST /templeton/v1/hive. The API returns the WebHCat job ID, which can then be polled via GET /templeton/v1/jobs/{jobId} until the job reaches a terminal state. Useful inside scheduled ETL pipelines that already produce input data on the cluster's storage.

POST /templeton/v1/hive with a Hive query that aggregates orders.csv and poll /templeton/v1/jobs/{jobId} until the status is SUCCEEDED.

Run streaming MapReduce jobs

Submit a MapReduce streaming job using Python or shell mappers via POST /templeton/v1/mapreduce/streaming, supplying input and output paths on cluster storage. WebHCat hands the job to YARN and returns a job ID for tracking. Useful when teams have legacy streaming jobs that pre-date Spark.

POST /templeton/v1/mapreduce/streaming with mapper.py and reducer.py and the agreed input and output paths on the cluster's primary storage container.

Active job monitoring and kill switch

List running jobs via GET /templeton/v1/jobs, page through with the LISTAFTERID variant, and kill any job that exceeds its expected duration via DELETE /templeton/v1/jobs/{jobId}. Useful for SREs operating shared HDInsight clusters where runaway jobs can starve other workloads.

List jobs on the cluster, find any in RUNNING state for more than 2 hours, and DELETE /templeton/v1/jobs/{jobId} for each.

AI agent job orchestration via Jentic

A data engineering agent uses Jentic to find the right Job_Submit operation for a query, load its schema, and execute it without holding raw cluster credentials. Job IDs returned by submission are then re-fed into Job_Get for polling. Jentic injects the bearer token at execution time.

Use Jentic to search for 'submit a hive job to hdinsight', load Job_SubmitHiveJob, and execute it for the agreed query against cluster 'analytics-prod'.

Key Endpoints

10 endpoints — jentic publishes the only available openapi specification for hdinsightjobmanagementclient, keeping it validated and agent-ready.

METHOD

PATH

DESCRIPTION

POST

/templeton/v1/hive

Submit a Hive job

POST

/templeton/v1/mapreduce/jar

Submit a MapReduce job from a JAR

POST

/templeton/v1/mapreduce/streaming

Submit a MapReduce streaming job

POST

/templeton/v1/pig

Submit a Pig job

POST

/templeton/v1/sqoop

Submit a Sqoop job

GET

/templeton/v1/jobs/{jobId}

Get a specific job's status

DELETE

/templeton/v1/jobs/{jobId}

Kill a running job

GET

/ws/v1/cluster/apps/{appId}/state

Get YARN application state

POST

/templeton/v1/hive

Submit a Hive job

POST

/templeton/v1/mapreduce/jar

Submit a MapReduce job from a JAR

POST

/templeton/v1/mapreduce/streaming

Submit a MapReduce streaming job

POST

/templeton/v1/pig

Submit a Pig job

POST

/templeton/v1/sqoop

Submit a Sqoop job

Why though Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

HDInsight cluster gateway credentials sit encrypted in the Jentic vault. Agents receive a scoped bearer token at execution time and never see the raw cluster password.

Intent-based discovery

Agents search Jentic for 'submit a hive job to hdinsight' and Jentic returns the matching Job_SubmitHiveJob operation with its input schema, so the agent can submit jobs without learning the WebHCat URL conventions.

Time to first call

Direct WebHCat integration: 1-2 days for cluster auth setup and job-state polling. Through Jentic: under 30 minutes — search, load schema, execute.

Related APIs

Alternatives and complements available in the Jentic catalogue.

Complementary

HDInsight Management

Manages HDInsight clusters and applications at the ARM level rather than submitting jobs to them.

Use HDInsightManagementClient when the agent needs to provision the cluster itself; use this Job API to submit work to a running cluster.

Alternative

Azure Databricks

Spark-first analytics service on Azure with its own jobs and notebooks API.

Use Databricks when the agent needs Spark jobs on a managed runtime rather than YARN-based Hive, Pig, or MapReduce.

Complementary

Azure Data Factory

Orchestrates pipelines that can submit HDInsight jobs as one step among many.

Pair with this when the agent needs scheduled, multi-step orchestration around HDInsight job submissions.

FAQs

Specific to using HDInsightJobManagementClient API through Jentic.

Why is there no official OpenAPI spec for HDInsightJobManagementClient?

Microsoft Azure does not publish a stand-alone OpenAPI specification for the HDInsight WebHCat job submission surface. Jentic generates and maintains this spec from the Templeton REST surface so that AI agents and developers can call HDInsightJobManagementClient via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.

What authentication does the HDInsight Job API use?

The API uses cluster gateway authentication over OAuth2, declared as the azure_auth security scheme. The host is the cluster DNS name (clustername.azurehdinsight.net), not management.azure.com. Through Jentic, the cluster gateway credentials are stored encrypted and exchanged for a bearer token at execution time.

Can I submit a Sqoop job through this API?

Yes. POST /templeton/v1/sqoop accepts a Sqoop command string and submits the job to the cluster. The response includes the WebHCat job ID, which can be polled at GET /templeton/v1/jobs/{jobId}.

What are the rate limits?

Limits are enforced by the cluster gateway and YARN scheduler rather than Azure ARM. Concurrent job count depends on cluster size and queue capacity — the API itself does not throttle, but the cluster will reject submissions when YARN is saturated.

How do I submit a Hive job through Jentic?

Run jentic.search('submit a hive job to an hdinsight cluster'), load the Job_SubmitHiveJob schema, and execute it with the cluster DNS name in the host parameter and your Hive query in the body. The response contains the WebHCat job ID for polling Job_Get.