Managed Streaming for Kafka

Name: Managed Streaming for Kafka API
Brand: Managed Streaming for Kafka
Availability: InStock

★ Only Publicly Available OpenAPI DocumentCloud InfrastructureComputehmac36 EndpointsREST

For Agents

Provision and manage Apache Kafka clusters, configurations, and SCRAM secrets on AWS without operating the underlying broker fleet.

Connect the Managed Streaming for Kafka to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Managed Streaming for Kafka, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Install Jentic One Beta (Self-Hosted)

Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with Managed Streaming for Kafka API.

Provision MSK clusters via /v1/clusters and /api/v2/clusters

List and describe broker nodes for a cluster

Apply and revise Kafka broker configurations

Attach SCRAM secrets stored in AWS Secrets Manager to enable SASL/SCRAM authentication

GET STARTED

Start building with Managed Streaming for Kafka API

Explore with Jentic

View OpenAPI Document

Jentic publishes the only available OpenAPI document for Managed Streaming for Kafka, keeping it validated and agent-ready.

Jentic publishes the only available OpenAPI specification for Amazon Managed Streaming for Apache Kafka (MSK), keeping it validated and agent-ready. The MSK control plane API has 36 endpoints for provisioning Kafka clusters, managing broker counts, applying broker configurations, attaching SCRAM secrets stored in Secrets Manager, and tracking cluster operations such as version upgrades and storage expansions. It does not handle topic-level produce or consume calls — those run over native Kafka protocol against the broker bootstrap endpoints. MSK is built for teams that want managed brokers, automated patching, and ZooKeeper or KRaft mode without operating the underlying instances themselves.

Use Cases

Patterns agents use Managed Streaming for Kafka API for, with concrete tasks.

★ Provisioning Kafka for Data Pipelines

Stand up a managed Kafka cluster for streaming ingestion pipelines that fan out events to data warehouses, search indexes, and downstream services. The MSK API provisions broker nodes across multiple Availability Zones, applies a server property configuration, and exposes bootstrap endpoints once the cluster is active. Operators avoid running ZooKeeper or KRaft directly while keeping wire-protocol compatibility with existing Kafka clients.

Create an MSK cluster named 'events-prod' with 3 broker nodes of kafka.m5.large in subnets subnet-a, subnet-b, subnet-c

SASL/SCRAM Authentication Setup

Wire SCRAM authentication into an MSK cluster by associating secrets stored in AWS Secrets Manager. The /v1/clusters/{clusterArn}/scram-secrets endpoint attaches a list of secret ARNs to the cluster, after which clients connect over SASL_SSL using the username and password recorded in each secret. This avoids embedding broker credentials in client code and works with audit logs in CloudTrail.

Attach secret arn:aws:secretsmanager:us-east-1:123:secret:msk-app-1 to cluster arn:aws:kafka:us-east-1:123:cluster/events-prod

Cluster Lifecycle Operations

Manage long-running operations such as broker count increases, storage expansions, and Kafka version upgrades through the cluster operations endpoints. The /v1/operations/{clusterOperationArn} endpoint reports operation state, target configuration, and any errors so platform teams can monitor changes without watching the AWS console. Tagging clusters and configurations supports per-team cost allocation and inventory views.

Get the status of cluster operation arn:aws:kafka:us-east-1:123:cluster-operation/abc and return the operation type and current state

AI Agent Streaming Infrastructure Operations

AI agents call the MSK API through Jentic to provision new clusters for ad-hoc workloads, fetch bootstrap brokers when wiring downstream services, and audit configuration drift across environments. Through Jentic, the agent searches by intent, loads the input schema, and executes with credentials held in the vault. This is faster than implementing SigV4 signing inside the agent runtime and prevents AWS keys from entering the LLM context.

Search Jentic for 'get kafka bootstrap brokers', load the GetBootstrapBrokers schema, and execute against cluster arn:aws:kafka:us-east-1:123:cluster/events-prod

Key Endpoints

36 endpoints — jentic publishes the only available openapi specification for amazon managed streaming for apache kafka (msk), keeping it validated and agent-ready.

METHOD

PATH

DESCRIPTION

POST

/v1/clusters

Create an MSK cluster

GET

/v1/clusters

List MSK clusters

GET

/v1/clusters/{clusterArn}

Describe a cluster

POST

/v1/configurations

Create a Kafka configuration

POST

/v1/clusters/{clusterArn}/scram-secrets

Attach SCRAM secrets to a cluster

GET

/v1/operations/{clusterOperationArn}

Describe a cluster operation

POST

/v1/clusters

Create an MSK cluster

GET

/v1/clusters

List MSK clusters

GET

/v1/clusters/{clusterArn}

Describe a cluster

POST

/v1/configurations

Create a Kafka configuration

POST

/v1/clusters/{clusterArn}/scram-secrets

Attach SCRAM secrets to a cluster

Why though Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

AWS access keys for the MSK control plane are stored encrypted in the Jentic vault. Jentic performs SigV4 signing at execution time, so raw AWS secrets never enter the agent's context. SCRAM secrets remain in Secrets Manager and are referenced by ARN.

Intent-based discovery

Agents search Jentic by intent such as 'provision kafka cluster' or 'get bootstrap brokers' and Jentic returns the matching MSK operation with its input schema, so the agent calls the right endpoint without parsing AWS docs.

Time to first call

Direct MSK integration: 1-3 days to set up SigV4, IAM, and operation polling. Through Jentic: under 1 hour — search, load the schema, execute.

Related APIs

Alternatives and complements available in the Jentic catalogue.

Alternative

Amazon Kinesis Analytics

Streaming SQL analytics over Kinesis or Kafka data

Choose Kinesis Analytics when you want managed streaming SQL processing on top of an existing stream rather than provisioning brokers

Alternative

Amazon MQ

Managed message broker service for ActiveMQ and RabbitMQ

Pick Amazon MQ when applications already speak ActiveMQ or RabbitMQ wire protocols rather than Kafka

Complementary

Amazon EventBridge

Serverless event bus for AWS services and SaaS

Combine with MSK when you need fan-out routing into AWS services without writing custom Kafka consumers

Complementary

AWS Lambda

Serverless function execution

Use Lambda event source mappings to consume MSK topics into managed function invocations

FAQs

Specific to using Managed Streaming for Kafka API through Jentic.

Why is there no official OpenAPI spec for Amazon MSK?

AWS does not publish an OpenAPI specification. Jentic generates and maintains this spec so that AI agents and developers can call Managed Streaming for Kafka via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.

What authentication does the Amazon MSK API use?

The MSK control plane API uses AWS Signature Version 4 (SigV4) HMAC request signing. The Kafka data plane (produce and consume) uses native Kafka protocol with IAM, mTLS, or SASL/SCRAM. Through Jentic, AWS credentials sit in the encrypted vault and SigV4 signing happens at execution time.

Can I publish messages to a Kafka topic with the MSK API?

No. The MSK control plane API does not expose produce or consume operations. Use a native Kafka client connected to the bootstrap brokers returned by GetBootstrapBrokers. The MSK API handles cluster, configuration, and credential management only.

How do I provision a new Kafka cluster through Jentic?

Search Jentic for 'create kafka cluster', load the schema for POST /v1/clusters or /api/v2/clusters, submit the broker count, instance type, and subnet list, then poll GET /v1/operations/{clusterOperationArn} until the cluster is ACTIVE. Jentic handles SigV4 signing throughout.

What are the rate limits for the Amazon MSK API?

Per-region quotas apply, including a default limit on clusters per account, brokers per cluster, and configurations per region. Control plane TPS limits also apply per operation. Check the MSK service quotas in the AWS console for the values in your account.

Can I attach SCRAM secrets to an existing MSK cluster?

Yes. POST /v1/clusters/{clusterArn}/scram-secrets accepts a list of Secrets Manager secret ARNs. Each secret must be encrypted with a customer-managed KMS key and contain a JSON object with username and password fields for SCRAM authentication.

Can I monitor a cluster upgrade with the MSK API?

Yes. Long-running operations such as version upgrades and storage expansions return a clusterOperationArn that can be polled with GET /v1/operations/{clusterOperationArn}. The response includes operation type, current state, and any error details.