canonical: https://jentic.com/apis/pipecat.ai/pipecat-ai

# Pipecat AI API

Pipecat AI provides real-time voice AI infrastructure for building voice agents, phone bots, and conversational AI applications with ultra-low latency. The API handles speech-to-text streaming, large language model integration, text-to-speech synthesis, function calling, and WebRTC transport for voice interactions. Built for developers creating AI phone agents, voice assistants, IVR systems, and real-time conversational interfaces with natural interruption handling and sub-second response times.

## For AI agents

Build real-time voice AI agents with streaming speech recognition, LLM integration, natural voice synthesis, and function calling. Supports 35 endpoints for voice pipelines, conversation management, and telephony integration.

## Scope

Does not provide phone number provisioning, call recording storage, or conversation analytics dashboards - use for real-time voice AI pipeline orchestration and conversation management only.

## Capabilities

- Stream audio input with real-time speech-to-text transcription and interruption detection
- Integrate with LLMs (OpenAI, Anthropic, Gemini) for natural conversation understanding and generation
- Synthesize natural-sounding speech with multiple voice providers (ElevenLabs, Azure, Google, Deepgram)
- Execute function calls during conversations for API lookups, database queries, and system actions
- Manage WebRTC and phone (Twilio, Telnyx) transport layers for voice delivery
- Handle conversation state, context persistence, and multi-turn dialogue flows
- Configure voice pipeline stages including VAD, STT, LLM, TTS, and transport components

## Use cases

### AI Phone Support Agent

Build AI-powered phone agents that handle customer support calls with natural conversation flow, function calling for order lookups, and intelligent routing to human agents. The API provides real-time STT for understanding customer requests, LLM integration for generating responses, TTS for natural speech output, and function calling to query CRM systems and databases. Agents handle interruptions naturally and maintain conversation context across topics.

Example prompt: Create a voice agent connected to Twilio phone number that greets callers, uses STT to understand their request, calls an order status API function when they mention an order number, and synthesizes a natural response with order details via ElevenLabs TTS

### Real-Time Voice Assistant for Web Apps

Embed voice AI capabilities in web applications using WebRTC transport for low-latency voice interactions. The API handles the full voice pipeline including audio streaming, speech recognition, LLM reasoning, response generation, and voice synthesis. Ideal for virtual sales assistants, voice-enabled dashboards, accessibility features, and hands-free application control.

Example prompt: Initialize a WebRTC voice pipeline with browser audio input, configure Deepgram STT, OpenAI GPT-4 for conversation, and Azure TTS for responses, then stream audio back to browser with <200ms latency

### Appointment Scheduling Voice Bot

Automate appointment booking over the phone with a voice agent that understands scheduling requests, checks calendar availability via function calls, confirms appointments, and sends confirmations. The agent handles complex multi-turn conversations including date/time disambiguation, service selection, and rescheduling requests. Integration with calendar APIs enables real-time availability checks.

Example prompt: Build a voice agent that answers incoming calls, asks for service type and preferred time, calls a calendar API to check availability, offers alternative slots if requested time is unavailable, confirms the appointment, and sends SMS confirmation

### Voice Pipeline Configuration and Testing

Configure and test voice AI pipelines with different combinations of STT providers (Deepgram, AssemblyAI, Whisper), LLMs (GPT-4, Claude, Gemini), TTS engines (ElevenLabs, Azure, Google), and transport layers (WebRTC, Twilio, Telnyx). The API allows swapping components, A/B testing voice quality and latency, and monitoring pipeline performance metrics including transcription accuracy and response times.

Example prompt: Create three voice pipeline configurations with different TTS providers (ElevenLabs, Azure, Google), run test conversations through each, measure latency and voice naturalness scores, and select the optimal pipeline for production deployment

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/v1/agents` | Create a new voice agent |
| POST | `/v1/agents/{agentId}/conversations` | Start a conversation session |
| POST | `/v1/pipelines` | Configure voice pipeline with STT/LLM/TTS stages |
| POST | `/v1/functions` | Register a function for calling during conversations |
| POST | `/v1/telephony/connect` | Connect agent to phone number |
| POST | `/v1/webrtc/session` | Initialize WebRTC voice session |
| GET | `/v1/conversations/{conversationId}/transcript` | Retrieve conversation transcript |

## Key resources

- **Voice Agents** — Create and manage voice agent configurations with pipeline components
- **Conversations** — Track conversation sessions, state, and transcripts
- **Pipelines** — Configure STT, LLM, TTS, and transport stages with provider selection
- **Functions** — Define function schemas for tool calling during conversations
- **Telephony** — Integrate with phone providers (Twilio, Telnyx) for inbound/outbound calls
- **WebRTC** — Manage WebRTC connections for browser-based voice interactions
- **Analytics** — Monitor conversation metrics, latency, and pipeline performance

## Why Jentic

- **Setup:** Wiring Pipecat AI by hand means managing its bearer token, sequencing agent, pipeline, and telephony calls, and handling retries yourself. Through Jentic you install once, import Pipecat AI from the API Directory, store the token once, and your agent calls it.
- **Permission scoping:** Pipecat AI puts the agent id in the URL path (`/v1/agents/{agentId}/conversations`), so a rule can pin your agent to one Pipecat agent and its conversations. You choose the operations it may call, so pipeline or function changes are not included unless you add them.
- **Credential handling:** Your Pipecat AI bearer token is stored once, encrypted, by your own Jentic One instance and injected as an Authorization header at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'create a voice agent for phone calls' or 'fetch a conversation transcript', and Jentic returns the matching Pipecat AI operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **ElevenLabs API** — Text-to-speech synthesis with natural voices used within Pipecat pipelines
- **Deepgram API** — Real-time speech-to-text transcription used within Pipecat pipelines
- **Twilio Voice API** — Telephony infrastructure for phone number provisioning and call routing

## FAQ

### What authentication does the Pipecat AI API use?

The API uses bearer token authentication with API keys generated in the Pipecat dashboard. Each request includes an Authorization: Bearer {token} header. Through Jentic, bearer tokens are stored encrypted and injected automatically - they never appear in agent prompts or conversation logs.

### What is the latency for voice responses?

Pipecat AI is optimized for sub-200ms response latency from speech input to audio output. Actual latency depends on the chosen STT provider, LLM speed, and TTS engine. Deepgram STT + GPT-4 + ElevenLabs TTS typically achieves 150-250ms end-to-end latency, which feels natural in conversation with proper interruption handling.

### Can I use my own LLM instead of OpenAI or Anthropic?

Yes. The pipeline configuration accepts custom LLM endpoints that conform to the OpenAI Chat Completions API format. You can integrate self-hosted models, Azure OpenAI, or any LLM service with compatible API. Function calling requires the LLM to support tool use capabilities.

### What phone providers are supported?

Pipecat AI integrates with Twilio, Telnyx, and Vonage for telephony transport. You configure your phone provider credentials and Pipecat handles the SIP/WebRTC connection, audio streaming, and DTMF handling. Inbound and outbound calling are supported with conversation recording and analytics.

### How does function calling work during conversations?

You register functions with JSON schemas via POST `/v1/functions.` During conversation, when the LLM determines a function call is needed (e.g., 'check order status'), Pipecat executes your webhook endpoint with parameters, receives the result, and the LLM incorporates it into the response. This enables real-time API integration during voice calls.

### What are the pricing and rate limits?

Pipecat AI offers a free tier with 100 conversation minutes per month. Paid plans start at $99/month for 1,000 minutes with additional per-minute charges beyond quota. Rate limits are 100 concurrent conversations for standard plans. Enterprise plans offer custom limits and SLA guarantees.

### Can I limit what my agent is allowed to do with the Pipecat AI API?

Yes. Because you run Jentic One yourself, your own rules decide which Pipecat AI operations and credentials the agent may use. Since the agent id sits in the URL path, such as `/v1/agents/{agentId}/conversations`, you can pin the agent to a single Pipecat agent and its conversations. You can also choose exactly which operations it may call, so pipeline configuration via `/v1/pipelines` or function registration via `/v1/functions` stays off limits unless you explicitly allow it.