canonical: https://jentic.com/apis/confident-ai.com/confident-ai

# Confident AI

The Confident AI API runs evaluation and observability for LLM applications. It pushes datasets and goldens, runs metric-based evals over traces, spans, and threads, versions prompts and metrics, and red-teams models with attack methods and vulnerability frameworks. Dashboards, annotation queues, and scheduled alerts turn the results into reviewable reports and human-in-the-loop labeling. Authentication uses an API key sent in the CONFIDENT_API_KEY header, and the service is reachable on US and EU hosts.

## For AI agents

Run metric-based LLM evaluations over datasets, traces, spans, and threads, version prompts and metrics, red-team models, and route results into dashboards and annotation queues through Confident AI.

## Scope

Does not host or serve the LLMs themselves, and does not handle application deployment. Use for LLM evaluation, observability, prompt versioning, and red-teaming only.

## Capabilities

- Run metric-based evaluations over LLM traces, spans, and threads
- Push datasets and goldens to evaluate against
- Version prompts and pull them by label, commit, or version
- Create and version custom metrics and metric collections
- Red-team a model with attack methods and vulnerability frameworks
- Route eval results into dashboards, annotation queues, and alerts

## Use cases

### Agent-Run LLM Evaluations

An AI agent connected through Jentic can grade an LLM application as part of a build or release step: it pushes a dataset of goldens, runs an evaluation, and reads the resulting metric scores to decide whether the change is safe to ship. The agent drives the eval end to end without a human opening the Confident AI dashboard.

Example prompt: Push a dataset of goldens, run an evaluation, and read the metric scores to decide whether to ship

### Trace-Level Scoring

Observability data is only useful when it is scored. Confident AI evaluates a captured trace, span, or thread against a metric, so an agent watching production traffic can attach a relevancy or faithfulness score to an individual trace and flag regressions as they happen.

Example prompt: Evaluate a captured trace against a relevancy metric and record the score

### Prompt Versioning

Teams need to manage prompts like code. Confident AI versions prompts with branches, commits, and labels, so an agent can push a new prompt version and pull it back by a label such as production, keeping the running application pinned to a reviewed prompt.

Example prompt: Push a new prompt version and pull it by the production label

### Automated Red-Teaming

Safety testing needs adversarial inputs. Confident AI red-teams a model with attack methods and vulnerability frameworks, so an agent can launch a red-team evaluation and review which vulnerabilities fired before a model reaches users.

Example prompt: Run a red-team evaluation using an attack method and review the vulnerability findings

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/v2/evaluate` | Run evals over test cases |
| POST | `/v2/datasets` | Push a dataset |
| POST | `/v2/datasets/{datasetId}/run` | Run a dataset evaluation |
| POST | `/v2/metrics` | Create a metric |
| POST | `/v2/prompts` | Push a prompt |
| POST | `/v2/evaluate/traces/{traceUuid}` | Evaluate a single trace |

## Key resources

- **Evaluations** — Run evals and evaluate individual traces, spans, and threads
- **Datasets and Goldens** — Push datasets, manage goldens, and run dataset evaluations
- **Metrics and Prompts** — Create and version metrics, metric collections, and prompts
- **Red-Teaming** — Attack methods, vulnerability frameworks, and classifiers for safety testing

## AI readiness

This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.

- **Score:** 58 / 100
- **Maturity:** Foundational
- **Dimensions:**
  - Foundational Compliance: 64 / 100
  - Developer Experience & Jentic Compatibility: 74 / 100
  - AI-Readiness & Agent Experience: 61 / 100
  - Agent Usability: 43 / 100
  - Security: 50 / 100
  - AI Discoverability: 79 / 100
- **View full report:** https://jentic.com/apis/confident-ai.com/confident-ai/scorecard
- **How the score is calculated:** https://docs.jentic.com/reference/api-readiness-framework/overview/
- **More about the dimensions:** https://docs.jentic.com/reference/api-readiness-framework/specification/#dimensional-model-overview

### Score it yourself

Every API in the directory is allowlisted, so you can re-score it with no key required.

- **Score your own API:** https://jentic.com/scorecard.md
- **Scoring CLI agent skill:** https://github.com/jentic/jentic-api-scorecard/blob/main/skills/jentic-api-scorecard/SKILL.md

```sh
npx @jentic/api-scorecard-cli score <openapi-url>
```

## Why Jentic

- **Setup:** Wiring Confident AI by hand means handling its CONFIDENT_API_KEY header, choosing the US or EU host to match your data residency, and matching eval payloads to each metric. Through Jentic you install once, import Confident AI from the API Directory, store the key once, and your agent calls it.
- **Permission scoping:** Confident AI puts resource ids in the URL path (`/v2/datasets/{datasetId}`, `/v2/prompts/{promptId}`), so a rule can bound your agent to the operations you pick, such as pushing datasets and running evaluations. You choose what it may call, so deletes of datasets, metrics, or prompts are not included unless you add them.
- **Credential handling:** Your Confident AI key is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'run an LLM evaluation' or 'evaluate a trace', and Jentic returns the matching Confident AI operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Langfuse** — Langfuse is an open-source LLM observability and evaluation platform.
- **Arize** — Arize offers LLM and ML observability with drift and performance monitoring.
- **Braintrust** — Braintrust is an eval and prompt-experimentation platform for AI products.
- **OpenAI** — OpenAI provides the models whose outputs Confident AI evaluates.

## FAQ

### What authentication does the Confident AI API use?

The Confident AI API authenticates with an API key sent in the CONFIDENT_API_KEY request header, created on the Confident AI platform per its OpenAPI spec. When you call it through Jentic, that key is stored once by your own Jentic One instance and injected at execution time, so it never reaches the agent's prompt or logs.

### Can I evaluate a single LLM trace with the Confident AI API?

Yes. Beyond running a full dataset evaluation, the evaluate operations score an individual trace, span, or thread against your metrics, so an agent can attach a relevancy or faithfulness score to one captured interaction rather than re-running the whole suite.

### Does the Confident AI API offer an EU region?

Yes. The spec lists two hosts: the default US host api.confident-ai.com and an EU host eu.api.confident-ai.com. Point your agent at the host that matches where your evaluation data must reside, using the credential issued for that region.

### What are the rate limits for the Confident AI API?

The OpenAPI spec does not define numeric rate limits for the Confident AI API. Check the Confident AI documentation at https://documentation.confident-ai.com for the current request limits that apply to your plan.

### Can I limit what my agent is allowed to do with the Confident AI API?

Yes. Write a rule that allows only the dataset-push and evaluate operations, so the agent can upload goldens and run evaluations but cannot delete datasets, metrics, or prompts, and every call it makes is logged.

### How do I run an evaluation with the Confident AI API through Jentic?

Search Jentic with an intent like 'run an LLM evaluation', and Jentic returns the evaluate operation at `/v2/evaluate` with its input schema so your agent passes the test cases and metrics. To run it on your own infrastructure, install Jentic One from its GitHub repo.

### Is there a Confident AI MCP server?

You don't need an MCP server to give your agent Confident AI. Jentic connects it directly from the API Directory: import the API, store your key once, and your agent calls the evaluate, dataset, metric, and prompt operations on demand.