Install Jentic One Beta

Connect the Confident AI to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Confident AI, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.

Step 1: Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fconfident-ai.com%2Fconfident-ai" | sh

Step 2: Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fconfident-ai.com%2Fconfident-ai" | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with Confident AI API.

Run metric-based evaluations over LLM traces, spans, and threads

Push datasets and goldens to evaluate against

Version prompts and pull them by label, commit, or version

Create and version custom metrics and metric collections

Red-team a model with attack methods and vulnerability frameworks

GET STARTED

Start building with Confident AI API

Explore with Jentic One

View OpenAPI Document

Use Cases

Patterns agents use Confident AI API for, with concrete tasks.

★ Agent-Run LLM Evaluations

An AI agent connected through Jentic can grade an LLM application as part of a build or release step: it pushes a dataset of goldens, runs an evaluation, and reads the resulting metric scores to decide whether the change is safe to ship. The agent drives the eval end to end without a human opening the Confident AI dashboard.

Push a dataset of goldens, run an evaluation, and read the metric scores to decide whether to ship

Trace-Level Scoring

Observability data is only useful when it is scored. Confident AI evaluates a captured trace, span, or thread against a metric, so an agent watching production traffic can attach a relevancy or faithfulness score to an individual trace and flag regressions as they happen.

Evaluate a captured trace against a relevancy metric and record the score

Prompt Versioning

Teams need to manage prompts like code. Confident AI versions prompts with branches, commits, and labels, so an agent can push a new prompt version and pull it back by a label such as production, keeping the running application pinned to a reviewed prompt.

Push a new prompt version and pull it by the production label

Automated Red-Teaming

Safety testing needs adversarial inputs. Confident AI red-teams a model with attack methods and vulnerability frameworks, so an agent can launch a red-team evaluation and review which vulnerabilities fired before a model reaches users.

Run a red-team evaluation using an attack method and review the vulnerability findings

Key Endpoints

277 endpoints — the confident ai api runs evaluation and observability for llm applications.

METHOD

PATH

DESCRIPTION

POST

/v2/evaluate

Run evals over test cases

POST

/v2/datasets

Push a dataset

POST

/v2/datasets/{datasetId}/run

Run a dataset evaluation

POST

/v2/metrics

Create a metric

POST

/v2/prompts

Push a prompt

POST

/v2/evaluate/traces/{traceUuid}

Evaluate a single trace

POST

/v2/evaluate

Run evals over test cases

POST

/v2/datasets

Push a dataset

POST

/v2/datasets/{datasetId}/run

Run a dataset evaluation

POST

/v2/metrics

Create a metric

POST

/v2/prompts

Push a prompt

POST

/v2/evaluate/traces/{traceUuid}

Evaluate a single trace

Why Jentic?

What agents get from Jentic-routed access to this vendor.

Setup

Wiring Confident AI by hand means handling its CONFIDENT_API_KEY header, choosing the US or EU host to match your data residency, and matching eval payloads to each metric. Through Jentic you install once, import Confident AI from the API Directory, store the key once, and your agent calls it.

Permission scoping

Confident AI puts resource ids in the URL path (/v2/datasets/{datasetId}, /v2/prompts/{promptId}), so a rule can bound your agent to the operations you pick, such as pushing datasets and running evaluations. You choose what it may call, so deletes of datasets, metrics, or prompts are not included unless you add them.

Credential isolation

Your Confident AI key is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.

Intent-based discovery

Agents search Jentic by intent such as 'run an LLM evaluation' or 'evaluate a trace', and Jentic returns the matching Confident AI operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

FAQs

Specific to using Confident AI API through Jentic.

What authentication does the Confident AI API use?

The Confident AI API authenticates with an API key sent in the CONFIDENT_API_KEY request header, created on the Confident AI platform per its OpenAPI spec. When you call it through Jentic, that key is stored once by your own Jentic One instance and injected at execution time, so it never reaches the agent's prompt or logs.

Can I evaluate a single LLM trace with the Confident AI API?

Yes. Beyond running a full dataset evaluation, the evaluate operations score an individual trace, span, or thread against your metrics, so an agent can attach a relevancy or faithfulness score to one captured interaction rather than re-running the whole suite.

Does the Confident AI API offer an EU region?

Yes. The spec lists two hosts: the default US host api.confident-ai.com and an EU host eu.api.confident-ai.com. Point your agent at the host that matches where your evaluation data must reside, using the credential issued for that region.

What are the rate limits for the Confident AI API?

The OpenAPI spec does not define numeric rate limits for the Confident AI API. Check the Confident AI documentation at https://documentation.confident-ai.com for the current request limits that apply to your plan.

Can I limit what my agent is allowed to do with the Confident AI API?

Yes. Write a rule that allows only the dataset-push and evaluate operations, so the agent can upload goldens and run evaluations but cannot delete datasets, metrics, or prompts, and every call it makes is logged.

How do I run an evaluation with the Confident AI API through Jentic?

Search Jentic with an intent like 'run an LLM evaluation', and Jentic returns the evaluate operation at /v2/evaluate with its input schema so your agent passes the test cases and metrics. To run it on your own infrastructure, install Jentic One from its GitHub repo.

Is there a Confident AI MCP server?

You don't need an MCP server to give your agent Confident AI. Jentic connects it directly from the API Directory: import the API, store your key once, and your agent calls the evaluate, dataset, metric, and prompt operations on demand.

Confident AI

AI ReadinessFoundational (C+)58/100

See full scorecard

Official vendor OpenAPI document · agent-readyDeveloper ToolsTesting QaapiKey277 EndpointsREST

For Agents

Run metric-based LLM evaluations over datasets, traces, spans, and threads, version prompts and metrics, red-team models, and route results into dashboards and annotation queues through Confident AI.

Use for: Run an evaluation on a set of LLM outputs, Push a dataset of goldens to Confident AI, Evaluate a single trace against my metrics, Create a custom evaluation metric

Not supported: Does not host or serve the LLMs themselves, and does not handle application deployment. Use for LLM evaluation, observability, prompt versioning, and red-teaming only.

The Confident AI API runs evaluation and observability for LLM applications. It pushes datasets and goldens, runs metric-based evals over traces, spans, and threads, versions prompts and metrics, and red-teams models with attack methods and vulnerability frameworks. Dashboards, annotation queues, and scheduled alerts turn the results into reviewable reports and human-in-the-loop labeling. Authentication uses an API key sent in the CONFIDENT_API_KEY header, and the service is reachable on US and EU hosts.

Jentic One on Github View OpenAPI Document

Show dimension breakdown

Foundational Compliance

Base layer of spec validity and structural soundness.

Grade: BSignals: 4

94%

Lint Results

Aggregated quality score from linter diagnostics, weighted by severity.

100%

Resolution Completeness

Percentage of `$ref` references that resolve successfully.

Specification Validity

Checks whether the API description parses successfully and conforms to its declared specification (e.g., OpenAPI).

60%

Structural Integrity

Structural correctness score based on schema issues using logarithmic dampening.

Developer Experience & Jentic Compatibility

Clarity, completeness, and ingestion readiness for developers and tooling.

Grade: A-Signals: 4

46%

Example Density

How richly the API is illustrated with examples.

100%

Example Validity

Percentage of examples that conform to their schemas.

50%

Response Coverage

Percentage of operations with complete response definitions (success, client error, server error).

100%

Tooling Readiness

Health of API ingestion, bundling, and resolution within Jentic pipelines.

AI-Readiness & Agent Experience

Semantic breadth, depth, and agent comprehension for AI systems.

Grade: B-Signals: 4

80%

Description Coverage

Coverage of descriptions across API elements.

Error Standardization

Coverage of RFC 9457 Problem Details for error responses.

100%

OperationId Quality

Coverage, uniqueness, and casing consistency of operationIds for AI inference.

64%

Summary Coverage

Coverage of summaries across operations/tags/info.

Agent Usability

Functional utility, complexity comfort, and AI orchestration readiness.

Grade: D-Signals: 1

43%

Complexity Comfort

Agent comfort level based on API operational and structural complexity.

Security

Trust, risk posture, and security compliance.

Grade: C-Signals: 1

50%

Authentication Strength

Average quality of security schemes based on authentication method strength (weakest link for OAuth2).

AI Discoverability

Findability, semantic richness, and reasoning readiness.

Grade: A-Signals: 1

79%

Descriptive Richness

Clarity and depth of descriptions across API elements.

Connect the Confident AI to your agent

Capabilities

Start building with Confident AI API

Use Cases

Key Endpoints

Why Jentic?

Related APIs

FAQs

Confident AI

Jentic AI Readiness Score

Confident AI

Foundational Compliance

Lint Results

Resolution Completeness

Specification Validity

Structural Integrity

Developer Experience & Jentic Compatibility

Example Density

Example Validity

Response Coverage

Tooling Readiness

AI-Readiness & Agent Experience

Description Coverage

Error Standardization

OperationId Quality

Summary Coverage

Agent Usability

Complexity Comfort

Security

Authentication Strength

AI Discoverability

Descriptive Richness