Algolia Crawler API

Name: Algolia Crawler API API
Brand: Algolia Crawler API
Availability: InStock

★ Only Publicly Available OpenAPI DocumentDeveloper ToolsMonitoring Observabilitybasic20 EndpointsREST

For Agents

Drive Algolia's hosted web crawler programmatically: create crawlers, run and pause them, test URL extraction, and inspect crawl runs across 20 endpoints.

Quickstart

Get started with Algolia Crawler API in minutes using your preferred integration method.

# Add to your MCP client config (Claude Desktop, Cursor, Windsurf)
{
  "jentic": {
    "url": "https://api.jentic.com/mcp",
    "auth": "oauth"
  }
}

# Then ask your agent:
"trigger an Algolia crawler reindex"

# → Jentic returns the GET /events tool with parameter schema, agent executes.

Capabilities

What an agent can do with Algolia Crawler API API.

Create a new Algolia crawler with POST /1/crawlers and update its configuration with PATCH /1/crawlers/{id}/config

Trigger a full reindex with POST /1/crawlers/{id}/reindex or pause and resume crawlers

Test crawl a single URL via POST /1/crawlers/{id}/test to validate extractors before a full run

Crawl a batch of URLs on demand with POST /1/crawlers/{id}/urls/crawl

GET STARTED

Start building with Algolia Crawler API API

Explore with Jentic

View OpenAPI Document

Use for: I need to create a new Algolia web crawler for a documentation site, Trigger a fresh reindex on an existing Algolia crawler, Pause an Algolia crawler before a major site migration, Test extraction on a single URL before kicking off a full crawl

Not supported: Does not serve search queries, ingest direct records, or manage A/B tests — use for managing Algolia hosted web crawlers, runs, and configurations only.

Jentic publishes the only available OpenAPI document for Algolia Crawler API, keeping it validated and agent-ready.

Jentic publishes the only available OpenAPI specification for the Algolia Crawler API, keeping it validated and agent-ready. The Crawler API drives Algolia's hosted web crawler — programmatically creating crawlers, updating their configuration, starting and pausing crawls, testing URL extraction, listing runs, downloading run logs, and managing registered domains. It is distinct from the standard Algolia search and ingestion APIs and uses its own credential pair. Authentication is HTTP basic auth with the Crawler user id and Crawler API key.

Use Cases

Patterns agents use Algolia Crawler API API for, with concrete tasks.

★ Documentation site crawler bootstrapping

A docs platform team creates a new Algolia crawler for a freshly launched documentation portal by calling POST /1/crawlers with the start URLs and extraction config, validates extraction with POST /1/crawlers/{id}/test on a sample URL, and then kicks off the first full crawl with POST /1/crawlers/{id}/reindex. The same workflow can be re-run when site structure changes substantially.

POST /1/crawlers with the start URLs, POST /1/crawlers/{id}/test against a representative URL, then POST /1/crawlers/{id}/reindex

On-demand recrawl after a publish event

A CMS publishes new articles and triggers an Algolia recrawl by calling POST /1/crawlers/{id}/urls/crawl with the affected URLs. This avoids waiting for the next scheduled reindex and keeps Algolia search results fresh without re-crawling the entire site.

POST /1/crawlers/{id}/urls/crawl with the list of newly published URLs and check the response task id with GET /1/crawlers/{id}/tasks/{taskID}

Crawler health monitoring

An ops dashboard polls GET /1/crawlers/{id}/stats/urls and GET /1/crawlers/{id}/crawl_runs to display recent run durations, success rates, and URL counts. When a run fails, the dashboard fetches the log file via GET /1/crawlers/{id}/{logId}/download and surfaces the relevant error lines for triage.

GET /1/crawlers/{id}/crawl_runs, render the latest runs, and pull GET /1/crawlers/{id}/{logId}/download for any failing run id

Agent-driven crawler configuration through Jentic

A Jentic agent receives an intent like 'add /blog to the Algolia crawler' and chains GET /1/crawlers/{id}/config, PATCH /1/crawlers/{id}/config with the updated URL patterns, and then POST /1/crawlers/{id}/reindex. The agent never sees the basic auth credentials — Jentic injects them at execution time.

Search Jentic for 'update algolia crawler config', execute PATCH /1/crawlers/{id}/config with the new patterns, then POST /1/crawlers/{id}/reindex

Key Endpoints

20 endpoints — jentic publishes the only available openapi specification for the algolia crawler api, keeping it validated and agent-ready.

METHOD

PATH

DESCRIPTION

POST

/1/crawlers

Create a crawler

POST

/1/crawlers/{id}/reindex

Start a fresh crawl

POST

/1/crawlers/{id}/test

Test crawl a single URL

POST

/1/crawlers/{id}/urls/crawl

Crawl specific URLs on demand

PATCH

/1/crawlers/{id}/config

Update crawler configuration

GET

/1/crawlers/{id}/crawl_runs

List crawler runs

GET

/1/crawlers/{id}/{logId}/download

Download a crawler run log

POST

/1/crawlers

Create a crawler

POST

/1/crawlers/{id}/reindex

Start a fresh crawl

POST

/1/crawlers/{id}/test

Test crawl a single URL

POST

/1/crawlers/{id}/urls/crawl

Crawl specific URLs on demand

PATCH

/1/crawlers/{id}/config

Update crawler configuration

Why though Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

The Crawler user id and Crawler API key are stored encrypted in the Jentic credential vault and combined into the Authorization basic auth header at execution time. The agent never sees the raw credentials.

Intent-based discovery

Agents search Jentic by intent (e.g. 'reindex an algolia crawler', 'crawl specific URLs'), and Jentic returns the matching Crawler API operation with its input schema for direct execution.

Time to first call

Direct Algolia Crawler integration: 2-3 days to wire basic auth, configuration management, and run polling. Through Jentic: under an hour with search, load, and execute.

Related APIs

Alternatives and complements available in the Jentic catalogue.

Complementary

Algolia Search API

The Search API serves queries against the indices that the Crawler API populates

Use the Search API alongside the Crawler API once crawled content is indexed and ready to serve queries

Alternative

Algolia Ingestion API

Use the Ingestion API to push records directly into Algolia when a hosted crawler is not the right fit

Choose the Ingestion API when content lives in a database or external source; choose the Crawler API for crawling public web content

Complementary

Algolia Monitoring API

Use the Monitoring API to watch overall Algolia infrastructure health alongside crawler-specific stats

Use the Monitoring API alongside the Crawler API for application-level uptime visibility

FAQs

Specific to using Algolia Crawler API API through Jentic.

Why is there no official OpenAPI spec for the Algolia Crawler API?

Algolia does not publish a downstream-stable OpenAPI specification for direct agent use. Jentic generates and maintains this spec from Algolia's source bundles so AI agents and developers can call the Crawler API via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.

What authentication does the Algolia Crawler API use?

The Crawler API uses HTTP basic auth with a Crawler user id and Crawler API key — distinct from the standard Algolia application id and api key. Jentic stores the credential pair encrypted and builds the Authorization header at execution time so the secret never enters the agent's prompt.

Can I trigger an on-demand recrawl of specific URLs?

Yes. POST /1/crawlers/{id}/urls/crawl with a list of URLs. The endpoint returns a task id you can poll with GET /1/crawlers/{id}/tasks/{taskID} until the recrawl completes.

What are the rate limits for the Algolia Crawler API?

The OpenAPI spec does not publish per-endpoint rate limits for the Crawler API. Algolia enforces them server-side; respect 4xx responses and back off when triggering rapid reindex or test calls in succession.

How do I update an Algolia crawler's configuration through Jentic?

Search Jentic for 'update algolia crawler config', load the schema for PATCH /1/crawlers/{id}/config, and execute it with the updated URL patterns or extractor settings. Then POST /1/crawlers/{id}/reindex to apply the change to live results.

Can I download the log for a specific crawler run?

Yes. GET /1/crawlers/{id}/{logId}/download returns the run log file. Pair this with GET /1/crawlers/{id}/crawl_runs to find the run id you need before downloading.