canonical: https://jentic.com/apis/algolia.com/algolia-crawler-api

# Algolia Crawler API

Jentic publishes the only available OpenAPI specification for the Algolia Crawler API, keeping it validated and agent-ready. The Crawler API drives Algolia's hosted web crawler - programmatically creating crawlers, updating their configuration, starting and pausing crawls, testing URL extraction, listing runs, downloading run logs, and managing registered domains. It is distinct from the standard Algolia search and ingestion APIs and uses its own credential pair. Authentication is HTTP basic auth with the Crawler user id and Crawler API key.

## For AI agents

Drive Algolia's hosted web crawler programmatically: create crawlers, run and pause them, test URL extraction, and inspect crawl runs across 20 endpoints.

## Scope

Does not serve search queries, ingest direct records, or manage A/B tests - use for managing Algolia hosted web crawlers, runs, and configurations only.

## Capabilities

- Create a new Algolia crawler with POST /1/crawlers and update its configuration with PATCH /1/crawlers/{id}/config
- Trigger a full reindex with POST /1/crawlers/{id}/reindex or pause and resume crawlers
- Test crawl a single URL via POST /1/crawlers/{id}/test to validate extractors before a full run
- Crawl a batch of URLs on demand with POST /1/crawlers/{id}/urls/crawl
- List crawler runs and download per-run log files for diagnostics
- Manage registered domains and inspect crawler stats across the application

## Use cases

### Documentation site crawler bootstrapping

A docs platform team creates a new Algolia crawler for a freshly launched documentation portal by calling POST /1/crawlers with the start URLs and extraction config, validates extraction with POST /1/crawlers/{id}/test on a sample URL, and then kicks off the first full crawl with POST /1/crawlers/{id}/reindex. The same workflow can be re-run when site structure changes substantially.

Example prompt: POST /1/crawlers with the start URLs, POST /1/crawlers/{id}/test against a representative URL, then POST /1/crawlers/{id}/reindex

### On-demand recrawl after a publish event

A CMS publishes new articles and triggers an Algolia recrawl by calling POST /1/crawlers/{id}/urls/crawl with the affected URLs. This avoids waiting for the next scheduled reindex and keeps Algolia search results fresh without re-crawling the entire site.

Example prompt: POST /1/crawlers/{id}/urls/crawl with the list of newly published URLs and check the response task id with GET /1/crawlers/{id}/tasks/{taskID}

### Crawler health monitoring

An ops dashboard polls GET /1/crawlers/{id}/stats/urls and GET /1/crawlers/{id}/crawl_runs to display recent run durations, success rates, and URL counts. When a run fails, the dashboard fetches the log file via GET /1/crawlers/{id}/{logId}/download and surfaces the relevant error lines for triage.

Example prompt: GET /1/crawlers/{id}/crawl_runs, render the latest runs, and pull GET /1/crawlers/{id}/{logId}/download for any failing run id

### Agent-driven crawler configuration through Jentic

A Jentic agent receives an intent like 'add /blog to the Algolia crawler' and chains GET /1/crawlers/{id}/config, PATCH /1/crawlers/{id}/config with the updated URL patterns, and then POST /1/crawlers/{id}/reindex. The agent never sees the basic auth credentials - Jentic injects them at execution time.

Example prompt: Search Jentic for 'update algolia crawler config', execute PATCH /1/crawlers/{id}/config with the new patterns, then POST /1/crawlers/{id}/reindex

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | /1/crawlers | Create a crawler |
| POST | /1/crawlers/{id}/reindex | Start a fresh crawl |
| POST | /1/crawlers/{id}/test | Test crawl a single URL |
| POST | /1/crawlers/{id}/urls/crawl | Crawl specific URLs on demand |
| PATCH | /1/crawlers/{id}/config | Update crawler configuration |
| GET | /1/crawlers/{id}/crawl_runs | List crawler runs |
| GET | /1/crawlers/{id}/{logId}/download | Download a crawler run log |

## Key resources

- **Crawlers** — Create, list, retrieve, update, and delete crawlers; pause, resume, reindex, and test
- **Crawl runs** — List crawler runs, retrieve stats, download run logs, and delete run records
- **Configuration versions** — List and retrieve historical versions of a crawler configuration
- **Tasks** — Inspect and cancel long-running crawler tasks
- **Domains** — List domains registered with the crawler

## Why Jentic

- **Setup:** Wiring the Algolia Crawler API by hand means encoding its HTTP basic credentials, targeting the crawler.algolia.com host, and orchestrating crawler runs, reindexes, and config patches yourself. Through Jentic you install once, import the Algolia Crawler API from the API Directory, store the basic credential once, and your agent calls it.
- **Permission scoping:** The Crawler API puts the crawler id in the URL path (/1/crawlers/{id}/reindex, /1/crawlers/{id}/config, /1/crawlers/{id}/urls/crawl), so a rule can pin your agent to one crawler. You choose the operations it may call, so patching a crawler config is not included unless you add it.
- **Credential handling:** Your Algolia Crawler basic credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'reindex a crawler' or 'crawl a specific set of URLs', and Jentic returns the matching Algolia Crawler operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Algolia Search API** — The Search API serves queries against the indices that the Crawler API populates
- **Algolia Ingestion API** — Use the Ingestion API to push records directly into Algolia when a hosted crawler is not the right fit
- **Algolia Monitoring API** — Use the Monitoring API to watch overall Algolia infrastructure health alongside crawler-specific stats

## FAQ

### Why is there no official OpenAPI spec for the Algolia Crawler API?

Algolia does not publish a downstream-stable OpenAPI specification for direct agent use. Jentic generates and maintains this spec from Algolia's source bundles so AI agents and developers can call the Crawler API via structured tooling. It is validated against the live API and kept up to date. Get started with Jentic One, the self-hosted execution layer.

### What authentication does the Algolia Crawler API use?

The Crawler API uses HTTP basic auth with a Crawler user id and Crawler API key - distinct from the standard Algolia application id and API key. Jentic stores the credential pair encrypted and builds the Authorization header at execution time so the secret never enters the agent's prompt.

### Can I trigger an on-demand recrawl of specific URLs?

Yes. POST /1/crawlers/{id}/urls/crawl with a list of URLs. The endpoint returns a task id you can poll with GET /1/crawlers/{id}/tasks/{taskID} until the recrawl completes.

### What are the rate limits for the Algolia Crawler API?

The OpenAPI spec does not publish per-endpoint rate limits for the Crawler API. Algolia enforces them server-side; respect 4xx responses and back off when triggering rapid reindex or test calls in succession.

### How do I update an Algolia crawler's configuration through Jentic?

Search Jentic for 'update algolia crawler config', load the schema for PATCH /1/crawlers/{id}/config, and execute it with the updated URL patterns or extractor settings. Then POST /1/crawlers/{id}/reindex to apply the change to live results.

### Can I download the log for a specific crawler run?

Yes. GET /1/crawlers/{id}/{logId}/download returns the run log file. Pair this with GET /1/crawlers/{id}/crawl_runs to find the run id you need before downloading.

### Can I limit what my agent is allowed to do with the Algolia Crawler API?

Yes. Jentic One runs self-hosted, so your own rules decide which Crawler operations the agent may call and which credential it uses. Because the Crawler API carries the crawler id in the URL path for calls like POST /1/crawlers/{id}/reindex, PATCH /1/crawlers/{id}/config, and POST /1/crawlers/{id}/urls/crawl, you can pin the agent to a single crawler. You choose the exact operations it may run, so a config-patching call such as PATCH /1/crawlers/{id}/config is unavailable unless you add it.