canonical: https://jentic.com/apis/anakin.io/anakin

# Anakin API

The Anakin API is a web scraping and data extraction platform exposing 12 endpoints for URL scraping, batch scraping, web scraping jobs, web crawling, sitemap-style URL discovery (map), AI-powered web search, and longer-running agentic research jobs. Each long-running operation is a POST that returns a job id, with a paired GET `/job/{id}` or resource-specific status endpoint for polling. Authentication is via the X-API-Key header against api.anakin.io/v1. Typical consumers are LLM RAG pipelines, sales prospecting tools, and research agents that need clean structured content from arbitrary websites.

## For AI agents

Lets agents scrape one URL, batch up to 10 URLs, run web-scraper or crawl jobs, discover URLs on a domain, run AI-powered web search, or kick off agentic research jobs - all via X-API-Key authenticated POST/GET pairs.

## Scope

Does not handle structured-data APIs, browser automation flows, or proxy management - use for content scraping, crawling, mapping, and AI-powered web search only.

## Capabilities

- Scrape a single URL (or up to 10 in a batch) and retrieve the cleaned content as a job result
- Run a web-scraper job that handles JavaScript-heavy pages and follows pagination rules
- Crawl a website starting from a root URL and collect content across multiple pages
- Discover the URL map of a website without fully crawling it
- Run AI-powered web search with synthesised results suitable for grounding an LLM
- Kick off an agentic-search job that performs multi-step research and returns a structured answer
- Poll job status and pull final results once a long-running scrape, crawl, or research job completes

## Use cases

### RAG Ingestion of a Documentation Site

LLM platform teams can stand up a documentation RAG pipeline by calling POST /map to discover the docs URL set, POST /crawl to crawl the resulting site, then poll GET `/crawl/{id}` for the cleaned content. The two-step (kick off, then poll) pattern is consistent across long-running endpoints, so the same client code wraps every job type. Most teams stand up an end-to-end pipeline in a day.

Example prompt: Call POST /map with url='https://docs.example.com', then POST /crawl with the discovered root URL, then poll GET `/crawl/{id}` until status='completed' and return the cleaned page content

### Sales Prospecting Page Extraction

Sales-prospecting tools can call POST /scrape on a company homepage or POST /scrape-batch on a list of 10 candidate URLs to pull pricing, headcount, and tech-stack signals. The job-based shape lets the tool process inbound leads asynchronously rather than blocking on every scrape. The X-API-Key model keeps the integration trivial.

Example prompt: Call POST /scrape-batch with 10 candidate company URLs, retrieve the resulting job id, then poll GET `/job/{id}` until completion and return each page's cleaned content

### Agentic Research for Investment Notes

Investment and research teams can hand off a research question to POST /agentic-search, get back a job id, and poll GET `/agentic-search/{id}` for a multi-step researched answer. The agentic endpoint handles the search-and-synthesise loop server-side so the calling team does not need to wire up its own tool-calling scaffold. Pair with POST /search for lighter-weight grounded answers.

Example prompt: Call POST /agentic-search with a question like 'recent funding rounds for vector database startups', poll GET `/agentic-search/{id}`, and return the final structured answer

### AI Agent Web Research Loop

Research and copywriting agents can use Anakin through Jentic to scrape, crawl, and search without managing an X-API-Key directly. The agent searches Jentic for a scrape, crawl, search, or agentic-search intent, loads the operation schema, and Jentic injects the key at execution time. Useful when an agent already chains LLM tool calls and just needs a clean web-data tool in the mix.

Example prompt: Search Jentic for 'scrape a single URL', execute the operation backed by POST /scrape with url='https://example.com/pricing', poll GET `/job/{id}` until completion, and return the cleaned page content

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/scrape` | Scrape a single URL |
| POST | `/scrape-batch` | Batch scrape up to 10 URLs |
| GET | `/job/{id}` | Get scrape job status and results |
| POST | `/web-scraper` | Start a JavaScript-aware web scraper job |
| POST | `/map` | Discover URLs on a website |
| POST | `/crawl` | Crawl multiple pages starting from a root URL |
| POST | `/search` | AI-powered web search |
| POST | `/agentic-search` | Start an agentic research job |

## Key resources

- **URL Scraper** — Scrape one or many URLs and retrieve the cleaned content via a job
- **Web Scraper** — Run JavaScript-aware scraping jobs with pagination support
- **Map** — Discover the URL set on a website without fetching every page
- **Crawl** — Crawl multiple pages starting from a root URL
- **Search** — Run AI-powered web search with synthesised results
- **Agentic Search** — Multi-step research jobs that produce a structured answer

## Why Jentic

- **Setup:** Wiring the Anakin API by hand means managing the X-API-Key header and polling the `/job/{id}` endpoint for async scrape and crawl results yourself. Through Jentic you install once, import Anakin from the API Directory, store the API key once, and your agent calls it.
- **Permission scoping:** Anakin takes the target URL in the request body rather than as a resource id in the path, so scope the agent to the operations it needs, such as scrape and search. You choose the operations it may call, so batch scraping or full-site crawl are not included unless you add them.
- **Credential handling:** Your Anakin X-API-Key is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'scrape a URL' or 'crawl a website', and Jentic returns the matching Anakin operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Apify** — Actor-based scraping platform with a wider library of pre-built scrapers
- **Browserless** — Hosted headless Chrome with a programmatic API for scraping and automation
- **ScraperAPI** — Proxy-and-scrape API that handles rotation and bot detection at the request layer
- **Outscraper** — Specialised scrapers for Google Maps, reviews, and SERPs to pair with Anakin's general scraping

## FAQ

### What authentication does the Anakin API use?

Anakin uses API key authentication via the X-API-Key header on every request. There is no OAuth flow. Through Jentic, the X-API-Key is stored encrypted in the vault and injected at execution time, so the agent never sees the raw key.

### Can I scrape multiple URLs in a single call with the Anakin API?

Yes. POST /scrape-batch accepts up to 10 URLs in one request and returns a job id. Poll GET `/job/{id}` until the status flips to completed and the response carries the cleaned content for each URL. For higher volume, queue multiple batches rather than expanding a single batch.

### What are the rate limits for the Anakin API?

The spec does not document rate limits explicitly. Anakin applies them per plan tier - paid plans support more concurrent jobs and higher daily caps than the free trial. Treat HTTP 429 as a signal to back off and reduce concurrent jobs rather than retry tightly.

### How do I run an agentic research job through Jentic?

Search Jentic for 'run agentic research'. Jentic returns the operation backed by POST /agentic-search, you load the schema (the question is the main input), execute, then poll GET `/agentic-search/{id}` for the result. Jentic injects the X-API-Key from the vault. Run it through Jentic One, the self-hosted execution layer.

### Are the long-running endpoints synchronous?

No. Scrape, web-scraper, map, crawl, and agentic-search endpoints are all kick-off-and-poll. Each POST returns a job id, and a paired GET endpoint (`/job/{id}`, `/web-scraper/{id}`, `/map/{id}`, `/crawl/{id}`, `/agentic-search/{id}`) returns status and final results. Plan for asynchronous handling in any client wrapping this API.

### Can the Anakin API handle JavaScript-heavy pages?

Yes. POST /web-scraper is the JavaScript-aware path designed for SPAs and rendered pages, while POST /scrape is the lighter HTML-fetch path. Pick /web-scraper when the page needs rendering and /scrape when a clean HTML response is enough.

### Can I limit what my agent is allowed to do with the Anakin API?

Yes. Because you run Jentic One self-hosted, your own rules decide which Anakin operations and credentials the agent may use, so you can allow just the ones it needs, such as POST /scrape for single-URL fetches and POST /search for grounded web search. Anakin takes the target URL in the request body rather than as a path id, so heavier operations like POST /scrape-batch or POST /crawl stay unavailable to the agent unless you explicitly add them. The X-API-Key is held by your instance and injected at execution time, so the agent can only reach the endpoints you have permitted.