canonical: https://jentic.com/apis/spider.cloud/spider-cloud

# Spider API

Web crawling, scraping, and data extraction API with advanced features like screenshot capture, anti-bot bypass, and content transformation. The API exposes 11 endpoints secured with bearer authentication.

## For AI agents

Programmatically crawl website, scrape page. Covers 11 operations with bearer authentication.

## Scope

Does not handle payments, communications, or crm - use for developer tools only.

## Capabilities

- Crawl website
- Scrape page
- Bypass anti-bot protection
- Search web
- Collect links
- Capture screenshot
- Transform content

## Use cases

### Developer Tools Operations

Use the Spider API to perform developer tools operations programmatically. The API provides 11 endpoints covering core functionality including crawl website, scrape page, bypass anti-bot protection.

Example prompt: Call POST /crawl to crawl website

### Automated Crawl Management

Automate crawl operations by combining multiple Spider API endpoints. Agents can scrape page and then bypass anti-bot protection in a single workflow.

Example prompt: Call POST /scrape to scrape page, then verify the result

### AI Agent Integration via Jentic

AI agents discover and call Spider API endpoints through Jentic without managing credentials directly. An agent searches for the required operation by intent, receives the matching endpoint schema, and executes the call with Jentic-managed authentication. This eliminates the need to read API documentation or handle bearer tokens manually.

Example prompt: Search Jentic for 'crawl website', load the operation schema, and execute with Jentic-managed credentials

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/crawl` | Crawl website |
| POST | `/scrape` | Scrape page |
| POST | `/unblocker` | Bypass anti-bot protection |
| POST | `/search` | Search web |
| POST | `/links` | Collect links |
| POST | `/screenshot` | Capture screenshot |
| POST | `/transform` | Transform content |
| POST | `/fetch/{domain}/{path}` | Fetch with AI config |

## Key resources

- **Crawl** — Website crawling operations
- **Scrape** — Page scraping operations
- **Data** — Data management and retrieval
- **Transform** — Content transformation
- **Screenshot** — Screenshot capture

## AI readiness

This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.

- **Score:** 68 / 100
- **Maturity:** AI-Aware
- **Dimensions:**
  - Foundational Compliance: 100 / 100
  - Developer Experience & Jentic Compatibility: 63 / 100
  - AI-Readiness & Agent Experience: 47 / 100
  - Agent Usability: 94 / 100
  - Security: 60 / 100
  - AI Discoverability: 100 / 100
- **View full report:** https://jentic.com/apis/spider.cloud/spider-cloud/scorecard
- **How the score is calculated:** https://docs.jentic.com/reference/api-readiness-framework/overview/
- **More about the dimensions:** https://docs.jentic.com/reference/api-readiness-framework/specification/#dimensional-model-overview

### Score it yourself

Every API in the directory is allowlisted, so you can re-score it with no key required.

- **Score your own API:** https://jentic.com/scorecard.md
- **Scoring CLI agent skill:** https://github.com/jentic/jentic-api-scorecard/blob/main/skills/jentic-api-scorecard/SKILL.md

```sh
npx @jentic/api-scorecard-cli score <openapi-url>
```

## Why Jentic

- **Setup:** Wiring the Spider API by hand means carrying its bearer token on every request and wiring the crawl, scrape, search, and screenshot operations yourself. Through Jentic you install once, import the Spider API from the API Directory, store the token once, and your agent calls it.
- **Permission scoping:** Spider passes its targets in the request body, so limit the agent to the operations it needs, such as scraping a page or running a crawl. You choose the operations it may call, so it only runs the crawl, scrape, or search actions you allow and nothing more.
- **Credential handling:** Your Spider bearer token is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'crawl a website' or 'take a screenshot of a page', and Jentic returns the matching Spider operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Github** — Alternative developer tools API
- **Gitlab** — Alternative developer tools API

## FAQ

### What authentication does the Spider API use?

The Spider API uses a Bearer token in the Authorization header. Through Jentic, these credentials are stored encrypted in your Jentic One instance and injected at execution time, so raw secrets never enter the agent context.

### Can I crawl website with the Spider API?

Yes. Use the POST /crawl endpoint. The API returns structured JSON responses that agents can parse and act on directly.

### What are the rate limits for the Spider API?

Rate limits are not specified in the OpenAPI spec. Check the vendor documentation for current limits. Through Jentic, rate limiting is handled automatically with retry logic built into the execution layer.

### How do I crawl website through Jentic?

Install the Jentic SDK with pip install jentic, authenticate through Jentic One, the self-hosted execution layer, then search for 'crawl website'. Jentic returns the matching Spider API operation with its input schema. Load the schema and execute the call - credentials are injected automatically.

### How many endpoints does the Spider API have?

The Spider API exposes 11 endpoints covering crawl, scrape, data operations.

### Can I limit what my agent is allowed to do with the Spider API?

Yes. Jentic One is self-hosted by you, so your own rules decide which Spider operations and which credentials the agent may use. Because Spider passes its targets in the request body, you can allow only the operations the agent needs, such as scraping a page with POST /scrape or running a crawl with POST /crawl, while withholding actions like search or screenshot capture. The agent can call only the endpoints you permit and nothing else.