canonical: https://jentic.com/apis/crawlbrulee.com/crawlbrulee

# CrawlBrulee API

The CrawlBrulee API extracts structured content from web pages, scraping a single URL synchronously or submitting it as an asynchronous job you collect later by job id. It builds a site link map from sitemap discovery and homepage links, renders JavaScript-heavy pages before extraction, and routes requests through a proxy pinned to a location. Supporting operations report billing-cycle credit usage and the identity of the token making the call. Responses are JSON and authentication is a bearer token.

## For AI agents

Scrape a URL synchronously or as an async job, map a site's links, and collect results by job id, returning extracted page content as structured data. Authenticates with a bearer token and supports JavaScript rendering and proxy routing.

## Scope

Does not handle data warehousing, scheduled recurring crawls, or browser automation. Use for on-demand web scraping and site mapping only.

## Capabilities

- Scrape a single URL synchronously and return its extracted content
- Submit an asynchronous scrape job and collect the result by job id
- Build a site link map from sitemap discovery and homepage links
- Render JavaScript-heavy pages before extracting their content
- Route a scrape through a proxy pinned to a chosen location
- Report credit usage for the current billing cycle

## Use cases

### Agent-Driven Web Extraction

An AI agent that needs the content of a web page can call CrawlBrulee through Jentic to scrape a URL on demand and read back clean, structured content instead of raw HTML. The agent passes the URL to the synchronous scrape operation, optionally asking it to render JavaScript first, without the developer wiring the bearer auth or proxy options by hand. Jentic returns the matching operation and its input schema so the agent scrapes inline.

Example prompt: Scrape https://example.com/pricing with JavaScript rendering enabled and return the extracted pricing text

### Site Mapping and Asynchronous Crawling

A pipeline that needs many pages from a site can first build a link map and then scrape each URL as an asynchronous job to avoid blocking on slow pages. CrawlBrulee builds the map from sitemap discovery plus homepage links, accepts async scrape jobs with a webhook, and holds each job's status and result for retrieval by job id.

Example prompt: Build a link map for a site, then submit an async scrape job for each URL and poll for the results

### Rendering Dynamic Pages

Content behind client-side rendering is invisible to a plain HTTP fetch, so a scraper has to run the page's JavaScript first. CrawlBrulee renders the page before extraction and can route the request through a proxy pinned to a location, so an agent retrieves the same content a browser in that region would see.

Example prompt: Scrape a single-page app URL with rendering on and the request routed through a proxy in a chosen location

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/api/scrape` | Scrape a URL synchronously |
| POST | `/api/scrape/async` | Submit an asynchronous scrape job |
| GET | `/api/scrape/status/{job_id}` | Check an async scrape job's status |
| GET | `/api/scrape/result/{job_id}` | Retrieve a completed scrape job's result |
| POST | `/api/map` | Build a site link map |
| GET | `/api/usage` | Report billing-cycle usage and credits |
| GET | `/api/whoami` | Return the token's organization identity |

## Key resources

- **Scrape** — Scrape a single URL synchronously or submit it as an asynchronous job.
- **Jobs** — Check the status of an async scrape job and retrieve its result by job id.
- **Map** — Build or return a cached site link map from sitemap discovery and homepage links.
- **Account** — Report billing-cycle credit usage and the identity of the token making the call.

## AI readiness

This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.

- **Score:** 41 / 100
- **Maturity:** Foundational
- **Dimensions:**
  - Foundational Compliance: 74 / 100
  - Developer Experience & Jentic Compatibility: 69 / 100
  - AI-Readiness & Agent Experience: 17 / 100
  - Agent Usability: 94 / 100
  - Security: 60 / 100
  - AI Discoverability: 68 / 100
- **View full report:** https://jentic.com/apis/crawlbrulee.com/crawlbrulee/scorecard
- **How the score is calculated:** https://docs.jentic.com/reference/api-readiness-framework/overview/
- **More about the dimensions:** https://docs.jentic.com/reference/api-readiness-framework/specification/#dimensional-model-overview

### Score it yourself

Every API in the directory is allowlisted, so you can re-score it with no key required.

- **Score your own API:** https://jentic.com/scorecard.md
- **Scoring CLI agent skill:** https://github.com/jentic/jentic-api-scorecard/blob/main/skills/jentic-api-scorecard/SKILL.md

```sh
npx @jentic/api-scorecard-cli score <openapi-url>
```

## Why Jentic

- **Setup:** Wiring CrawlBrulee by hand means handling its bearer token auth, polling the status and result operations for async scrapes, and setting proxy and location options per request. Through Jentic you install once, import CrawlBrulee from the API Directory, store the token once, and your agent calls it.
- **Permission scoping:** CrawlBrulee carries the target URL in the request body, so rules bound which operations your agent may call, not which site. Limit it to the scrape and map operations it needs, and leave async job submission or the account identity lookup out unless you add them.
- **Credential handling:** Your CrawlBrulee token is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'scrape a URL' or 'map a website', and Jentic returns the matching CrawlBrulee operation with its input schema so the agent calls the right endpoint directly.

## Related APIs

- **Firecrawl API** — Also scrapes, crawls, and maps sites into markdown or structured JSON.
- **Crawlbase Crawling API** — Fetches and scrapes pages with JavaScript rendering and returns parsed data.
- **ScraperAPI** — Scrapes a page through rotating proxies and returns the raw response.
- **Outscraper API** — Extracts structured business and review data from directories and maps.

## FAQ

### What authentication does the CrawlBrulee API use?

The CrawlBrulee API authenticates with a bearer token on every operation, per its OpenAPI spec. Through Jentic the token is stored once, encrypted, by your own Jentic One instance and injected at call time, so it never reaches the agent's prompt or logs.

### Is there a CrawlBrulee MCP server?

You don't need an MCP server to give your agent the CrawlBrulee API. Jentic connects it directly from the API Directory: import it, store your token once, and your agent can scrape pages and map sites straight away.

### Can I limit what my agent is allowed to do with the CrawlBrulee API?

Yes. CrawlBrulee carries the target URL in the request body, so a rule bounds which operations your agent may call rather than which site. Allow just the scrape and map operations it needs, and leave the account identity and usage lookups out unless you add them, and every call it makes is logged.

### Can I scrape JavaScript-heavy pages with the CrawlBrulee API?

Yes. The scrape operations can render a page's JavaScript before extracting its content, so client-side-rendered pages return the same content a browser would see rather than an empty shell. You can also route the request through a proxy pinned to a location.

### What are the rate limits for the CrawlBrulee API?

The spec exposes a usage operation that reports your total, used, and available credits for the billing cycle, but it does not publish a fixed requests-per-second ceiling. Check the CrawlBrulee documentation for the current limits on your plan.

### How do I scrape a page with the CrawlBrulee API through Jentic?

Search Jentic for an intent like 'scrape a URL and return its content' and it returns the CrawlBrulee scrape operation with its input schema, so your agent sends the URL and reads back the extracted content. To run it on your own infrastructure, install Jentic One from its GitHub repo.