Crawlbase Crawling API

Name: Crawlbase Crawling API API
Brand: Crawlbase Crawling API
Availability: InStock

★ Only Publicly Available OpenAPI DocumentData EnrichmentData TransformationapiKey2 EndpointsREST

For Agents

Fetch and scrape any public web page, including JavaScript-rendered content, and return parsed HTML or structured JSON. Use it to give an agent the contents of a URL when a direct API is not available.

Quickstart

Get started with Crawlbase Crawling API in minutes using your preferred integration method.

# Add to your MCP client config (Claude Desktop, Cursor, Windsurf)
{
  "jentic": {
    "url": "https://api.jentic.com/mcp",
    "auth": "oauth"
  }
}

# Then ask your agent:
"crawl a web page"

# → Jentic returns the GET /events tool with parameter schema, agent executes.

Capabilities

What an agent can do with Crawlbase Crawling API API.

Fetch the rendered HTML of any public URL through a managed proxy network

Crawl JavaScript-heavy pages by switching to the JS token without changing the request shape

Submit long target URLs via POST when query-string length limits would otherwise truncate them

Apply built-in scrapers to return structured JSON for supported page types instead of raw HTML

GET STARTED

Start building with Crawlbase Crawling API API

Explore with Jentic

View OpenAPI Document

Jentic publishes the only available OpenAPI document for Crawlbase Crawling API, keeping it validated and agent-ready.

Jentic publishes the only available OpenAPI specification for Crawlbase Crawling API, keeping it validated and agent-ready. Crawlbase provides a web crawling and scraping endpoint that fetches arbitrary URLs and returns either raw HTML or extracted structured data. The API supports both standard requests and a JavaScript-rendered mode for sites that hydrate content client-side, with output formatters that turn pages into machine-readable JSON. Authentication is done with a token query parameter and the same endpoint accepts GET or POST depending on how the target URL is supplied.

Use Cases

Patterns agents use Crawlbase Crawling API API for, with concrete tasks.

★ Scrape JavaScript-rendered pages

Fetch the fully rendered HTML of a single-page application or JavaScript-heavy site by calling the Crawlbase crawling endpoint with the JavaScript token. Crawlbase runs a real browser server-side so the agent receives the post-hydration markup it would see in Chrome, without operating its own headless browser fleet.

Send a GET request to https://api.crawlbase.com/?token={JS_TOKEN}&url=https://example.com/spa and return the rendered HTML body to the caller.

Bulk product page extraction

Iterate over a list of product URLs and use the crawling endpoint to fetch each one through Crawlbase's proxy network, optionally passing scraper or format parameters to receive structured JSON. Useful for price monitoring, catalog ingestion, or feeding a downstream LLM with normalised page content rather than raw HTML.

For each URL in a list of 50 product pages, POST to https://api.crawlbase.com/ with token and url fields and store the returned body keyed by source URL.

Bypass anti-bot blocks for research

Use the crawling endpoint as a managed fetch layer when the target site rate-limits or fingerprints the agent's own IP. Crawlbase rotates proxies and handles retries server-side, so the agent only sees a successful HTML or JSON response and does not need to manage proxy pools, CAPTCHAs, or browser sessions itself.

Fetch https://example.com/research-page through Crawlbase, retry once with the JavaScript token if the standard token returns a non-200 status, and return the final HTML.

AI agent web access through Jentic

Give a Jentic-orchestrated agent the ability to read arbitrary web pages without exposing the Crawlbase token in its prompt. The agent searches Jentic for a crawling operation, loads the schema for the Crawlbase endpoint, and executes a fetch with just the target URL while Jentic injects the token from the vault.

Through Jentic, run search('crawl a web page'), load the Crawlbase crawling operation, and execute it with url=https://news.example.com to retrieve the article HTML for summarisation.

Key Endpoints

2 endpoints — jentic publishes the only available openapi specification for crawlbase crawling api, keeping it validated and agent-ready.

METHOD

PATH

DESCRIPTION

GET

Crawl a URL passed in the query string

POST

Crawl a URL passed in the request body for long URLs

GET

Crawl a URL passed in the query string

POST

Crawl a URL passed in the request body for long URLs

Why though Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

Crawlbase tokens are stored encrypted in the Jentic vault. The agent never sees the token value and Jentic injects it as the token query parameter at execution time, keeping the credential out of prompts and traces.

Intent-based discovery

Agents search Jentic with intents like 'crawl a web page' or 'scrape a JavaScript site' and Jentic returns the Crawlbase / operation with its input schema, so the agent can call the right endpoint without reading the Crawlbase docs.

Time to first call

Direct integration: a few hours to handle token selection, retries, and proxy errors. Through Jentic: under 30 minutes to search, load schema, and execute a first crawl.

Related APIs

Alternatives and complements available in the Jentic catalogue.

Alternative

ScraperAPI

Competing managed proxy and scraping API with similar URL-fetch semantics.

Choose ScraperAPI when the user already has a ScraperAPI key or needs its specific geotargeting and session features; Crawlbase is preferable when JavaScript rendering toggles via a separate token is the cleaner fit.

Alternative

ZenRows

Anti-bot-focused scraping API that also fetches rendered HTML for a given URL.

Pick ZenRows when targets have aggressive bot detection that needs its premium proxy and JS rendering combo; Crawlbase is the simpler choice for plain HTML fetches.

Complementary

Apify

Actor-based scraping platform useful for orchestrating crawls beyond a single URL fetch.

Use Apify when the agent needs to run a multi-page crawler or pre-built actor, and call Crawlbase when it just needs the raw content of one URL at a time.

FAQs

Specific to using Crawlbase Crawling API API through Jentic.

Why is there no official OpenAPI spec for Crawlbase Crawling API?

Crawlbase does not publish an OpenAPI specification. Jentic generates and maintains this spec so that AI agents and developers can call Crawlbase Crawling API via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.

What authentication does the Crawlbase Crawling API use?

The API uses an API key passed as the token query parameter on every request. There are two token variants, a normal token for standard crawling and a JavaScript token for browser-rendered pages. Through Jentic the token lives in the encrypted vault and is injected at execution time, so it never appears in the agent's prompt or logs.

Can I crawl JavaScript-rendered pages with the Crawlbase Crawling API?

Yes. Send the same GET / or POST / request but use the JavaScript token instead of the normal token, and Crawlbase will run the page in a real browser server-side and return the post-hydration HTML.

When should I use POST instead of GET on the crawling endpoint?

Use POST / when the target URL or its query string is long enough to risk truncation in a GET request, or when you want to send the URL in the body rather than expose it in request logs. The behaviour is otherwise the same as GET /.

How do I crawl a URL with Crawlbase through Jentic?

Install Jentic with pip install jentic, then run an async search for 'crawl a web page', load the returned Crawlbase operation, and execute it with the target url field. Jentic resolves to GET / or POST / on https://api.crawlbase.com and attaches the stored token automatically.

What output formats does the Crawlbase Crawling API return?

By default the endpoint returns the raw HTML of the fetched page. Crawlbase also supports built-in scrapers and format parameters that return structured JSON for certain page types, configured per request through query parameters on the same / endpoint.