Crawlbase Crawling API
50
AI ReadinessFoundational (D+)50/100
Agent-ready OpenAPI document · curated by JenticData EnrichmentData TransformationapiKey2 EndpointsREST
Know of an official OpenAPI document? Contribute it →
For Agents
Fetch and scrape any public web page, including JavaScript-rendered content, and return parsed HTML or structured JSON. Use it to give an agent the contents of a URL when a direct API is not available.
Use for: I need to scrape the HTML of a product page that blocks bots, Fetch the rendered content of a single-page application URL, Retrieve a JSON-structured version of a search results page, Get the HTML of a page using a residential proxy in a specific country
Not supported: Does not handle data parsing for arbitrary domains, proxy account management, or stored crawl history - use for single-URL fetch and rendering only.
Jentic publishes the only available OpenAPI specification for Crawlbase Crawling API, keeping it validated and agent-ready. Crawlbase provides a web crawling and scraping endpoint that fetches arbitrary URLs and returns either raw HTML or extracted structured data. The API supports both standard requests and a JavaScript-rendered mode for sites that hydrate content client-side, with output formatters that turn pages into machine-readable JSON. Authentication is done with a token query parameter and the same endpoint accepts GET or POST depending on how the target URL is supplied.
Install Jentic One Beta
Connect the Crawlbase Crawling API to your agent
Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Crawlbase Crawling API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.
Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.
1
Step 1: Jentic One Host machine
# On the machine that will host your Jentic One instance:
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fcrawlbase.com%2Fcrawlbase" | sh2
Step 2: Agent machine
# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL "https://jentic.com/install.sh?src=apis&api=%2Fapis%2Fcrawlbase.com%2Fcrawlbase" | sh
jentic register # connects your agent to your Jentic One instanceJentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.
Capabilities
What an agent can do with Crawlbase Crawling API.
Fetch the rendered HTML of any public URL through a managed proxy network
Crawl JavaScript-heavy pages by switching to the JS token without changing the request shape
Submit long target URLs via POST when query-string length limits would otherwise truncate them
Apply built-in scrapers to return structured JSON for supported page types instead of raw HTML
Set the response output format and request country routing through query parameters on each crawl
Use Cases
Patterns agents use Crawlbase Crawling API for, with concrete tasks.
★ Scrape JavaScript-rendered pages
Fetch the fully rendered HTML of a single-page application or JavaScript-heavy site by calling the Crawlbase crawling endpoint with the JavaScript token. Crawlbase runs a real browser server-side so the agent receives the post-hydration markup it would see in Chrome, without operating its own headless browser fleet.
Send a GET request to https://api.crawlbase.com/?token={JS_TOKEN}&url=https://example.com/spa and return the rendered HTML body to the caller.
Bulk product page extraction
Iterate over a list of product URLs and use the crawling endpoint to fetch each one through Crawlbase's proxy network, optionally passing scraper or format parameters to receive structured JSON. Useful for price monitoring, catalog ingestion, or feeding a downstream LLM with normalised page content rather than raw HTML.
For each URL in a list of 50 product pages, POST to https://api.crawlbase.com/ with token and url fields and store the returned body keyed by source URL.
Bypass anti-bot blocks for research
Use the crawling endpoint as a managed fetch layer when the target site rate-limits or fingerprints the agent's own IP. Crawlbase rotates proxies and handles retries server-side, so the agent only sees a successful HTML or JSON response and does not need to manage proxy pools, CAPTCHAs, or browser sessions itself.
Fetch https://example.com/research-page through Crawlbase, retry once with the JavaScript token if the standard token returns a non-200 status, and return the final HTML.
AI agent web access through Jentic
Give a Jentic-orchestrated agent the ability to read arbitrary web pages without exposing the Crawlbase token in its prompt. The agent searches Jentic for a crawling operation, loads the schema for the Crawlbase endpoint, and executes a fetch with just the target URL while Jentic injects the token from the vault.
Through Jentic, run search('crawl a web page'), load the Crawlbase crawling operation, and execute it with url=https://news.example.com to retrieve the article HTML for summarisation.
Key Endpoints
2 endpoints — jentic publishes the only available openapi specification for crawlbase crawling api, keeping it validated and agent-ready.
METHOD
PATH
DESCRIPTION
GET
/
Crawl a URL passed in the query string
POST
/
Crawl a URL passed in the request body for long URLs
GET
/
Crawl a URL passed in the query string
POST
/
Crawl a URL passed in the request body for long URLs
Jentic AI Readiness Score
This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.
C
Crawlbase Crawling API
- Foundational (D+)50/100
90
Foundational Compliance63
Developer Experience & Jentic Compatibility56
AI-Readiness & Agent Experience94
Agent Usability15
Security100
AI DiscoverabilityPowered by JenticScoring Framework 1.0.0 | Scoring Engine 0.4.0
Show dimension breakdown
90
Foundational Compliance
Base layer of spec validity and structural soundness.
Grade: ASignals: 4
99%
Lint Results
Aggregated quality score from linter diagnostics, weighted by severity.
100%
Resolution Completeness
Percentage of `$ref` references that resolve successfully.
100%
Specification Validity
Checks whether the API description parses successfully and conforms to its declared specification (e.g., OpenAPI).
60%
Structural Integrity
Structural correctness score based on schema issues using logarithmic dampening.
63
Developer Experience & Jentic Compatibility
Clarity, completeness, and ingestion readiness for developers and tooling.
Grade: B-Signals: 4
0%
Example Density
How richly the API is illustrated with examples.
100%
Example Validity
Percentage of examples that conform to their schemas.
50%
Response Coverage
Percentage of operations with complete response definitions (success, client error, server error).
100%
Tooling Readiness
Health of API ingestion, bundling, and resolution within Jentic pipelines.
56
AI-Readiness & Agent Experience
Semantic breadth, depth, and agent comprehension for AI systems.
Grade: CSignals: 4
59%
Description Coverage
Coverage of descriptions across API elements.
0%
Error Standardization
Coverage of RFC 9457 Problem Details for error responses.
100%
OperationId Quality
Coverage, uniqueness, and casing consistency of operationIds for AI inference.
67%
Summary Coverage
Coverage of summaries across operations/tags/info.
94
Agent Usability
Functional utility, complexity comfort, and AI orchestration readiness.
Grade: A+Signals: 1
94%
Complexity Comfort
Agent comfort level based on API operational and structural complexity.
15
Security
Trust, risk posture, and security compliance.
Grade: FSignals: 1
15%
Authentication Strength
Average quality of security schemes based on authentication method strength (weakest link for OAuth2).
100
AI Discoverability
Findability, semantic richness, and reasoning readiness.
Grade: A+Signals: 1
100%
Descriptive Richness
Clarity and depth of descriptions across API elements.
Score it yourself
Every API in the directory is allowlisted, so you can re-score it with no key required.
npx @jentic/api-scorecard-cli score <openapi-url>Why Jentic?
What agents get from Jentic-routed access to this vendor.
Setup
Wiring Crawlbase by hand means passing the token as a query parameter and choosing the standard or JavaScript token depending on whether a page needs rendering. Through Jentic you install once, import Crawlbase from the API Directory, store the token once, and your agent calls it.
Permission scoping
Crawlbase exposes a single crawl endpoint and takes the target URL in the request rather than as a URL resource id, so scope the agent to the operations it needs, such as fetching or rendering a page. The agent runs only the crawl operations you allow.
Credential isolation
Your Crawlbase token is stored once, encrypted, by your own Jentic One instance and injected as the token query parameter at execution time. It never enters the agent's prompt, logs, or context.
Intent-based discovery
Agents search Jentic by intent such as 'crawl a web page' or 'scrape a JavaScript site', and Jentic returns the matching Crawlbase operation with its input schema so the agent calls the right endpoint without browsing the reference docs.
Related APIs
Alternatives and complements available in the Jentic catalogue.
FAQs
Specific to using Crawlbase Crawling API through Jentic.
Why is there no official OpenAPI spec for Crawlbase Crawling API?
Crawlbase does not publish an OpenAPI specification. Jentic generates and maintains this spec so that AI agents and developers can call Crawlbase Crawling API via structured tooling. It is validated against the live API and kept up to date. Get started with Jentic One, the self-hosted execution layer.
What authentication does the Crawlbase Crawling API use?
The API uses an API key passed as the token query parameter on every request. There are two token variants, a normal token for standard crawling and a JavaScript token for browser-rendered pages. Through Jentic the token lives in the encrypted vault and is injected at execution time, so it never appears in the agent's prompt or logs.
Can I crawl JavaScript-rendered pages with the Crawlbase Crawling API?
Yes. Send the same GET / or POST / request but use the JavaScript token instead of the normal token, and Crawlbase will run the page in a real browser server-side and return the post-hydration HTML.
When should I use POST instead of GET on the crawling endpoint?
Use POST / when the target URL or its query string is long enough to risk truncation in a GET request, or when you want to send the URL in the body rather than expose it in request logs. The behaviour is otherwise the same as GET /.
How do I crawl a URL with Crawlbase through Jentic?
Install Jentic with pip install jentic, then run an async search for 'crawl a web page', load the returned Crawlbase operation, and execute it with the target url field. Jentic resolves to GET / or POST / on https://api.crawlbase.com and attaches the stored token automatically.
What output formats does the Crawlbase Crawling API return?
By default the endpoint returns the raw HTML of the fetched page. Crawlbase also supports built-in scrapers and format parameters that return structured JSON for certain page types, configured per request through query parameters on the same / endpoint.
Can I limit what my agent is allowed to do with the Crawlbase Crawling API?
Yes. Because Jentic One is self-hosted, you set the rules for which operations and credentials your agent may use, and Crawlbase exposes a single crawl endpoint (GET / or POST /) that takes the target URL in the request rather than as a resource id. You can scope the agent to only the crawl actions it needs, such as fetching a page or rendering a JavaScript-heavy page with the browser token, and it runs only the operations you allow. Your Crawlbase token is stored encrypted by your own instance and injected at execution time, so it never appears in the agent's prompt or logs.
GET STARTED