canonical: https://jentic.com/apis/dataflowkit.com/dataflowkit

# Dataflow Kit Web Scraper

Dataflow Kit is a managed web scraping service that renders JavaScript-driven pages using a hosted Headless Chrome fleet behind rotating proxies. The five-endpoint REST API covers raw page fetching, structured parsing via point-and-click recipes, search engine results page (SERP) scraping, and converting any URL to a PDF or full-page screenshot. Authentication is a single api_key query parameter and responses come back as JSON, ready to feed into a database or AI pipeline.

## For AI agents

Render JavaScript pages, extract structured data from URLs, scrape SERPs, and convert pages to PDF or PNG without managing browsers or proxies.

## Scope

Does not handle proxy purchase, residential IP rentals, or CAPTCHA-solving as standalone services - use for managed page rendering, parsing, SERP scraping, and URL-to-media conversion only.

## Capabilities

- Fetch a fully rendered HTML page with rotating proxies via POST /fetch
- Extract structured records from a URL using a Dataflow Kit visual recipe via POST /parse
- Scrape Google, Bing, and DuckDuckGo result pages via POST /serp
- Convert any URL to a PDF document via POST `/convert/url/pdf`
- Capture a full-page screenshot of a URL via POST `/convert/url/screenshot`
- Drive scraping jobs from agents without running a Chrome cluster or proxy pool

## Use cases

### JavaScript Page Rendering

Many modern sites render content client-side, so a plain HTTP GET returns an empty shell. Dataflow Kit's POST /fetch endpoint runs the page through a hosted Headless Chrome instance behind a rotating proxy, then returns the fully rendered HTML. Teams use this to scrape single-page applications, product pages, and infinite-scroll feeds without operating a Selenium farm. Setup is a single API key call and integrations typically land in a few hours.

Example prompt: POST /fetch with target URL https://example-shop.com/products/123 and return the rendered HTML body

### SERP Monitoring

SEO teams and competitive intelligence pipelines call POST /serp to pull structured Google, Bing, and DuckDuckGo result pages for tracked keywords. The endpoint handles proxy rotation, CAPTCHA evasion, and country/locale targeting so the consuming application only needs to send a query and receive a parsed list of organic and paid results.

Example prompt: POST /serp with query 'best running shoes 2026', engine 'google', country 'us' and return the top 10 organic results

### Page-to-PDF Archival

Compliance, legal, and research teams use POST `/convert/url/pdf` to capture a snapshot of a web page as a PDF at a specific moment in time. The endpoint renders the page, applies print styles, and returns a binary PDF - useful for archiving terms-of-service changes, news articles, or competitor pricing pages with timestamped fidelity.

Example prompt: POST `/convert/url/pdf` with URL https://example.com/terms and save the returned PDF to s3://archive/terms-2026-06-10.pdf

### AI Agent Web Data Extraction

AI agents that need fresh web data call Dataflow Kit through Jentic to fetch rendered HTML, scrape SERPs, or extract structured records using a saved Dataflow Kit recipe. Jentic isolates the api_key in its vault, so agents pass the target URL and recipe ID without ever touching the credential.

Example prompt: Call POST /parse with a saved recipe ID and a list of 50 product URLs to return a structured table of name, price, and stock status

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/fetch` | Render a URL via Headless Chrome and return HTML |
| POST | `/parse` | Extract structured data from a URL using a saved recipe |
| POST | `/serp` | Scrape search engine result pages |
| POST | `/convert/url/pdf` | Convert a rendered URL to a PDF |
| POST | `/convert/url/screenshot` | Capture a full-page screenshot of a URL |

## Key resources

- **fetch** — Render a URL with Headless Chrome and return raw HTML
- **parse** — Apply a saved point-and-click recipe to extract structured data from a URL
- **serp** — Scrape Google, Bing, and DuckDuckGo search engine result pages
- **url-to-pdf** — Convert a rendered URL to a PDF document
- **url-to-screenshot** — Capture a full-page screenshot of a rendered URL

## Why Jentic

- **Setup:** Wiring Dataflow Kit by hand means learning its api_key query auth against api.dataflowkit.com/v1 and building the rendering, parsing, and retry plumbing for scrape jobs yourself. Through Jentic you install once, import the Dataflow Kit Web Scraper from the API Directory, store the key once, and your agent calls it.
- **Permission scoping:** Dataflow Kit passes its target URLs and parse rules in the request body rather than a resource id in the path, so limit the agent to the operations it needs, such as fetching a page or scraping SERP results. Every operation the agent can run is one you added to that allowed set.
- **Credential handling:** Your Dataflow Kit api_key is stored once, encrypted, by your own Jentic One instance and appended as the api_key query parameter at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'render a JavaScript page' or 'scrape Google results', and Jentic returns the matching Dataflow Kit operation with its input schema so the agent calls the right endpoint without browsing the docs.

## Related APIs

- **ScraperAPI** — ScraperAPI offers similar managed proxy + Headless Chrome rendering with a slightly different pricing model and structured-output product.
- **ZenRows** — ZenRows is a competing proxy + headless rendering API focused on anti-bot bypass.
- **Apify** — Apify provides a marketplace of pre-built scrapers and orchestration; Dataflow Kit handles raw rendering and parsing primitives.

## FAQ

### What authentication does the Dataflow Kit Web Scraper API use?

Dataflow Kit uses an API key passed as the api_key query parameter on every request. Through Jentic, that key is stored encrypted in your Jentic One instance and injected at call time, so agents never handle the raw secret.

### Can I scrape JavaScript-rendered pages with the Dataflow Kit API?

Yes. POST /fetch routes the URL through a hosted Headless Chrome instance behind rotating proxies and returns the fully rendered HTML, which is the right call for single-page applications and dynamically loaded content.

### What are the rate limits for the Dataflow Kit Web Scraper API?

Concurrency and monthly request quotas depend on the Dataflow Kit plan tied to your api_key - see account.dataflowkit.com for current limits. The spec does not enforce per-endpoint limits.

### How do I scrape Google search results through Jentic?

Search Jentic for 'scrape Google search results', load the POST /serp operation, and execute it with your query, target engine, and country. Jentic injects the api_key automatically and returns the parsed result list.

### Can the Dataflow Kit API capture screenshots and PDFs of web pages?

Yes. POST `/convert/url/screenshot` returns a full-page PNG and POST `/convert/url/pdf` returns a print-ready PDF of any URL, both rendered through the same Headless Chrome backend used for /fetch.

### Does Dataflow Kit handle proxies and CAPTCHAs?

Yes. The service manages a rotating proxy pool and CAPTCHA evasion internally, so callers only need to supply the target URL or query - there is no separate proxy configuration in the API.

### Can I limit what my agent is allowed to do with the Dataflow Kit Web Scraper API?

Yes. Because you run Jentic One yourself, your own rules decide which Dataflow Kit operations the agent may call and which credentials it may use. Since Dataflow Kit takes target URLs and parse rules in the request body rather than a resource id in the path, you scope access per operation: allow only POST /fetch to render pages, or add POST /serp for search results, while withholding POST /parse, POST `/convert/url/pdf`, or POST `/convert/url/screenshot.` Every operation the agent can run is one you explicitly added to the allowed set.