canonical: https://jentic.com/apis/olostep.com/olostep

# Olostep API

The Olostep API scrapes, crawls, and maps websites and returns their content as markdown, HTML, text, JSON, raw PDF, or screenshots. A single scrape can wait before loading, run page actions, strip CSS selectors or images, use a named parser, and route through a chosen country, while batches process many URLs at once and crawls follow a site from a start_url within max_pages and max_depth limits. It also runs natural-language web searches, answers a task against the web in a requested json_format, and keeps monitors that rerun on a schedule and record change events. Schedules, JSON file uploads, and credit balance routes round out the surface.

## For AI agents

Scrape a URL to markdown or JSON, crawl or map a whole site, batch many URLs, search the web, get structured answers, and monitor pages for changes.

## Scope

Does not host websites, register domains, or run browser test suites. Use for scraping, crawling, searching, and monitoring web content only.

## Capabilities

- Scrape a url_to_scrape into markdown, html, text, json, raw_pdf, or screenshot formats, with page actions, a parser, and a scrape country
- Crawl a site from a start_url with include_urls, exclude_urls, max_pages, max_depth, and follow_robots_txt, then page through the crawled pages
- Map every URL on a site and narrow it with search_query and top_n
- Submit a batch of items with custom_id and url and list processed items by status with a cursor
- Run a natural-language web search with include_domains and exclude_domains, or ask a task and get an answer shaped by json_format
- Create a monitor from a natural-language query and frequency, notify by email, Slack, or SMS on changed events, and read its change events
- Schedule a recurring or one-time call to an Olostep endpoint with a cron_expression or execute_at time

## Use cases

### Web reading for an AI agent

An agent that answers questions from live web pages needs clean text, not raw HTML. Through Jentic it searches for 'scrape a web page to markdown', loads the Olostep create-scrape operation, and calls it with url_to_scrape and formats set to markdown, with the bearer key attached at call time. The agent gets page content it can quote, and can switch to a json format or a screenshot when the page needs it.

Example prompt: Call POST `/v1/scrapes` with url_to_scrape 'https://example.com/pricing', formats ['markdown'], and country 'US', then summarise the plans listed on the page

### Whole-site crawl for a knowledge base

To load a documentation site into a retrieval index, POST `/v1/crawls` starts from a start_url and follows links within include_urls and exclude_urls patterns, max_pages, and max_depth, honouring follow_robots_txt when set. GET `/v1/crawls/{crawlId}` reports progress, and GET `/v1/crawls/{crawlId}/pages` pages through results with a cursor and the requested formats.

Example prompt: POST `/v1/crawls` with start_url 'https://docs.example.com', max_pages 200, and max_depth 3, poll GET `/v1/crawls/{crawlId}` until it finishes, then page GET `/v1/crawls/{crawlId}/pages` with formats 'markdown'

### Change monitoring with alerts

A team tracking competitor pricing or policy pages sets up a monitor described in plain language with a natural-language frequency such as every day at 9am, which the spec bounds to a minimum interval of 10 minutes. The notification block sends changed or first_snapshot events to email, Slack, or SMS channels, and GET `/v1/monitors/{monitorId}/events` lists the recorded changes. Monitors can be paused and resumed without deleting them.

Example prompt: POST `/v1/monitors` with query 'pricing on https://competitor.example.com/pricing', frequency 'every day at 9am', and a notification channel of type 'slack', then read GET `/v1/monitors/{monitorId}/events`

### Bulk URL extraction

A data job with a list of product or listing URLs submits them as one batch to POST `/v1/batches`, each item carrying a url and an optional custom_id, with an optional parser and webhook. GET `/v1/batches/{batchId}` shows progress, and GET `/v1/batches/{batchId}/items` lists processed items filtered by status, with a retrieve_id that GET `/v1/retrieve` turns into the stored content.

Example prompt: POST `/v1/batches` with items listing each url and custom_id, poll GET `/v1/batches/{batchId}`, then page GET `/v1/batches/{batchId}/items` with status 'completed'

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| POST | `/v1/scrapes` | Scrape a URL |
| POST | `/v1/crawls` | Start a site crawl |
| GET | `/v1/crawls/{crawlId}/pages` | List crawled pages |
| POST | `/v1/maps` | Map URLs across a site |
| POST | `/v1/batches` | Start a batch of URLs |
| POST | `/v1/searches` | Run a web search |
| POST | `/v1/answers` | Get a structured answer |
| POST | `/v1/monitors` | Create a change monitor |

## Key resources

- **Scrapes and retrieval** — Single-URL scrapes in several output formats and retrieval of stored content by retrieve_id
- **Crawls, maps, and batches** — Site crawls with their pages, URL maps of a site, and batches of URLs with their processed items
- **Searches and answers** — Natural-language web searches and structured answers to a task
- **Monitors and schedules** — Change monitors with events, runs, planning, and logs, plus scheduled calls to Olostep endpoints
- **Files and credits** — JSON file uploads with download links, credit balance, and credit top-ups

## AI readiness

This API is usable in Jentic One now. Its AI-readiness score against Jentic's framework shows where it stands today and where improvements would make it even easier for agents to use.

- **Score:** 65 / 100
- **Maturity:** AI-Aware
- **Dimensions:**
  - Foundational Compliance: 99 / 100
  - Developer Experience & Jentic Compatibility: 69 / 100
  - AI-Readiness & Agent Experience: 42 / 100
  - Agent Usability: 94 / 100
  - Security: 60 / 100
  - AI Discoverability: 79 / 100
- **View full report:** https://jentic.com/apis/olostep.com/olostep/scorecard
- **How the score is calculated:** https://docs.jentic.com/reference/api-readiness-framework/overview/
- **More about the dimensions:** https://docs.jentic.com/reference/api-readiness-framework/specification/#dimensional-model-overview

### Score it yourself

Every API in the directory is allowlisted, so you can re-score it with no key required.

- **Score your own API:** https://jentic.com/scorecard.md
- **Scoring CLI agent skill:** https://github.com/jentic/jentic-api-scorecard/blob/main/skills/jentic-api-scorecard/SKILL.md

```sh
npx @jentic/api-scorecard-cli score <openapi-url>
```

## Why Jentic

- **Setup:** Wiring Olostep by hand means sending a bearer key, polling async crawls, batches, and answers by id, following cursors, and tracking credits. Through Jentic you install once, import Olostep from the API Directory, store the key once, and your agent calls it.
- **Permission scoping:** Olostep puts resource ids in the URL path, such as `/v1/monitors/{monitorId}` and `/v1/batches/{batchId}`, so a rule can pin your agent to one monitor or batch and the operations you pick. You choose the operations, so destructive or spending ones like deleting a monitor, deleting a file, or purchasing a credit top-up are not included unless you add them.
- **Credential handling:** Your Olostep API key is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'scrape a web page to markdown' or 'monitor a page for changes', and Jentic returns the matching Olostep operation with its input schema, including fields like url_to_scrape and formats, so the agent calls it correctly.

## Related APIs

- **Firecrawl API** — Scrape and crawl websites into LLM-ready markdown
- **Apify API** — Run prebuilt and custom scraping actors and read their datasets
- **Tavily API** — Web search tuned for AI agents
- **Exa** — Neural web search that finds pages by meaning

## FAQ

### What authentication does the Olostep API use?

Per its OpenAPI spec, every Olostep operation uses HTTP bearer authentication: send your Olostep API key as Authorization: Bearer followed by the key. The spec notes that access tokens issued for the Olostep dashboard are also accepted. Through Jentic the key is stored by your own instance and attached at call time, so the agent never handles it.

### Which output formats can an Olostep scrape return?

POST `/v1/scrapes` takes a formats array with values markdown, html, text, json, raw_pdf, and screenshot, defaulting to markdown. The same request can set wait_before_scraping, page actions, remove_css_selectors, remove_images, a parser, and a two-letter country to scrape from.

### What are the rate limits for the Olostep API?

The OpenAPI spec does not specify request rate limits. Usage draws on your team's credits, which GET `/user/credits/info` reports. Check the Olostep documentation at https://docs.olostep.com for current limits.

### Is there an Olostep MCP server?

You don't need an MCP server to give your agent Olostep. Jentic connects it directly from the API Directory: import it, store your API key once, and your agent calls the scrape, crawl, and search operations it needs on demand without loading another server's tool definitions into its context.

### How do I crawl a website through Jentic?

Search Jentic for 'crawl a website', load the Olostep create-crawl operation, and execute it with a start_url plus limits such as max_pages and max_depth. Poll the crawl by its id, then list its pages with a cursor. To run it on your own infrastructure, install Jentic One from its GitHub repo.

### Can Olostep tell me when a web page changes?

Yes. POST `/v1/monitors` takes a natural-language query and frequency, with a minimum interval of 10 minutes, plus an optional source_policy, output_schema, webhook, and notification channels of type email, slack, or sms for changed and first_snapshot events. GET `/v1/monitors/{monitorId}/events` lists the change events, and the pause and resume routes stop and restart it.

### Can I limit what my agent is allowed to do with the Olostep API?

Yes. Write a rule that allows only GET `/v1/monitors/{monitorId}` and GET `/v1/monitors/{monitorId}/events` for one monitor id, plus POST `/v1/scrapes`, and the agent can scrape pages and read that monitor's changes but cannot delete monitors, buy credits, or create schedules, and every call it makes is logged.