canonical: https://jentic.com/apis/botify.com/botify

# Botify API

Botify is an enterprise SEO platform that crawls large websites, joins crawl data with Google Analytics and log files, and surfaces technical SEO issues at scale. The API exposes analyses, projects, crawl statistics, URL-level datamodel queries, sitemap reports, internal linking metrics, and orphan URL detection so SEO teams can pipe insight into their dashboards and automation. Authentication uses a Django Rest token in the Authorization header. Use it when an enterprise site needs deep crawl analytics that go beyond a desktop crawler.

## For AI agents

Pull enterprise SEO crawl analyses, sitemap reports, internal-link metrics, and URL-level datamodel queries from Botify across 26 endpoints.

## Scope

Does not handle keyword tracking, backlink analysis, or paid search reporting - use for enterprise on-site crawl analytics, sitemap audits, and URL-level data only.

## Capabilities

- List analyses for a Botify project and inspect a specific analysis
- Retrieve crawl statistics including time-series and URL lists by status type
- Query URL-level data with aggregations and suggested filters
- Pull sitemap audit reports and out-of-config or sitemap-only URL samples
- Surface lost-PageRank URLs and link percentile distributions
- List orphan URLs detected via Google Analytics by medium and source
- Export filtered URL datasets for downstream analysis

## Use cases

### Crawl status reporting into a BI dashboard

Enterprise SEO teams pull crawl health metrics from Botify into a BI dashboard so leadership can monitor indexable pages, response code distribution, and crawl duration over time. GET `/analyses/{username}/{project_slug}/{analysis_slug}/crawl_statistics` returns the high-level snapshot and the matching /time endpoint returns the time-series view. Most teams wire this into Looker or Power BI in a sprint.

Example prompt: Pull GET crawl_statistics for project mybrand-www analysis 2026-06-01 and report counts of 200, 301, 404, and 5xx URLs

### Filtered URL exports for content audits

Content and SEO teams export a filtered slice of URLs (for example, low-traffic pages with thin content or pages with many internal links and no organic clicks) to drive an audit. POST `/analyses/{username}/{project_slug}/{analysis_slug}/urls/export` submits an export job with filters; GET `/urls/export/{url_export_id}` polls for completion. Audits typically run weekly with a saved filter.

Example prompt: Submit a URL export for analysis 2026-06-01 filtering to depth>=3 and pageviews<10, then poll the export id until complete and return the download URL

### Sitemap and orphan URL hygiene

SEO teams reconcile sitemaps against the live crawl to find URLs that should be indexed but are missing from sitemaps, and orphan URLs that get organic traffic but are not internally linked. The features/sitemaps and features/ganalytics/orphan_urls endpoints surface both lists for action by the content team. This is typically run after every monthly recrawl.

Example prompt: Get the sitemap report and out_of_config sample for project mybrand-www analysis 2026-06-01 and return the count of out-of-sitemap URLs

### Agent-driven SEO insight retrieval via Jentic

An AI agent answers questions like 'how many 404s did the latest crawl find on mybrand-www?' by calling Botify directly. Through Jentic the agent searches for 'Botify crawl statistics', loads the schema, and executes the call while the Authorization token stays inside your Jentic One instance.

Example prompt: Search Jentic for 'Botify crawl statistics', load the schema, and execute it for project mybrand-www analysis 2026-06-01, returning the 404 count

## Key endpoints

| Method | Path | Description |
| --- | --- | --- |
| GET | `/analyses/{username}/{project_slug}` | List analyses for a project |
| GET | `/analyses/{username}/{project_slug}/{analysis_slug}/crawl_statistics` | High-level crawl statistics for an analysis |
| POST | `/analyses/{username}/{project_slug}/{analysis_slug}/urls` | Query URL-level data with filters |
| POST | `/analyses/{username}/{project_slug}/{analysis_slug}/urls/aggs` | Run aggregations over URL data |
| POST | `/analyses/{username}/{project_slug}/{analysis_slug}/urls/export` | Start a filtered URL export |
| GET | `/analyses/{username}/{project_slug}/{analysis_slug}/features/sitemaps/report` | Sitemap audit report |
| GET | `/analyses/{username}/{project_slug}/{analysis_slug}/features/pagerank/lost` | Lost PageRank between analyses |
| GET | `/projects/{username}/{project_slug}/filters` | List saved filters on a project |

## Key resources

- **Analyses** — List and inspect crawl analyses for a project
- **Crawl Statistics** — High-level and time-series crawl metrics with URL lists by status
- **URLs** — Query, aggregate, and export URL-level datamodel data
- **Sitemaps** — Sitemap audit reports and out-of-config or sitemap-only samples
- **Links** — Internal link percentiles and lost-PageRank reports
- **Orphan URLs** — URLs receiving organic traffic without internal links, grouped by medium and source
- **Projects** — List projects for a user and access saved filters

## Why Jentic

- **Setup:** Wiring Botify by hand means passing the Django Rest token in the Authorization header and navigating its nested analysis paths keyed by username, project, and analysis slug. Through Jentic you install once, import the Botify API from the API Directory, store the token once, and your agent calls it.
- **Permission scoping:** Botify puts the account and project in the URL path (`/analyses/{username}/{project_slug}/...`), so a rule can pin your agent to one project's analyses. You choose the operations it may call, so URL query and export operations are only included if you add them, and a monitoring agent can be held to the crawl statistics reads.
- **Credential handling:** Your Botify Django Rest token is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.
- **Discovery method:** Agents search Jentic by intent such as 'pull Botify crawl statistics' or 'export Botify URLs', and Jentic returns the matching Botify operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

## Related APIs

- **Ahrefs API** — Backlink and keyword data that pairs with Botify's on-site crawl analytics.
- **Semrush API** — Keyword research and competitor data complementing on-site crawl analytics from Botify.
- **Moz API** — SEO data provider with link metrics and keyword data, lighter on enterprise crawl analytics.

## FAQ

### What authentication does the Botify API use?

Botify uses a Django Rest token in the Authorization request header (named DjangoRestToken in the spec). Generate the token from your Botify account; in Jentic it lives in the vault and is injected at execution so it never appears in agent prompts or logs.

### Can I export a filtered list of URLs from a Botify analysis?

Yes. POST `/analyses/{username}/{project_slug}/{analysis_slug}/urls/export` with the filters in the request body to start an export job, then poll GET `/analyses/{username}/{project_slug}/{analysis_slug}/urls/export/{url_export_id}` for the download link. Use POST `/urls/suggested_filters` to discover useful filter shapes for the dataset.

### How do I get crawl status counts for an analysis?

Call GET `/analyses/{username}/{project_slug}/{analysis_slug}/crawl_statistics` for the snapshot, or `/crawl_statistics/time` for a time series. To list URLs in a particular status bucket, call `/crawl_statistics/urls/{list_type}` with values such as ok, redirect, or error.

### How do I run a URL aggregation through Jentic?

Run pip install jentic, then search Jentic for 'Botify URL aggregation', load the POST `/analyses/{username}/{project_slug}/{analysis_slug}/urls/aggs` schema, and execute it with the aggregation body. Jentic handles the Authorization header automatically.

### What are the rate limits for the Botify API?

Botify does not document explicit per-second rate limits in the spec, but URL aggregation and export operations are heavyweight; large analyses commonly take many seconds to return. Use the export endpoint plus polling instead of repeated synchronous queries for large datasets.

### Can I detect orphan URLs that get organic traffic?

Yes. GET `/analyses/{username}/{project_slug}/{analysis_slug}/features/ganalytics/orphan_urls/{medium}/{source}` returns URLs that received organic traffic from the specified medium and source but were not internally linked in the crawl, which is the standard input for an internal linking remediation backlog.

### Can I limit what my agent is allowed to do with the Botify API?

Yes. Because you run Jentic One yourself, your own rules decide which Botify operations and credentials the agent may use. Since Botify puts the account and project in the URL path (`/analyses/{username}/{project_slug}/...`), you can pin an agent to a single project's analyses and grant only the operations it needs. For example, you can hold a monitoring agent to crawl statistics reads and include the heavier URL query and export operations only when you explicitly add them.