AI visibility
A daily background scan of your verified websites for AI search visibility, agent readiness and basic site health.
The AI visibility scan checks whether AI search and answer agents can find, crawl and read your website, which optional agent protocols it publishes, and a few classic site health basics. RankParse runs it in the background once a day for every eligible website and stores the results. The API reads those stored results and can queue a fresh scan.
| Operation | Endpoint | Credits |
|---|---|---|
| Get AI visibility | GET /v1/site-scan | 0 |
| Re-check AI visibility | POST /v1/site-scan/recheck | 0 |
Eligibility
A website is scanned only when both are true:
- It is verified: an active, verified Google Search Console or Bing Webmaster Tools property that matches the website's host is attached to it.
- The workspace is on a paid plan: Pro, Agency or Lifetime (including workspaces in a launch grace period that counts as one of these). The Free plan is not scanned.
An unverified website is never fetched. GET /v1/site-scan answers eligible: false with reason: "not_verified", and on the Free plan reason: "plan_required". Results stored before a workspace moved to Free are kept but not shown.
How a scan runs
- Daily: an hourly job queues verified websites in paid workspaces whose last scan is more than 24 hours old, least recently scanned first, up to 100 websites per run. A website queued in the last 2 hours is skipped.
- On request: Re-check AI visibility queues one website, at most once per 10 minutes. The scan usually finishes within a few minutes.
- What is fetched: the home page at
https://<host>/first (the first 1 MB is read; a429is retried once after 2 seconds), then robots.txt, the sitemap (up to 3 declared in robots.txt, tried in order, else/sitemap.xml), llms.txt, the home page again withAccept: text/markdown, the well-known files of the optional checks,http://<host>/and one DNS-over-HTTPS lookup. Everything excepthttp://<host>/is read on the origin the home page lands on after redirects. Each fetch has an 8 second timeout and blocks private addresses. The user agent isMozilla/5.0 (compatible; RankParse/1.0; +https://rankparse.com). - Unreachable home page: when the home page cannot be fetched or answers
400or above, the scan records anerror_code(for examplehome_http_503orhome_timeout) and leaves the stored checks as they were. - Unknown never overwrites: a check that cannot tell (timeout,
5xx, a body too large to read) isunknown. Anunknownresult keeps the previouspassorfailand its evidence, so a timeout neither raises nor clears an issue.
Check statuses and severity
Each check is pass, fail or unknown. Its severity says how much a failure matters:
bad: a real problem, listed as an issue.warn: worth fixing, listed as an issue.info: optional. An optional protocol feature or a deliberate choice. Never an issue and never counted in a score. Most websites need no MCP server card or API catalog, and blocking AI training crawlers is often intentional.
Checks
The API lists checks in this order. area groups them; the site_health area is reported as category: "site_health", every other area as category: "ai_visibility".
AI visibility
| Check | Area | Severity | Passes when |
|---|---|---|---|
robots_txt | discoverability | warn | /robots.txt answers 200 with a text file (not an HTML page, not another declared type). A 4xx fails. |
sitemap | discoverability | warn | A sitemap declared in robots.txt (up to 3 tried), or /sitemap.xml when none is declared, answers 200 with something other than an HTML page. |
ai_answer_bots_blocked | bot_access | bad | robots.txt does not block / for any AI search or answer agent: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User. A missing robots.txt passes. Fails when one is blocked by its own rule or by the User-agent: * rule. |
ai_training_bots_blocked | bot_access | info | robots.txt does not block / for any AI training crawler: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider. |
llms_txt | content | warn | /llms.txt answers 200 with a text file. |
markdown_negotiation | content | warn | The home page requested with Accept: text/markdown answers 2xx with Content-Type: text/markdown. |
structured_data | content | warn | The home page has at least one JSON-LD block that parses. Unknown when the page was cut at 1 MB and no valid block was found. |
link_header | discoverability | info | The home page response sends a Link header with a rel (RFC 8288). |
content_signal | bot_access | info | robots.txt has a Content-Signal line (search, ai-input, ai-train). |
web_bot_auth | bot_access | info | /.well-known/http-message-signatures-directory answers 200 with JSON. |
api_catalog | capabilities | info | /.well-known/api-catalog answers 200 with JSON (RFC 9727). |
oauth_discovery | capabilities | info | /.well-known/oauth-authorization-server (RFC 8414) or /.well-known/oauth-protected-resource (RFC 9728) answers 200 with JSON. |
mcp_server_card | capabilities | info | /.well-known/mcp/server-card.json answers 200 with JSON. |
agent_skills | capabilities | info | /.well-known/agent-skills/index.json answers 200 with JSON. |
oauth_protected_resource | capabilities | info | /.well-known/oauth-protected-resource is a JSON object with a non-empty resource string. |
a2a_agent_card | capabilities | info | /.well-known/agent-card.json, or the older /.well-known/agent.json, is a JSON object with non-empty name and url strings. |
auth_md | capabilities | info | /auth.md answers 200 with a non-empty text or Markdown file. |
dns_aid | discoverability | info | DNS has an SVCB record at _index._agents.<host> (DNS-AID, an IETF draft), looked up over DNS-over-HTTPS. |
Site health
| Check | Area | Severity | Passes when |
|---|---|---|---|
https_enforced | site_health | warn | http://<host>/ ends up on an https:// URL after redirects. |
hsts | site_health | info | The HTTPS home page sends Strict-Transport-Security. |
home_indexable | site_health | bad | The home page has no noindex (or none) in its robots meta tag or X-Robots-Tag header. |
canonical_host | site_health | warn | The home page has no canonical URL, or its canonical is on the same host (ignoring www.). |
home_title | site_health | warn | The home page has a title. |
home_description | site_health | warn | The home page has a meta description. |
For the file checks, a 200 that is an HTML page (a single-page app answering every path) fails with reason: "html_instead_of_file", a JSON check that is not JSON fails with reason: "not_json", and a 5xx or network error is unknown. When the home page was cut at 1 MB before </head>, a missing title, description, canonical or robots meta is unknown, not fail.
This scan is separate from the live Site health endpoint, which checks any domain on demand for credits.
Scoring
Each area and the overall result get a score: passing / (passing + failing) as a whole percentage, over bad and warn checks only. unknown results and optional (info) checks are left out, so a blog without an MCP server card can score 100. A score is null when no check counted. overall is the AI visibility score: site health checks count only in the site_health area.
Related
MCP tools: get_ai_visibility and recheck_ai_visibility (both take site_id; on the API-key MCP server).