Sitemap

Finds a domain's sitemap, walks sitemap indexes, and returns the URLs it lists. Use it to inventory a site's indexable pages or to check what a site tells crawlers about. This is a live fetch of the site's own files, not link graph data.

Request

Parameters

Example

curl "https://api.rankparse.com/v1/sitemap?domain=stripe.com&limit=50" \
  -H "X-API-Key: rp_your_key"

import requests

resp = requests.get(
    "https://api.rankparse.com/v1/sitemap",
    params={"domain": "stripe.com", "limit": 50},
    headers={"X-API-Key": "rp_your_key"},
    timeout=30,
)
resp.raise_for_status()
body = resp.json()
for entry in body["data"]:
    print(entry["url"], entry.get("lastmod"))
if body["truncated"]:
    print("Walk stopped early; the site has more URLs than returned.")

const res = await fetch(
  "https://api.rankparse.com/v1/sitemap?" +
    new URLSearchParams({ domain: "stripe.com", limit: "50" }),
  { headers: { "X-API-Key": process.env.RANKPARSE_API_KEY } }
);
if (!res.ok) throw new Error(`${res.status}: ${await res.text()}`);
const body = await res.json();
for (const entry of body.data) console.log(entry.url, entry.lastmod);

Response

{
  "domain": "stripe.com",
  "data": [
    {
      "url": "https://stripe.com/about",
      "lastmod": "2024-01-15",
      "changefreq": "monthly",
      "priority": "0.8"
    },
    {
      "url": "https://stripe.com/blog",
      "lastmod": "2024-03-20",
      "changefreq": "weekly",
      "priority": "0.9"
    }
  ],
  "credits_used": 2,
  "credits_remaining": 998,
  "total": 142,
  "sitemap_url": "https://stripe.com/sitemap.xml",
  "sitemap_type": "sitemap",
  "child_sitemaps": [],
  "child_sitemaps_total": 0,
  "child_sitemaps_fetched": [],
  "truncated": false,
  "source": "sitemap"
}

When no sitemap can be found or read, you still get a 200, but nothing is charged:

{
  "domain": "example.com",
  "data": [],
  "credits_used": 0,
  "credits_remaining": 1000,
  "total": 0,
  "sitemap_url": null,
  "sitemap_type": "none",
  "child_sitemaps": [],
  "child_sitemaps_total": 0,
  "child_sitemaps_fetched": [],
  "truncated": false,
  "source": "none"
}

Response fields

Behavior and limits

Discovery order

1. Every Sitemap: line in /robots.txt. If there is at least one, these are the roots and nothing else is tried.
2. Otherwise /sitemap.xml, then /sitemap_index.xml, accepted only if the document contains a urlset or sitemapindex.

The walk is bounded

• Sitemap indexes are followed breadth-first up to three tiers (root index, child index, urlset).
• At most 16 documents are read per request, in batches of 5 at a time, within a 12 second budget. Root documents get 8 seconds each, children 5.
• Children of a child index (the third tier): each index contributes at most 5, and at most 10 are read in total. Children of a root index are limited only by the 16 document cap.
• Each document must be 2 MB or smaller.
• When any bound is hit, or limit is reached while work remains, truncated is true.
• A child that fails to load is skipped without failing the request.

Billing

• Costs 2 credits up front. When no sitemap is found or readable (source: "none"), the 2 credits are refunded and credits_used is 0.
• Failures (4xx/5xx after the charge) are refunded automatically. Invalid domains are rejected before billing.
• Subject to the standard 60 requests per minute limit.

Legacy route

GET /v1/url-index redirects (301) to /v1/sitemap, forwarding domain and limit. See Not yet available.

Errors

An unreachable site or one without a sitemap is not an error: it returns 200 with source: "none".

See Errors for the shared error format.

Related

MCP tool: get_sitemap (arguments domain and optional limit).