Sitemap

GET /v1/sitemap

Discovers a domain's sitemap via robots.txt, then parses and returns the URL inventory. Costs 2 credits, refunded in full when no sitemap is found (source: "none", credits_used: 0).

Parameters

Request

curl "https://api.rankparse.com/v1/sitemap?domain=stripe.com&limit=50" \
  -H "X-API-Key: rp_your_key"

Response

{
  "data": [
    {
      "url": "https://stripe.com/about",
      "lastmod": "2024-01-15",
      "changefreq": "monthly",
      "priority": "0.8"
    },
    {
      "url": "https://stripe.com/blog",
      "lastmod": "2024-03-20",
      "changefreq": "weekly",
      "priority": "0.9"
    }
  ],
  "total": 142,
  "sitemap_url": "https://stripe.com/sitemap.xml",
  "sitemap_type": "sitemap",
  "child_sitemaps": [],
  "source": "sitemap",
  "credits_used": 2
}

When no sitemap is found

{
  "data": [],
  "total": 0,
  "sitemap_url": null,
  "sitemap_type": "none",
  "child_sitemaps": [],
  "source": "none",
  "credits_used": 0
}

Notes

• Discovery order: every Sitemap: directive in robots.txt → /sitemap.xml → /sitemap_index.xml
• Sitemap indexes are walked breadth-first, including indexes nested one level below the root, and the walk stops as soon as limit URLs are collected
• The walk is bounded: at most 16 documents and a 12-second budget. When a bound stops it early, truncated is true and data/total under-report the site
• child_sitemaps lists the child sitemaps discovered (capped at 200 entries); child_sitemaps_total is the true count and child_sitemaps_fetched names which of them were actually read for this response
• sitemap_url is the first sitemap that could be read, and sitemap_type describes that document. When robots.txt lists several sitemaps rather than an index, the rest appear in child_sitemaps
• A child sitemap that fails to fetch is skipped; it does not fail the request
• lastmod, changefreq, and priority are undefined if not present in the sitemap
• The legacy /v1/url-index route redirects here (301), forwarding domain and limit