Crawl History

Crawl History

Get when a domain was first indexed, its most recent snapshot, and total archive snapshot count via the Wayback Machine.

GET /v1/crawl-history

Returns a domain's internet history using the Wayback Machine CDX API. Costs 2 credits.

Parameters

Parameter	Type	Required	Description
`domain`	string	Yes	The domain to query

Request

curl "https://api.rankparse.com/v1/crawl-history?domain=example.com" \
  -H "X-API-Key: rp_your_key"

Response

{
  "data": {
    "domain": "example.com",
    "first_seen": "1999-01-13",
    "last_seen": "2024-11-20",
    "total_snapshots": 4231,
    "source": "wayback_machine"
  },
  "credits_used": 2
}

Notes

first_seen and last_seen are the earliest and latest Wayback Machine snapshot dates (YYYY-MM-DD)
total_snapshots is the total number of archived snapshots across all time
null values indicate the domain has no Wayback Machine history
source is always "wayback_machine"

Tech Stack

Real-time technology detection for a domain — frameworks, CMS, analytics, CDN, hosting, and 50+ more technologies.

Domain Overlap

Find domains that link to multiple target domains.