Wayback Machine CDX Snapshot List Scraper

Wayback Machine CDX Snapshot List Scraper

Scrape Wayback Machine CDX snapshot lists for any URL or domain, up to a million per run. Every row returns the timestamp, original URL, snapshot URL, HTTP status, MIME type, and content digest. Filter by date, status code, or MIME type. Export to CSV, JSON, Excel, or XML.

The Wayback Machine holds billions of historical page captures, but browsing them one at a time is slow. This Actor queries the CDX index directly, so you get a complete, filterable list of snapshots for a domain, host, or exact URL in seconds. No browser, no pagination tricks, the raw index data.

Who uses it	What they scrape Wayback Machine for
SEO analysts	Audit how a competitor's page content and status codes changed over years.
OSINT researchers	Map every capture of a domain to find deleted pages or hidden redirects.
Digital forensics teams	Collect a full timeline of a site's HTTP responses and content hashes as evidence.
Brand protection managers	Monitor historical use of a trademark across domains and subdomains.

What it does

This Actor queries the Wayback Machine CDX index for a URL or domain and returns each matching snapshot as a flat row with timestamp, original URL, snapshot URL, status code, MIME type, and digest.

• 🎯 Match type control: exact URL, path prefix, host only, or entire domain including all subdomains.
• 📅 Timestamp range filter: set a From and To date in yyyyMMddHHmmss format to narrow the window.
• 📋 MIME type filter: regex on the mimetype field, with an optional ! prefix to invert the match.
• 🚦 HTTP status filter: regex on the status code, so you can pull only 200s, all 4xx errors, or exclude 404s.
• 🧹 Collapse duplicates: deduplicate adjacent captures by timestamp:8 (daily), digest (content-identical), or urlkey.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with Wayback Machine data

📈 Audit competitor site history.

An SEO analyst runs the Actor on a competitor's domain with a status filter of 200 and a MIME filter of text/html to chart every successful page capture over five years.

🔍 Find deleted or hidden pages.

An OSINT investigator queries a domain with match type set to domain and no status filter, then sorts the output to spot URLs that later returned 404 or 301.

🧾 Build an evidence timeline.

A forensics analyst sets a narrow date range and collapses by digest to collect one capture per content change, producing a lightweight change log for a legal hold.

🛡️ Monitor trademark use.

A brand manager runs the Actor monthly on a set of domains, filtering for text/html snapshots, and reviews new captures for unauthorized use of brand terms.

Why choose this scraper

	What you get
Full index access	Query the CDX API directly instead of scraping the calendar UI, so you get every capture the index knows about.
Up to 1M rows	Collect as many as a million snapshots per run, far beyond the browser-based calendar view.
Server-side filters	MIME type, status code, and timestamp filters run on the CDX server, so you download only matching rows.
Fixed flat schema	Every snapshot arrives with the same columns: timestamp, original, snapshot URL, status, MIME, and digest.

How it compares

This Actor focuses on high-volume CDX index queries with server-side filtering, while other Wayback Machine scrapers often emphasize page content retrieval or browser-based calendar scraping.

Feature	ParseForge	Wayback Machine Scraper - Track Website Changes Over Time	Wayback Machine Archive Scraper	Wayback Machine Scraper - Archived Snapshots
Queries the CDX index directly	Yes	Not listed	Not listed	Yes
Up to 1,000,000 snapshots per run	Yes	Not listed	Not listed	Not listed
Server-side MIME type regex filter	Yes	Not listed	Not listed	Not listed
Server-side HTTP status regex filter	Yes	Not listed	Not listed	Not listed
Collapse adjacent duplicates by field	Yes	Not listed	Not listed	Yes
Downloads archived page content	Not listed	Not listed	Yes	Not listed

Configure the run

Drive the Actor with a single URL or domain, choose how broadly to match it, and apply server-side filters for date range, MIME type, HTTP status, and duplicate collapsing so only the snapshots you need reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
 "urlOrDomain": "apify.com",
 "maxItems": 10
}

A larger pull:

{
 "urlOrDomain": "apify.com",
 "maxItems": 200
}

Pricing

Pay-per-result: $0.011 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$1.10
1,000 results	$11.00
10,000 results	$110.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the Wayback Machine CDX Snapshot List Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to Wayback Machine through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/wayback-cdx-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check that the domain or URL was ever captured by the Wayback Machine. Try broadening the match type to domain and removing all filters. Also verify the URL format: include the scheme only for exact or prefix matches.

Why are my results capped at a lower number than I set?

The CDX index may have fewer captures than your maxItems value for that query. Try removing the From and To filters or broadening the match type to see if more snapshots exist.

The collapse filter is not reducing rows as expected.

Collapse removes only adjacent duplicates in the CDX result order. If identical captures are not adjacent, they will not be collapsed. Try sorting externally or using a more specific collapse field.

My MIME type regex is not matching anything.

The filter is a regex applied to the full mimetype string. Use ^text/ to match any text type, or ^image/ for images. Test your regex with a tool first, and remember to escape special characters.

The run times out on a very large domain.

Lower the maxItems value or add date range filters to reduce the result set. The CDX API can be slow for domains with millions of captures.

FAQ

Question	Answer
What is the CDX index?	The CDX index is the Internet Archive's searchable catalog of every snapshot stored in the Wayback Machine. It returns metadata about each capture, not the page content itself.
Does this Actor download the archived page content?	No. It returns the snapshot list from the CDX index. Each row includes a snapshot URL that you can use with another Actor or tool to fetch the actual archived HTML.
How many snapshots can I get in one run?	You can set the maximum up to 1,000,000 snapshots per run. The actual number returned depends on how many captures exist in the index for your query and filters.
What does the match type setting do?	Exact URL returns captures of only that precise URL. Path prefix returns captures whose path starts with the given string. Host returns captures on the same hostname. Domain returns captures on the host and all its subdomains.
How do I filter by date?	Use the From and To fields with a timestamp format like yyyyMMddHHmmss. For example, 20200101 gives all of January 1, 2020 onward, and 20200101120000 starts at noon that day.
Can I get only successful (200 OK) captures?	Yes. Set the HTTP status filter to 200, or use a regex like ^2[0-9]{2}$ to get any 2xx status code.
What does the collapse option do?	It removes adjacent duplicate rows based on a CDX field. For example, timestamp:8 keeps only one capture per day, and digest drops captures whose page content hash is identical to the previous one.
How do I invert a MIME type or status filter?	Prefix the regex with an exclamation mark. For example, !text/html returns all captures whose MIME type is not text/html.
Can I scrape multiple domains in one run?	Each run targets one URL or domain. To scrape multiple domains, run the Actor once per domain, or use a separate orchestrator Actor to loop over a list.
Is an Internet Archive account required?	No. The CDX API is publicly accessible. You do not need an account or API key to use this Actor.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Internet Archive. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.