Internet Archive Search Scraper

Internet Archive Search Scraper

Scrape Internet Archive search results by keyword, collection, media type, or creator, up to a million items per run. Each item returns its title, identifier, date, and optional full metadata including file lists. No API key required. Export to CSV, JSON, Excel, or XML.

The Internet Archive's public search API returns only five results at a time and requires pagination logic to collect at scale. This Actor runs your Lucene query against the archive's index, applies filters for collection, media type, creator, and date range, and returns every matching item in one dataset. It can also fetch the full metadata for each item, giving you file URLs, subject tags, and description text without writing a second request.

Who uses it	What they scrape Internet Archive for
Digital archivists	Audit a collection for missing metadata or date gaps.
Researchers	Gather all public-domain texts, audio, or films on a topic for a corpus.
Data journalists	Map the release history of government documents or NASA media.
Content curators	Build a list of open-source movies or Librivox audiobooks for a directory.

What it does

This Actor collects Internet Archive items from a search query and returns each one as a flat row with optional full metadata.

• 🔎 Lucene search query: full-text search with boolean operators, field scoping, and ranges.
• 📚 Collection filter: restrict results to a single collection slug like 'nasa' or 'opensource_movies'.
• 🎬 Media type filter: limit to texts, audio, movies, software, web, image, data, or collections.
• 👤 Creator filter: return only items by a specific creator or institution.
• 📅 Date range filter: set a start and end date to narrow results to a specific period.
• 📋 Full metadata fetch: toggle on to get per-item file lists, subject tags, and rich descriptions.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with Internet Archive data

📚 Build a public-domain corpus.

A digital humanities researcher searches for 'subject:philosophy AND mediatype:texts' to collect every public-domain philosophy book for text analysis.

🎬 Catalog open-source films.

A curator filters by collection 'opensource_movies' and media type 'movies' to generate a complete film list for a community cinema website.

📈 Track institutional uploads.

A data journalist sets creator to 'NASA' and a date range to monitor new media releases from the agency each month.

🔊 Gather audiobook metadata.

A developer restricts to collection 'librivoxaudio' and fetches full metadata to populate a podcast-style directory with chapter file URLs.

Why choose this scraper

	What you get
No API key or registration	The Internet Archive search index is public. You start the Actor and get results.
Full metadata on demand	Enable fetchDetails to pull file URLs, subject tags, and descriptions for every item.
Scales to a million items	Set maxItems and let the Actor paginate through the archive's index automatically.
Fixed flat schema	Every run returns the same columns, ready for CSV, JSON, Excel, or XML export.

How it compares

No other Store actor targets Internet Archive the same way, so the honest comparison is with the alternatives teams actually weigh.

	Internet Archive Search Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When Internet Archive changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor with a Lucene search query and optional filters for collection, media type, creator, and date range. Filters run server-side so only matching items reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
 "maxItems": 10,
 "searchQuery": "mars rover"
}

A larger pull:

{
 "maxItems": 200,
 "searchQuery": "mars rover"
}

Pricing

Pay-per-result: $0.02133 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$2.13
1,000 results	$21.33
10,000 results	$213.30

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the Internet Archive Search Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to Internet Archive through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/internet-archive-search-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your Lucene query syntax. A typo or an overly restrictive combination of filters (e.g., a collection that does not contain the chosen media type) can return zero items. Try a broad keyword first, then add filters one at a time.

The run is very slow with full metadata enabled.

Full metadata fetches one extra request per item. Reduce maxItems or disable the fetchDetails checkbox for a faster, index-only run. You can always re-run with details on a smaller subset later.

Some items are missing their file URLs even with full metadata on.

A small number of Internet Archive items have restricted or incomplete metadata. The Actor returns whatever the archive provides. If an item has no downloadable files, the file list field will be empty.

I get an error about the date format.

Use the exact format YYYY-MM-DD, for example 2020-01-31. Single-digit months and days must have a leading zero.

The Actor stopped before reaching maxItems.

The Internet Archive search index may return fewer total results than your maxItems value. The Actor collects every match the index reports. Lower your maxItems or broaden your query to capture more items.

FAQ

Question	Answer
What is a Lucene query and how do I write one?	It is the Internet Archive's search syntax. You can use simple keywords like 'mars rover' or structured queries like 'subject:photography AND mediatype:image'. The input field links to the archive's advanced search page for reference.
How do I find the right collection slug?	Visit archive.org and browse to the collection you want. The slug is the last part of the URL, for example 'nasa' in archive.org/details/nasa. Enter that slug in the Collection field.
Does this Actor download the actual files?	No. It collects metadata and file URLs. When you enable Full Metadata, you get a direct download link for each file in the item, which you can pass to a download tool.
How long does a run with full metadata take?	Fetching full metadata makes one extra request per item, so a run of 1,000 items takes a few minutes. Without it, the Actor reads only the search index and is much faster.
Can I search by date range?	Yes. Set a Date From and Date To in YYYY-MM-DD format. The Actor filters items by their publish date before returning them.
What media types can I filter by?	Texts, audio, movies, software, web, image, data, and collection. Choose one from the dropdown to restrict results to that type.
Is there a rate limit on the Internet Archive?	The archive's public search API is open, but very aggressive scraping may trigger temporary blocks. The Actor runs at a responsible pace by default.
Can I get the full description and subject tags for each item?	Yes. Enable the 'Fetch Full Metadata' checkbox. The output will include fields like description, subject, and a list of files with their URLs.
What output formats are supported?	You can export your dataset to CSV, JSON, Excel, or XML from the Apify platform after the run completes.
Can I run multiple searches in one Actor run?	Each run takes one search query. To run multiple queries, start a separate run for each or use the Apify API to trigger runs in a loop.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Internet Archive. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.