Website Content Crawler

Website Content Crawler

Crawl any website and extract clean text content for your AI models, LLM pipelines, and vector databases. Every page returns its full Markdown body, cleaned HTML, and metadata. No sitemap needed. Export to CSV, JSON, Excel, or XML.

Building a RAG pipeline or fine-tuning an LLM requires clean, structured text from websites, but writing a custom crawler for each site is slow and brittle. The Website Content Crawler reads public pages from any starting URL, follows links up to a set depth, and returns the full text content of each page in a consistent format. It handles HTML cleaning, Markdown conversion, and domain filtering so your data is ready for embeddings or training.

Who uses it	What they scrape Website Content Crawler for
AI engineers	Building a knowledge base for a RAG application or fine-tuning dataset.
Data scientists	Collecting a corpus of text from a specific domain for NLP analysis.
SEO specialists	Extracting all body text from a competitor's site to audit content strategy.
Developers	Feeding a vector database with Markdown-formatted pages from documentation sites.

What it does

This Actor crawls websites from a list of starting URLs and returns each page's full text content, cleaned HTML, and metadata as a flat row.

• 📝 Markdown output: every page is converted to clean Markdown, ready for LLM context windows.
• 🧹 HTML cleaning: navigation, scripts, and boilerplate are stripped, leaving only the main content.
• 🔗 Link crawling: follows internal links up to a configurable depth to discover all pages on a site.
• 🌐 Domain control: restrict the crawl to the same domain, or optionally include subdomains.
• 🎯 URL filtering: include or exclude pages with regex patterns to target only the content you need.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with Website Content Crawler data

🤖 Build a RAG knowledge base.

An AI engineer crawls a product documentation site to extract every page as Markdown, then loads the dataset into a vector store for a support chatbot.

📊 Audit a competitor's content.

An SEO specialist crawls a competitor's blog, filtering URLs to only article pages, and extracts the full body text to analyze topic coverage and keyword density.

🧠 Create a fine-tuning dataset.

A data scientist crawls a public forum, limits the depth to thread pages, and collects a corpus of question-answer pairs for instruction-tuning a model.

📚 Archive documentation for offline use.

A developer crawls an open-source library's docs site, stays on the same domain, and exports the full text to a single JSON file for local search.

Why choose this scraper

	What you get
Clean text extraction	Full page body in Markdown and cleaned HTML, stripped of navigation and scripts.
Configurable crawl scope	Set max depth, page limit, and domain restrictions to control the crawl size.
Regex URL filtering	Include or exclude specific URL patterns to target only relevant sections of a site.
AI-ready output	Output format is designed for direct ingestion into LLMs, vector databases, and LangChain.

How it compares

This Actor focuses on extracting clean, AI-ready text from any website, while the other Website Content Crawler listing offers similar Markdown extraction and LangChain integration.

Feature	ParseForge	Website Content Crawler
Full page text in Markdown	Yes	Yes
Cleaned HTML output	Yes	Yes
Regex URL filtering	Yes	Not listed
Subdomain inclusion control	Yes	Not listed
LangChain integration	Not listed	Yes

Configure the run

Drive the Actor from one or more starting URLs, set a crawl depth and page limit, and apply domain and regex filters so only matching pages reach your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 10,
  "sameDomain": true,
  "includeSubdomains": true
}

A larger pull:

{
  "startUrls": [
    {
      "url": "https://apify.com/docs"
    }
  ],
  "maxDepth": 2,
  "maxItems": 200,
  "sameDomain": true,
  "includeSubdomains": true
}

Pricing

Pay-per-result: $0.025 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$2.50
1,000 results	$25.00
10,000 results	$250.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the Website Content Crawler.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to Website Content Crawler through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/website-content-crawler"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your Include URL Patterns regex. If it is too restrictive, no discovered URLs will match. Try removing the pattern or testing it against a known page URL. Also verify the starting URL is accessible and returns a 200 status.

The crawl is taking too long.

Reduce the maxDepth or maxItems to limit the crawl scope. You can also add Exclude URL Patterns to skip large sections like archives or tag pages that generate many low-value URLs.

The extracted text contains navigation menus and footer links.

The Actor cleans common HTML boilerplate, but some sites use unusual markup. The cleaned HTML field removes scripts and styles, but complex layouts may still include some chrome. The Markdown field focuses on the main content area.

I am getting pages from other domains even with 'Stay on Same Domain' enabled.

Check if you have 'Include Subdomains' enabled. A subdomain like blog.example.com is treated as a different domain from www.example.com. Disable subdomain inclusion to restrict to the exact starting domain.

The Markdown output is missing formatting like headings or lists.

The Markdown conversion preserves semantic HTML elements like h1-h6, ul, ol, and p tags. If a site uses divs with CSS classes instead of semantic tags, the structure may be lost. The cleaned HTML field will still contain the original structure.

FAQ

Question	Answer
What format is the extracted text in?	Each page returns a Markdown version of the main content and a cleaned HTML version. The Markdown is ready for direct use with LLMs and vector databases.
Can I limit the crawl to only one section of a website?	Yes. Use the Include URL Patterns field with a regex like /docs/.* to crawl only pages under the /docs/ path, or use Exclude URL Patterns to skip sections like /blog/.
Does it handle JavaScript-rendered content?	This Actor fetches the initial HTML of each page. For sites that load content dynamically with JavaScript, you may need a headless browser crawler instead.
How many pages can I crawl in one run?	You can set the maximum pages up to 1,000,000 per run. The actual number depends on your max depth, domain restrictions, and URL filters.
Will it follow links to external websites?	Not by default. The 'Stay on Same Domain' option is on by default, which restricts the crawl to the starting domain. You can disable it, but the Actor will still only follow links it discovers from the pages it crawls.
Can I use this to scrape a single page?	Yes. Set maxDepth to 0 and maxItems to 1, and it will extract only the content from the starting URL without following any links.
What is the output schema?	Each row in the dataset includes the page URL, the full Markdown body, the cleaned HTML, the page title, and metadata like the crawl depth and load timestamp.
Does it download images or PDFs?	No, this Actor extracts text content only. It does not download binary files, images, or PDFs.
Can I crawl multiple websites in one run?	Yes. Add multiple starting URLs from different domains. If 'Stay on Same Domain' is on, each starting URL will spawn a separate crawl restricted to its own domain.
Is this suitable for a production RAG pipeline?	Yes. The Markdown output is designed for direct ingestion into vector databases like Pinecone or Weaviate, and the Actor integrates with LangChain for loading documents.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Apify. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.