RAG Web Browser

RAG Web Browser

Turn any web page or search query into clean, structured data for your RAG pipelines and LLMs. Fetch content as markdown, text, or HTML, with optional outbound links. No sitemaps or complex crawling rules required.

Building a RAG application or feeding an LLM requires clean, structured content, but websites are messy. This Actor fetches web pages from search queries or direct URLs and returns their content in a consistent, machine-readable format. It handles the extraction so you get markdown, plain text, or raw HTML ready for your vector database.

Who uses it	What they scrape Web for
LLM developers	Building a knowledge base for a RAG chatbot by fetching and cleaning documentation pages.
Data engineers	Ingesting web content into a data pipeline for downstream processing and analysis.
SEO specialists	Extracting clean text content from competitor pages for content gap analysis.
Researchers	Collecting and formatting articles from multiple sources for literature reviews.

What it does

This Actor fetches web pages from a search query or a list of URLs and returns each page's content as a flat row with markdown, text, and optional HTML and outbound links.

• 🔍 Search-based fetching: Provide a search query and the Actor fetches the top results, up to 100 pages.
• 🔗 Direct URL fetching: Paste a list of specific URLs to fetch their content directly, bypassing search.
• 📝 Multiple output formats: Get content as markdown, plain text, and HTML in a single run for maximum flexibility.
• 🔗 Outbound link extraction: Optionally collect all links from each page to map the information neighborhood.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with Web data

🤖 Populate a RAG knowledge base.

An LLM developer runs the Actor on a set of documentation URLs to fetch clean markdown, then chunks and embeds the results into a vector store for a support chatbot.

📊 Aggregate research from search results.

A market analyst enters a search query about a new industry trend, fetches the top 20 articles as plain text, and runs them through a summarization model.

🔗 Map a website's link graph.

An SEO specialist fetches a competitor's homepage with the 'Include Links' option enabled to extract all outbound links and understand their partnership strategy.

Why choose this scraper

	What you get
Clean, structured output	Every page is returned as markdown, text, and optionally HTML, ready for chunking and embedding.
No complex setup	Start with a simple search query or a list of URLs. No sitemaps, CSS selectors, or crawling rules needed.
Built for RAG	Designed specifically to feed Retrieval-Augmented Generation pipelines with consistently formatted documents.

How it compares

No other Store actor targets Web the same way, so the honest comparison is with the alternatives teams actually weigh.

	RAG Web Browser	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When Web changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor with a search query or a list of direct URLs, and control the number of pages fetched and the output formats returned. The Input tab lists every parameter.

A first run with the defaults:

{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 10,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}

A larger pull:

{
  "query": "Anthropic Claude API pricing",
  "maxResults": 10,
  "maxItems": 200,
  "outputFormats": [
    "markdown",
    "text"
  ],
  "includeLinks": false
}

Pricing

Pay-per-result: $0.012 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$1.20
1,000 results	$12.00
10,000 results	$120.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the RAG Web Browser.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to Web through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/rag-web-browser"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why is the returned text or markdown empty or incomplete?

The page might rely on JavaScript to load content. The Actor fetches the static HTML. Try viewing the page source in your browser to see what is available without JavaScript. If the content is missing, you may need a headless browser solution.

I'm getting fewer results than my 'Max Search Results' setting.

The search engine may return fewer results than requested for your query. Try a more general search query or use direct URLs if you know the specific pages you need.

Why am I hitting the 'Maximum results' limit as a free user?

Free accounts on Apify have a limit of 10 items per run for this Actor. To process more items, upgrade to a paid plan and increase the 'Maximum results' value.

The HTML output looks different from what I see in my browser.

The Actor returns the raw HTML source code. Your browser applies CSS and JavaScript to render the visual page. The source code is the correct input for machine processing and content extraction.

FAQ

Question	Answer
What is the difference between using a search query and direct URLs?	A search query lets you find and fetch content dynamically based on a topic. Direct URLs let you specify exactly which pages to fetch, which is useful when you already know the target pages.
What output formats are supported?	The Actor always returns markdown. You can optionally include plain text and the raw HTML. Select the formats you need in the 'Output Formats' input field.
How many pages can I fetch in one run?	Free users are limited to 10 items. Paid users can fetch up to 1,000,000 items by adjusting the 'Maximum results' setting.
Does this Actor execute JavaScript on the pages?	The Actor fetches the initial HTML content. For pages that rely heavily on client-side JavaScript to render content, the returned HTML may not include dynamically loaded elements.
Can I use this to scrape a whole website?	This Actor is designed for fetching a specific set of pages from a search or a URL list. For crawling an entire website by following links, use the Website Content Crawler.
What is the 'Include Links' option for?	When enabled, the Actor extracts all outbound links found on each page and includes them in the output. This is useful for analyzing a page's link profile or discovering related resources.
How do I use the output in my RAG pipeline?	The markdown or text output is clean and ready for chunking. You can feed it directly into a text splitter and then into an embedding model to create vectors for your vector database.
Is there a way to filter the search results before fetching?	The Actor fetches the top N results from the search query as specified in 'Max Search Results'. It does not apply content-based filters before fetching, but you can process the output afterward.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by the respective website owners. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.