Australia Open Data (data.gov.au) Scraper

Australia Open Data (data.gov.au) Scraper

Scrape Australia's open data catalogue for dataset metadata, resources, and publisher details, up to a million per run. Walk the full catalogue or fetch a single dataset by ID. Filter by keyword. Export to CSV, JSON, Excel, or XML.

Finding public sector data across Australia's official open data portal means clicking through pages or wrestling with the CKAN API. This scraper reads the catalogue directly, collecting dataset titles, descriptions, publishers, licenses, tags, and resource links in one flat schema. No API key, no registration.

Who uses it	What they scrape data.gov.au for
Data journalists	Discovering newly published government datasets for a story on public spending or environmental trends.
Policy analysts	Building a searchable index of all federal and state open data releases for a policy review.
Open data advocates	Auditing the catalogue for dataset completeness, license compliance, or update frequency.
Researchers	Gathering a corpus of dataset metadata to study open data publishing patterns across Australian agencies.

What it does

This Actor collects dataset metadata from data.gov.au by walking the full catalogue or fetching a single dataset by ID, and returns each dataset as a flat row with its resources, publisher, and tags.

• 📋 Catalog mode: walks every dataset in the data.gov.au catalogue, respecting the max items limit you set.
• 🎯 Dataset mode: fetches a single dataset by its UUID or slug, returning its full metadata and resource list.
• 🔍 Keyword filter: narrows the catalog walk to dataset IDs containing your search term, so only relevant records reach your dataset.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with data.gov.au data

📊 Build a searchable open data index.

A data portal manager runs the catalog mode weekly to populate an internal search tool with every dataset title, publisher, and tag from data.gov.au.

🔎 Monitor new dataset releases.

A policy researcher runs the scraper with a keyword filter on 'water' to track new water-related datasets published by Australian agencies each month.

📁 Audit dataset metadata quality.

An open data advocate exports the full catalogue to check which agencies are missing license information or have stale update dates.

🗂️ Collect resource links for a single dataset.

A data engineer uses dataset mode with a known UUID to pull all download URLs and format details for a specific climate dataset before ingestion.

Why choose this scraper

	What you get
Full catalogue access	Walk every public dataset on data.gov.au without paginating manually.
Flat row per dataset	Each dataset returns as one row with its title, description, publisher, license, tags, and resources.
No API key needed	Reads the public CKAN catalogue endpoints directly. No registration, no rate-limit headaches.
Flexible output	Export to CSV, JSON, Excel, or XML for direct use in spreadsheets, databases, or notebooks.

How it compares

This scraper focuses on raw catalogue metadata export from data.gov.au, while the alternatives below add value layers like cross-dataset joins or target a specific register on the same portal.

Feature	ParseForge	Australia Open Data Studio	Australia Open Data Scraper
Full catalogue walk	Yes	Not listed	Yes
Single dataset fetch by ID	Yes	Not listed	Not listed
Keyword search filter	Yes	Not listed	Yes
Cross-dataset join presets	Not listed	Yes	Not listed
Resource download URLs	Yes	Not listed	Yes
No API key required	Yes	Not listed	Not listed

Configure the run

Drive the Actor in catalog mode to walk all datasets, or switch to dataset mode and supply a single ID. An optional keyword filter narrows the catalog walk to matching dataset IDs. The Input tab lists every parameter.

A first run with the defaults:

{
 "maxItems": 10,
 "mode": "catalog"
}

A larger pull:

{
 "maxItems": 200,
 "mode": "catalog"
}

Pricing

Pay-per-result: $0.02 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$2.00
1,000 results	$20.00
10,000 results	$200.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the Australia Open Data (data.gov.au) Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to data.gov.au through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/australia-data-gov-au-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting no results?

Check your search query spelling. The filter matches dataset IDs, not titles or descriptions. Try a shorter substring or leave it blank to confirm the catalogue is reachable.

The run stopped before collecting all datasets.

Check the 'Maximum datasets' setting. If it is lower than the total catalogue size, the scraper stops when it hits that number. Increase it and rerun.

Dataset mode returns an error.

Verify the dataset ID you entered. It must be the exact UUID or slug from the dataset's URL on data.gov.au. The field is case-sensitive.

The run is taking a long time.

Walking the full catalogue with a high max items value takes time because each dataset requires a separate API call. Use a keyword filter to narrow the scope, or reduce the max items for a quicker test.

Some fields are empty in my output.

Not every dataset on data.gov.au has all metadata fields populated. Empty cells mean the publisher left that field blank. This is expected behaviour.

FAQ

Question	Answer
Does this scraper download the actual dataset files?	No. It collects dataset metadata: titles, descriptions, publishers, licenses, tags, and resource URLs. You get the links to the files, not the files themselves.
Do I need an API key or login for data.gov.au?	No. The scraper reads the public CKAN catalogue API that data.gov.au exposes without authentication. No registration is required.
How many datasets can I scrape in one run?	You set the maximum with the 'Maximum datasets' field, up to 1,000,000 per run. The catalogue contains fewer than that, so setting a high number walks everything.
Can I filter by publisher or date range?	The current keyword filter matches against dataset IDs. For publisher or date filtering, export the full catalogue and filter in your spreadsheet or script.
What is a dataset ID?	It is the UUID or URL slug that identifies a single dataset on data.gov.au. You can find it in the dataset page URL, e.g. /dataset/ds-dga-xxxxx.
Does this cover state-level open data portals?	This scraper targets the federal data.gov.au catalogue. State portals like data.nsw.gov.au or data.vic.gov.au run on separate CKAN instances and are not included.
What output formats are supported?	You can export your results to CSV, JSON, Excel, or XML directly from the Apify dataset tab after the run completes.
How often is the data.gov.au catalogue updated?	The catalogue is updated continuously as agencies publish or modify datasets. Run the scraper on a schedule to capture changes.
Can I get the tags and groups for each dataset?	Yes. Each row includes the dataset's tags and the groups (categories) it belongs to on data.gov.au.
What happens if I leave the search query blank in catalog mode?	The scraper walks every dataset in the catalogue, starting from the first, until it hits your max items limit.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Australian Government. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.