Panama Open Data Scraper - Datos Abiertos

Panama Open Data Scraper - Datos Abiertos CKAN API

Extract all 5,661 datasets from datosabiertos.gob.pa, Panama's national open data portal, in a single run. Every row carries 38 fields: title, description, publishing ministry, named contacts with emails, licence, declared update frequency, tags, categories, every downloadable file with its format and size, and a computed freshness age. No login, no API key, no registration. Exports to CSV, JSON, Excel and XML.

Panama publishes the portal as a CKAN instance, and the portal search page will only show you 20 datasets at a time behind a paginated UI. There is no bulk download button and no official export. This scraper walks the whole catalogue through the CKAN index, follows every publisher, category and tag directory, and reaches into the portal DataStore to give you the real column schema and exact row count behind each file, which the website never shows you at all.

Who uses it	What they scrape Panama open data for
Journalists and transparency researchers	Public payroll (planilla) records across ministries, salary by position and contract status
Economists and analysts	Migration flows, port and maritime traffic, banking and budget execution series
Compliance and due diligence teams	Which government bodies publish what, who maintains it and how current it is
Civic tech and data engineers	A complete portal mirror with file URLs and DataStore schemas to build pipelines on
Academics and NGOs	Health, education and social development statistics with citable licences and provenance

What it does

• 📚 Whole catalogue - all 5,661 datasets with 38 fields per row, or any slice of them
• 📎 Every downloadable file - format, MIME type, size, hash and a direct download URL
• 🗄️ DataStore tables - exact row counts, column names and PostgreSQL types for the 48% of files loaded into the portal database, plus real sample rows
• 🏛️ 114 publishing bodies - ministries, municipalities, courts, universities and authorities, with live dataset counts
• 📇 Publisher contacts - deduplicated names, emails and email domains for the people who maintain each dataset
• 🏷️ Full directories - 3,777 tags, 18 categories, 15 licences and the declared update frequency breakdown
• 🔍 14 filters - by organization, category, tag, format, licence, update cadence, date, file count and free text, all applied by the portal index before anything is billed
• ✅ Quality scoring - a 0 to 100 metadata completeness score with an A to F grade and the exact list of what the publisher left out
• 🔗 Link checking - HEAD every file to prove it actually downloads

What you can do with Panama open data

Track public payroll across the whole state. Panama publishes planilla (payroll) datasets for ministries, authorities and municipalities, and 101 datasets carry the planilla tag. Pull them all with their DataStore tables and you get position, name, role, monthly salary, start date and contract status as queryable rows rather than hundreds of separate spreadsheets.

Monitor what government actually publishes. Every row carries a declared update frequency and a computed age. Filter to the 1,284 datasets that promise monthly updates, then sort by days since modified, and the ones that broke their own promise surface immediately.

Build a research corpus with clean provenance. 5,640 of the 5,661 datasets carry an open licence, 3,881 of them CC-BY and 1,757 CC-Zero. Each row keeps the licence id, title and URL alongside the publisher and the source URL, so citation and reuse terms travel with the data.

Map the machinery of the Panamanian state. The organization directory returns 114 bodies with live dataset counts, descriptions and logos, from Caja de Seguro Social with 688 datasets to single-dataset municipalities. It is a usable inventory of who holds which public records.

Why choose this scraper

	What you get
Coverage	The full 5,661 dataset catalogue, not the 20 per page the website shows
Depth	10 output collections, from datasets and files down to DataStore column types and publisher contacts
Beyond the website	Exact row counts and PostgreSQL column types the portal UI never displays
Filters that are real	Every one of the 14 filters was measured against the unfiltered total before shipping
Honest sorting	Only sort orders the portal actually honours are offered; the ones that silently fall back to relevance are not
Speed and cost	3,000 dataset rows with nested file lists in 28 seconds, at 76 MB of memory

How it compares

The generic CKAN exporters on Apify point at any portal and hand back whatever the API returns, which means two events, raw upstream keys and no knowledge of the site. This actor is built against Panama specifically: it knows the portal's custom frecuency field (misspelled upstream, so a generic tool filtering on frequency silently returns nothing), it knows which endpoint the site's F5 firewall rejects, and it ships verified dropdowns of all 71 active publishers and 17 categories rather than making you guess slugs. It costs more per row than a generic exporter and it returns data those tools cannot reach.

	This actor	Generic CKAN exporters
Billed events	19	2
Output collections	10	1
DataStore schema and row counts	Yes	No
Verified publisher and category dropdowns	Yes, 71 and 17	No, free text slugs
Panama frecuency filter	Yes	No
Price per 1,000 rows	$7	About $2

What a Panama dataset looks like

{
  "id": "1aa157e4-275c-4287-ad6f-214effc96566",
  "name": "proc-admon-discapacidad-2026",
  "title": "Proc. Admon.- Discapacidad 2026",
  "url": "https://www.datosabiertos.gob.pa/dataset/proc-admon-discapacidad-2026",
  "notes": "Discapacidad del Primer Interviniente en Casos de Mediación Comunitaria",
  "notesLength": 71,
  "organizationName": "procuraduria-de-la-administracion",
  "organizationTitle": "Procuraduria de la Administración",
  "organizationUrl": "https://www.datosabiertos.gob.pa/organization/procuraduria-de-la-administracion",
  "author": "Raúl Alvarado",
  "authorEmail": "ralvarado@procuraduria-admon.gob.pa",
  "maintainer": "Ambar Bonilla",
  "maintainerEmail": "ambarbonilla@procuraduria-admon.gob.pa",
  "sourceUrl": "https://www.procuraduria-admon.gob.pa/",
  "licenseId": "cc-zero",
  "licenseTitle": "Creative Commons CCZero",
  "isOpenLicense": "Yes",
  "version": "1.0",
  "updateFrequency": "Semestral",
  "metadataCreated": "2026-07-19T22:55:03.324552",
  "metadataModified": "2026-08-27T14:31:21.503464",
  "daysSinceModified": 0,
  "isStale": "No",
  "numResources": 3,
  "numTags": 1,
  "formats": ["CSV", "PDF", "XLSX"],
  "totalSizeBytes": 143752,
  "totalSizeMb": 0.137,
  "hasDatastore": "No",
  "tags": ["Discapacidad"],
  "categories": ["Estadísticas de Gobierno", "Finanzas de Gobierno"],
  "scrapedAt": "2026-08-27T19:03:00.512Z"
}

Configure the run

Pick a collection under What to scrape, then narrow it. The dropdowns for publishers, categories, formats, licences and update frequency are filled with the portal's live values and show the dataset count next to each one, so you can see what a filter is worth before you run it. Filters are applied by the portal search index, so narrowing costs nothing extra. You can also paste dataset URLs straight into Dataset URLs to scrape exactly those and skip the search entirely.

Every payroll dataset updated monthly, with its file list and a quality score:

{
  "dataset": "datasets",
  "searchQuery": "planilla",
  "frequencies": ["Mensual"],
  "sortBy": "modified-desc",
  "includeDatasetResources": true,
  "includeQualityScore": true,
  "maxItems": 500
}

Every queryable DataStore table with its schema and 20 real rows:

{
  "dataset": "datastore-tables",
  "includeDatastorePreview": true,
  "datastorePreviewRows": 20,
  "maxItems": 1000
}

The complete publisher directory with profiles:

{
  "dataset": "organizations",
  "includeOrganizationProfile": true,
  "maxItems": 200
}

Free users

Runs on the Apify free plan return the first 10 rows as a preview, which is enough to see the full field shape of every collection. Upgrade your plan to lift the cap to 1,000,000 rows.

Run it

1. Create a free Apify account and claim the $5 free credit.
2. Open the actor and pick a collection under What to scrape. Leave the defaults to pull the 10 most recently updated datasets.
3. Add filters if you want a slice, and tick any optional blocks you need. Set Max Items.
4. Click Start, then export the dataset as CSV, JSON, Excel or XML, or pull it from the Apify API.

Use with AI agents (MCP)

claude mcp add apify --transport sse https://mcp.apify.com/sse?actors=parseforge/panama-open-data-scraper

Then ask in plain language:

• "Find every Panama government payroll dataset updated monthly and show me which ministries publish them."
• "List the Panamanian public bodies with the most open datasets and how fresh their data is."
• "Get the column schema and 10 sample rows for the Metro de Panama payroll table."

Troubleshooting

No results at all. The most common cause is stacking filters that do not overlap, for example a category and a tag that never appear together. Clear the filters, run once, then add them back one at a time. The log prints how many datasets matched before any row is written.

Fewer rows than I asked for. Your filter matched fewer datasets than Max Items, or you are on the free plan and capped at 10. The log states the match count and warns explicitly when the free cap applies. The Only datasets with a queryable DataStore table filter is applied after the search, so it can also reduce the count below the match total.

A field comes back "Not Disclosed" or "N/A". The publisher left it empty. Panama's portal does not require a version, a source URL or an update frequency, so those are genuinely missing rather than missed by the scraper. Switch on the quality score block to see exactly which fields each publisher omitted.

DataStore fields are empty. Only about 48% of files are loaded into the portal DataStore; the rest are downloadable files only. Tick Only datasets with a queryable DataStore table to keep just the ones that have it.

The run is slow. The file download check sends one HEAD request per file and is the slowest option by a wide margin. Turn it off unless you specifically need dead-link detection, and it will run at full speed.

FAQ

Question	Answer
Do I need an API key or a login?	No. The portal is fully anonymous and so is this actor.
How many datasets are there?	5,661 at the time of writing, across 71 active publishing bodies.
How fast is it?	3,000 dataset rows with nested file lists in 28 seconds on the platform.
Can I get the actual data inside the files?	Yes, for the 48% loaded into the DataStore. Use the DataStore sample rows block, or take the direct download URL from any file row.
What is the frecuency field?	The portal's own publication cadence field. It is misspelled upstream, which is why generic CKAN tools filtering on frequency return nothing.
Is the data in Spanish?	Yes. Titles, descriptions and category names are Spanish, as published. Field names in the output are English.
Can I scrape specific datasets only?	Yes. Paste their portal URLs into Dataset URLs and the search is skipped entirely.
Are the files reliable?	In a 90 URL sample every file downloaded successfully. The portal self hosts its uploads. Use the file download check to verify a specific set.
Can I run it on a schedule?	Yes, with Apify Schedules. Sort by recently updated and filter by date to catch only what changed.
What licence is the data under?	5,640 of 5,661 datasets carry an open licence, mostly CC-BY and CC-Zero. The exact licence travels on every row.

Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

Disclaimer: this is an unofficial tool and is not affiliated with, endorsed by, or connected to the Government of Panama, the Autoridad Nacional para la Innovación Gubernamental, or datosabiertos.gob.pa. It collects only data that is already published publicly on the portal, under the open licences the publishers attached to it. Publisher contact details are official institutional contacts published by the government as dataset metadata, not private personal data. Review GDPR, CCPA and PIPL obligations for your own use case before processing or redistributing.