NCI GDC Cases Scraper

NCI GDC Cases Scraper

Scrape cancer case metadata from the NCI Genomic Data Commons API, filtered by project or primary site. Each case row includes its submitter ID, disease type, demographic data, and available biospecimen counts. No API key required. Export to CSV, JSON, Excel, or XML.

The NCI Genomic Data Commons (GDC) holds clinical and biospecimen metadata for tens of thousands of cancer cases across major research projects like TCGA and TARGET. Browsing the GDC Data Portal or writing manual API queries makes it hard to pull a clean, flat dataset for offline analysis. This Actor reads the GDC cases endpoint directly, lets you filter by project ID or primary site, and returns every matching case in a consistent schema you can open in any spreadsheet.

Who uses it	What they scrape NCI Genomic Data Commons for
Bioinformatics researchers	Pull all cases from a specific TCGA project for cohort discovery before requesting BAM files.
Clinical data managers	Audit case metadata completeness across projects before a data freeze.
Epidemiologists	Collect cases filtered by primary site to estimate available sample sizes for a study.
Data engineers in cancer centers	Automate nightly ingestion of new GDC case records into an institutional data lake.

What it does

This Actor collects cancer case records from the NCI GDC API by project ID or primary site, and returns each case as a flat row with its clinical identifiers, demographic fields, and biospecimen counts.

• 🔬 Project filter: supply a GDC project ID like TCGA-BRCA to limit results to one study.
• 🫁 Primary site filter: restrict cases to a specific organ or tissue site, such as Lung or Breast.
• 📏 Row cap: set a maximum number of cases per run, from 1 up to 1,000,000, so you control the output size.

Results export to CSV, JSON, Excel, or XML, or straight from the API.

What you can do with NCI Genomic Data Commons data

🧬 Build a TCGA cohort manifest.

A bioinformatician sets projectId to TCGA-LUAD, pulls 500 cases, and uses the submitter IDs to request RNA-seq alignments from the GDC Data Transfer Tool.

🫁 Survey available samples by primary site.

An epidemiologist filters by primary site Lung, collects all matching cases, and groups by disease type to estimate sample sizes for a multi-center study.

📋 Audit metadata completeness.

A data manager runs the Actor weekly for a project, checks which cases lack vital status or tumor stage, and flags them for curation.

🔄 Feed a data pipeline.

A data engineer schedules the Actor daily with no filters, dumps all GDC cases to cloud storage, and triggers downstream ETL that joins cases with files and annotations.

Why choose this scraper

	What you get
Flat schema	One row per case, ready for pandas, R, or Excel with no nested JSON to unpack.
No API key	The GDC public API requires no registration or token; the Actor calls it directly.
Large runs	Collect up to a million cases in a single run when you need the full catalog.
Deterministic output	Same filters produce the same rows every run, so you can version your cohorts.

How it compares

No other Store actor targets NCI Genomic Data Commons the same way, so the honest comparison is with the alternatives teams actually weigh.

	NCI GDC Cases Scraper	Build it in-house	By hand
Setup	Run it now, zero config	Days of engineering	None, but hours per pull
When NCI Genomic Data Commons changes	Maintained for you	You fix it	You re-learn the page
Proxies, retries, anti-bot	Built in	Your problem	Browser only
Output	Fixed JSON schema, CSV/Excel export	Whatever you build	Copy-paste
Cost	Pay per result	Engineering time	Analyst hours

Configure the run

Drive the Actor with a project ID, a primary site, or both together, and set a row cap so only the number of cases you need reaches your dataset. The Input tab lists every parameter.

A first run with the defaults:

{
  "maxItems": 10
}

A larger pull:

{
  "maxItems": 200
}

Pricing

Pay-per-result: $0.0085 per result collected. You pay only for the results written to your dataset.

Results collected	Approximate cost
100 results	$0.85
1,000 results	$8.50
10,000 results	$85.00

New Apify accounts start with $5 in free credit.

Free users

Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.

Run it

1. Create a free Apify account with $5 in credit.
2. Open the NCI GDC Cases Scraper.
3. Set your inputs and any filters, then click Start.
4. Export the results as CSV, Excel, JSON, or XML from the Dataset tab.

Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.

Use with AI agents (MCP)

Give an AI agent live access to NCI Genomic Data Commons through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/nci-gdc-cases-scraper"

Then prompt it in plain language to run the scraper and read back the results.

Troubleshooting

Why am I getting zero results?

Check that your project ID or primary site value matches the GDC naming exactly. Project IDs are case-sensitive. Try running without filters first to confirm the API is reachable.

The run timed out.

Lower the maxItems value. The GDC API paginates large result sets, and very high caps can take longer than the default timeout. Start with 1,000 and increase gradually.

Some fields are empty in my output.

Not every case has every field populated. Missing clinical data is common in real-world cancer registries. The Actor returns whatever the GDC provides for each case.

I got fewer rows than my maxItems setting.

The Actor returns only the cases that match your filters. If the GDC has fewer matching cases than your cap, you will receive all of them.

The primary site filter is not working as I expected.

Primary site values in the GDC are standardized terms like 'Bronchus and lung' rather than 'Lung'. Check the GDC Data Portal for the exact term used in your cases of interest.

FAQ

Question	Answer
Do I need a GDC account or API key?	No. The Actor calls the public GDC API, which requires no authentication for case metadata queries.
What is a project ID?	A project ID is a GDC program identifier like TCGA-BRCA or TARGET-AML. You can find them on the GDC Data Portal projects page or in the GDC API documentation.
Can I filter by disease type instead of primary site?	The input schema exposes project ID and primary site filters. Disease type is returned in the output rows, so you can filter it downstream in your analysis tool.
How many cases can I collect in one run?	You can set the maximum up to 1,000,000 cases. The GDC contains fewer than that total, so the cap mainly controls how many rows you want to receive.
Does this Actor download genomic files?	No. It collects case-level metadata only. For file downloads, use the GDC Data Transfer Tool or the GDC Files API.
What fields does each case row contain?	Each row includes the submitter ID, project ID, primary site, disease type, gender, race, ethnicity, vital status, and counts of available aliquots and samples. The exact fields appear in the sample output on the Actor's page.
Can I run this on a schedule?	Yes. Apify schedules let you run the Actor hourly, daily, or weekly. New cases added to the GDC will appear in your next run automatically.
What export formats are supported?	You can export your dataset to CSV, JSON, Excel, XML, or RSS from the Apify storage tab.
Is the output deterministic?	Yes. The same project ID and primary site filters return the same cases every run, assuming the GDC has not added or removed cases.
Can I combine project ID and primary site filters?	Yes. When both are set, the Actor returns cases that match both conditions, which is useful for narrowing a large project to one tissue type.

Related actors

Browse the full ParseForge collection for more scrapers.

🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.

⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by National Cancer Institute. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.