Genomics & Variants Pack Scraper

🧬 Genomics & Variants Pack Scraper

🚀 Export variant records, clinical significance, genomic coordinates and ontology terms from dbSNP, ClinVar, UCSC and EBI OLS in seconds.

Type a gene symbol such as BRCA1 or an rsID such as rs334 and the Actor queries 4 public genomics databases in one run: NCBI dbSNP, NCBI ClinVar, the UCSC Genome Browser and the EBI Ontology Lookup Service. Every match lands in one flat, 22-field row, ready for CSV, JSON, Excel or XML. A BRCA1 search alone reaches 44,306 dbSNP records, 16,062 ClinVar records and 594 OLS terms.

📋 What the Genomics & Variants Pack Scraper does

Pulling variant data by hand means four portals, four query syntaxes and a spreadsheet to stitch them together. This Actor sends the same search term to each selected source, throttles NCBI to its 3 requests per second limit so no source is dropped, and collects records round-robin across sources until your item limit is reached.

• 🧬 dbSNP variants: rsID, GRCh38 chromosome and position, gene symbols, variant class, functional consequence, HGVS notation and the first reported allele frequency.
• 🧪 ClinVar assertions: germline classification (pathogenic, benign, conflicting and so on), review status, coordinates with start and stop, variant type, associated traits and VCV accession.
• 🗺️ UCSC coordinates: gene and transcript matches with chromosome, start and end on hg38, hg19, mm39, mm10, rn7 or danRer11, plus the track they come from.
• 📖 OLS ontology terms: matching terms from HGNC, NCIT, EFO, MONDO and other ontologies with description, synonyms and IRI.
• 🔁 One schema for all four: identical column order in every row; values a source cannot provide are marked N/A (not applicable) or Not Disclosed (missing for that record), never left blank.

Who uses it	What they collect
Clinical geneticists	Every ClinVar classification and review status for the variants of a gene.
Bioinformatics engineers	A local evidence cache of rsIDs, coordinates and consequences for annotation pipelines.
Pharma R&D teams	Allele frequencies and functional annotations linked to a target gene.
Academic researchers	Ontology mappings and gene coordinates for systematic reviews.

🎬 Full Demo (🚧 Coming soon)

A short walkthrough video is in production.

📊 Output

Each row is one variant, genomic feature or ontology term. Fields, in dataset order:

Field	Description
🌐 source	dbsnp, clinvar, ucsc or ols
🆔 id	rsID, VCV accession, UCSC track/match id or ontology term id
📌 title	Human-readable name of the record
🔗 url	Link to the record on the source website
🏷 recordType	variant, genomic feature or ontology term
🧬 chromosome	Chromosome without the chr prefix
📍 start / end	Coordinates on the record's assembly
🗺 assembly	GRCh38, hg38, hg19 and so on
🧪 geneSymbol	Gene symbol(s), comma separated
🔬 variantType	snv, del, ins, copy number loss and so on
⚕ clinicalSignificance	ClinVar germline classification or dbSNP clinical significance
⭐ reviewStatus	ClinVar review status
🧩 consequence	Functional or molecular consequence
📊 alleleFrequency	First reported allele frequency with its study
📝 description	HGVS notation, cDNA change, feature or term description
📚 collection	dbSNP build, ClinVar, UCSC track or ontology prefix
🔖 accession	RefSeq accession, VCV version, transcript id or term IRI
🩺 traits	ClinVar trait names (array)
🔁 aliases	SPDI, ClinVar aliases or ontology synonyms (array)
🕒 scrapedAt	ISO timestamp of the collection
❌ error	null on success; error rows carry only this field and are never charged

Three real records from a BRCA1 run:

[
  {
    "source": "dbsnp",
    "id": "rs2552282559",
    "title": "rs2552282559",
    "url": "https://www.ncbi.nlm.nih.gov/snp/rs2552282559",
    "recordType": "variant",
    "chromosome": "17",
    "start": "43127349",
    "end": "43127349",
    "assembly": "GRCh38",
    "geneSymbol": "BRCA1, NBR2",
    "variantType": "snv",
    "clinicalSignificance": "Not Disclosed",
    "reviewStatus": "N/A",
    "consequence": "upstream_transcript_variant, 2KB_upstream_variant, intron_variant",
    "alleleFrequency": "C=0.00001/1 (TOMMO)",
    "description": "NC_000017.11:g.43127349G>C, NC_000017.10:g.41279366G>C",
    "collection": "dbSNP build 157",
    "accession": "NC_000017.11",
    "traits": [],
    "aliases": ["NC_000017.11:43127348:G:C"],
    "scrapedAt": "2026-09-08T01:10:39.899Z",
    "error": null
  },
  {
    "source": "clinvar",
    "id": "VCV004886868",
    "title": "GRCh38/hg38 17q21.31(chr17:43057598-43068066)x1",
    "url": "https://www.ncbi.nlm.nih.gov/clinvar/variation/4886868/",
    "recordType": "variant",
    "chromosome": "17",
    "start": "43057598",
    "end": "43068066",
    "assembly": "GRCh38",
    "geneSymbol": "BRCA1",
    "variantType": "copy number loss",
    "clinicalSignificance": "Likely pathogenic",
    "reviewStatus": "no assertion criteria provided",
    "consequence": "Not Disclosed",
    "alleleFrequency": "Not Disclosed",
    "description": "GRCh38/hg38 17q21.31(chr17:43057598-43068066)x1",
    "collection": "ClinVar",
    "accession": "VCV004886868.1",
    "traits": ["Breast-ovarian cancer, familial, susceptibility to, 1"],
    "aliases": [],
    "scrapedAt": "2026-09-08T01:10:40.664Z",
    "error": null
  },
  {
    "source": "ucsc",
    "id": "mane/ENST00000357654.9",
    "title": "BRCA1 (NM_007294.4 / ENST00000357654.9)",
    "url": "https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr17%3A43044295-43125364",
    "recordType": "genomic feature",
    "chromosome": "17",
    "start": "43044295",
    "end": "43125364",
    "assembly": "hg38",
    "geneSymbol": "BRCA1",
    "variantType": "N/A",
    "clinicalSignificance": "N/A",
    "reviewStatus": "N/A",
    "consequence": "N/A",
    "alleleFrequency": "N/A",
    "description": "breast cancer type 1 susceptibility protein isoform 1",
    "collection": "MANE Select Plus Clinical: Representative transcript from RefSeq & GENCODE",
    "accession": "ENST00000357654.9",
    "traits": [],
    "aliases": [],
    "scrapedAt": "2026-09-08T01:10:43.239Z",
    "error": null
  }
]

✨ Why choose this Actor

	What you get
One search, four databases	dbSNP, ClinVar, UCSC and OLS answered from a single term list.
Rate-limit safe	NCBI calls are serialised and retried with backoff, so ClinVar is never dropped for hitting the 3 requests per second ceiling.
Exact item counts	Records are collected round-robin across sources, so Max Items 10 means 10 rows, not 9 plus an error.
No API keys	All four sources are open; nothing to register.
Six assemblies	UCSC lookups on human, mouse, rat and zebrafish reference genomes.
Honest blanks	N/A and Not Disclosed instead of empty cells, so filters in Excel behave.

📈 How it compares to alternatives

Single-source gene scrapers on the Store return gene identifiers from one database. This Actor works at the variant level and joins four sources.

Feature	This Actor	Single-source gene scrapers
dbSNP variants with coordinates and frequency	Yes	No
ClinVar classification and review status	Yes	No
UCSC coordinates on 6 assemblies	Yes	Sometimes
EBI OLS ontology terms and synonyms	Yes	No
Multiple search terms per run	Yes	Varies
Ceiling	NCBI serves 3 requests per second; UCSC search returns the matches it indexes for a term	Depends on source

🚀 How to use

1. Create a free Apify account with $5 in credit.
2. Open the Genomics & Variants Pack Scraper.
3. Enter one or more gene symbols or rsIDs, pick the sources, set Max Items and click Start.
4. Download the dataset as CSV, Excel, JSON or XML from the Storage tab, or read it through the API.

A first run:

{
  "searchTerms": ["BRCA1"],
  "maxItems": 10,
  "sources": ["dbsnp", "clinvar", "ucsc", "ols"],
  "genome": "hg38"
}

Free-plan runs return up to 10 records as a preview. Upgrade your Apify plan to collect up to 1,000,000 records per run.

💼 Business use cases

🧬 Variant classification support. A clinical laboratory pulls every ClinVar assertion and dbSNP frequency for a panel of genes each week and flags variants whose classification changed.

📊 Annotation pipeline cache. A bioinformatics team schedules nightly runs for 50 genes and loads the rows into an internal database that powers their interpretation tool.

💊 Target assessment. A pharma research group compares allele frequencies and functional consequences across candidate genes before selecting a lead.

📖 Knowledge graph enrichment. A data team maps gene symbols to HGNC, NCIT and MONDO terms from OLS to link variant records with disease ontologies.

🔌 Automating Genomics & Variants Pack Scraper

• Make and Zapier: trigger a run when a new gene is added to a sheet and push the rows to Airtable or Google Sheets.
• Slack: post a summary when a scheduled run finds variants whose clinicalSignificance changed.
• Airbyte: sync the dataset into BigQuery, Snowflake or Postgres.
• GitHub Actions: rebuild a variant evidence file on every release of your pipeline.
• Google Drive: save each run as CSV or Excel for a lab share.

Every integration starts from the Apify API or the ApifyClient for JavaScript and Python.

🌟 Beyond business use cases

• Research: collect coordinates and ontology mappings for a systematic review in one export.
• Personal: look up the public record of a variant from a consumer genotyping report.
• Non-profit: patient advocacy groups can track new ClinVar submissions for their gene of interest.
• Experimentation: feed dbSNP consequences and OLS synonyms into an LLM to prototype variant summaries.

🤖 Ask an AI assistant about this scraper

Give an AI agent live access to the four databases through the Model Context Protocol:

claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/genomics-variants-pack-scraper"

Then ask in plain language, for example "List the likely pathogenic ClinVar variants in TP53 and their review status".

❓ Frequently Asked Questions

🧬 Which databases does this Actor query?

NCBI dbSNP, NCBI ClinVar, the UCSC Genome Browser and the EBI Ontology Lookup Service (OLS 4). Select any subset in the Sources field.

🔎 What search terms can I use?

A gene symbol (BRCA1, TP53), an rsID (rs334) or a free-text term. Gene symbols are sent to NCBI as field-qualified queries so they answer in under a second.

🔑 Do I need an API key for NCBI, UCSC or OLS?

No. All four APIs are public. The Actor identifies itself to NCBI with a tool name and contact email as their guidelines request.

📦 How many records can I collect?

Free users get 10 records as a preview. Paid Apify users can set Max Items up to 1,000,000 per run.

🎯 Why do I get exactly Max Items rows?

Sources are read round-robin, one record at a time, until the limit is reached. With four sources and Max Items 10 you get 3, 3, 2 and 2 rows.

🗺 Which genome assemblies does the UCSC source support?

hg38, hg19, mm39, mm10, rn7 and danRer11. dbSNP and ClinVar coordinates are reported on GRCh38 regardless of this setting.

📖 What does the OLS source return?

Ontology terms whose label or synonyms match the search term, with the ontology prefix, description, synonyms and IRI.

⚠️ What do N/A and Not Disclosed mean?

N/A means the field does not exist for that source (an ontology term has no chromosome). Not Disclosed means the source carries the field but this record has no value (most dbSNP entries have no clinical significance).

🚦 Does it handle NCBI rate limiting?

Yes. NCBI allows 3 requests per second per IP. The Actor serialises dbSNP and ClinVar calls with a 400 ms gap and retries HTTP 429 with exponential backoff.

❌ What happens if a source is down?

The other sources keep going. The failed source adds one row containing only an error field, which is never charged, and the run finishes with a status message that says so.

⏰ Can I run this on a schedule?

Yes. Create an Apify schedule to run the Actor daily or weekly and connect the dataset to your pipeline.

📄 What output formats are supported?

CSV, JSON, Excel and XML from the Apify Console or API.

🔌 Integrate with any app

The dataset is available through the Apify API the moment the run finishes. Use the run-sync-get-dataset-items endpoint for one-shot calls, webhooks to trigger downstream jobs, or the Make, Zapier, Airbyte and LangChain integrations listed on the Actor page.

🔗 Recommended Actors

• Healthgrades Doctor & Provider Scraper
• Website Content Crawler
• USAspending Awards Scraper
• Audio Transcriber - Speech to Text

---

💡 Pro Tip: browse the complete ParseForge collection.

🆘 Need Help? Open our contact form with your run ID, your input and what you expected.

⚠️ Disclaimer: This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by the National Center for Biotechnology Information, the University of California Santa Cruz or the European Bioinformatics Institute. It collects only publicly available data. You are responsible for using the data in compliance with each source's terms of use and applicable data-protection laws.