ParseForge
Hex.pm Scraper
parseforge/hex-pm-scraper
Developer toolsAutomationOther
Scrape Elixir and Erlang package metadata from Hex.pm by keyword. Returns downloads, version, license, and repository links as flat rows. Export to JSON, CSV, Excel, or XML.
- Total users
- 2
- Monthly active
- 1
- Total runs
- 99
- Bookmarked
- 0
- Rating
- Not rated yet
- Last modified
- 12 days ago
Overview
Hex.pm Scraper
Scrape Elixir and Erlang package metadata from Hex.pm by keyword, sorted by downloads or recency, up to a million per run. Every package row includes its downloads, version, license, and repository links. No API key or authentication. Export to JSON, CSV, Excel, or XML.
Hex.pm is the central package registry for the Elixir and Erlang ecosystems, but browsing it manually or through its API limits how many packages you can review at once. This Actor reads the public package listing directly, filters by a search keyword, and returns each match in one consistent flat schema. You can pull the most downloaded packages, the newest releases, or everything matching a library name like Phoenix or Ecto.
| Who uses it | What they scrape Hex.pm for |
|---|---|
| Elixir developers | Auditing dependency popularity and recent updates before adding a library to a project. |
| DevOps engineers | Tracking version histories and repository links across an organization's allowed package list. |
| Security researchers | Checking license types and latest versions for compliance across the Erlang/Elixir supply chain. |
| Market analysts | Measuring package download trends to gauge ecosystem growth and library adoption. |
What it does
This Actor collects Hex.pm package listings by keyword and returns each package as a flat row with its downloads, version, license, and links.
- 🔍 Keyword search: Filter packages by name or term, like 'phoenix' or 'ecto', or leave empty to fetch the global list.
- 📊 Flexible sorting: Order results by total downloads, recent downloads, alphabetical name, or newest first insertion date.
- 📦 Bulk collection: Set a maximum from 1 up to 1,000,000 packages per run to fit a quick check or a full registry snapshot.
Results export to CSV, JSON, Excel, or XML, or straight from the API.
What you can do with Hex.pm data
📈 Monitor library adoption.
A developer pulls the top 100 packages sorted by recent downloads every week to spot which new libraries the Elixir community is adopting.
🔐 Audit license compliance.
A security engineer scrapes all packages matching an internal allowlist by name, then reviews the license field in the exported CSV before a product release.
🧪 Research package metadata.
A data analyst collects the full list of packages sorted by insertion date to study how the Hex.pm registry has grown over time.
🔗 Gather repository links.
A platform team exports package rows to get the source code repository URL for every dependency in their monorepo.
Why choose this scraper
| What you get | |
|---|---|
| No API key required | Reads the public Hex.pm pages directly with zero authentication or rate-limit setup. |
| Fixed flat schema | Every package row has the same columns, so you can append runs without reshaping. |
| Sort by what matters | Rank by total downloads for popularity, recent downloads for momentum, or newest first for fresh releases. |
How it compares
Three other Hex.pm scrapers exist on Apify. This Actor focuses on keyword search with flexible sorting and a high item ceiling, matching the core listing data the competitors offer.
| Feature | ParseForge | Hexpm Scraper | Hex.pm Package Scraper | Hex.pm Scraper - Elixir & Erlang Packages |
|---|---|---|---|---|
| Keyword search filter | Yes | Not listed | Yes | Not listed |
| Sort by total downloads | Yes | Not listed | Not listed | Not listed |
| Sort by newest first | Yes | Not listed | Not listed | Not listed |
| Up to 1,000,000 packages per run | Yes | Not listed | Not listed | Not listed |
| License field | Yes | Not listed | Yes | Yes |
| Repository links | Yes | Not listed | Yes | Yes |
Configure the run
Drive the Actor with a package name or keyword, choose a sort order, and set a maximum item count so only the packages you need land in your dataset. The Input tab lists every parameter.
A first run with the defaults:
{
"maxItems": 10
}
A larger pull:
{
"maxItems": 200
}
Pricing
Pay-per-result: $0.021 per result collected. You pay only for the results written to your dataset.
| Results collected | Approximate cost |
|---|---|
| 100 results | $2.10 |
| 1,000 results | $21.00 |
| 10,000 results | $210.00 |
New Apify accounts start with $5 in free credit.
Free users
Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.
Run it
- Create a free Apify account with $5 in credit.
- Open the Hex.pm Scraper.
- Set your inputs and any filters, then click Start.
- Export the results as CSV, Excel, JSON, or XML from the Dataset tab.
Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.
Use with AI agents (MCP)
Give an AI agent live access to Hex.pm through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:
claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/hex-pm-scraper"
Then prompt it in plain language to run the scraper and read back the results.
Troubleshooting
Why am I getting no results?
Check your searchQuery spelling. Hex.pm search matches package names and descriptions. Try a broader term or leave the query empty to confirm the listing pages are reachable.
The run stopped before reaching my maxItems.
The Actor stops when it has exhausted the available listing pages for your query. If fewer packages exist than your maxItems value, the run ends early with whatever was found.
Some fields are empty in my dataset.
Not every Hex.pm package fills in all metadata. A missing license or repository link means the package author did not provide that information on Hex.pm.
The sort order does not match what I see on the Hex.pm website.
The Actor uses the same sort parameters that Hex.pm exposes on its package listing pages. Ensure you selected the correct enum value: downloads for total, recent for recent downloads, name for alphabetical, or inserted_at for newest first.
My run is taking a long time.
Large maxItems values mean more pages to fetch. Reduce maxItems to a smaller number for a quicker test, then scale up when you are ready for a full collection.
FAQ
| Question | Answer |
|---|---|
| Do I need a Hex.pm account or API key to use this scraper? | No. The Actor reads the public Hex.pm package listing pages directly. You do not need to register an account, create an API key, or handle OAuth. |
| What data does each package row contain? | Each row returns the package name, description, total downloads, recent downloads, latest version number, license, and repository links as they appear on Hex.pm. |
| Can I scrape all packages on Hex.pm? | Yes. Leave the search query empty and set the maximum items high enough to cover the registry. The Actor will paginate through the public listing until it hits your limit. |
| How do I get only the newest packages? | Set the sort order to 'Newest First' (inserted_at). The Actor will return packages in reverse chronological order of when they were added to Hex.pm. |
| What export formats are supported? | You can export your dataset to JSON, CSV, Excel, or XML from the Apify platform after the run completes. |
| Can I search for multiple keywords in one run? | A single run accepts one search query. To collect packages for several keywords, run the Actor multiple times with different searchQuery values, or leave the query empty and filter the full dataset afterward. |
| Does this scraper get version history for each package? | The Actor returns the latest version number as shown on the package listing page. It does not fetch the full version history or changelog for each package. |
| Is there a rate limit? | The Actor respects Hex.pm's public pages. Apify's platform handles request pacing, but you should set a reasonable maxItems for your use case. |
| Can I filter by license type directly? | The input schema does not include a license filter. The Actor returns the license field for each package, and you can filter the exported dataset by license in your own tooling. |
| What is the difference between total downloads and recent downloads? | Total downloads is the all-time count for a package. Recent downloads reflects a shorter rolling window as reported by Hex.pm, useful for gauging current momentum. |
Related actors
Browse the full ParseForge collection for more scrapers.
🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.
⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Hex.pm. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.
Input
| Field | Type | What it does | Default |
|---|---|---|---|
| searchQuery | string | Filter packages by keyword (e.g. 'phoenix', 'ecto'). Leave empty to get all packages. | not set |
| sortBy | string (4 options) | How to sort the results | downloads |
| maxItems | integer | How many packages to collect per run. | 10 |
Pricing
from $19.00 per 1,000 results
| Charged for | What it is | Price each |
|---|---|---|
| result | Single result in the default dataset. | $0.019 to $0.021 |
Tiered: the lower figure is the price on a higher Apify plan. Billing and the free credit live on Apify.
API
One POST returns the dataset directly. Same shape for every scraper in the library, so swapping the slug is the only change.
curl -X POST "https://api.apify.com/v2/acts/parseforge~hex-pm-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"helloWorld": 123
}' Examples
Input that runs as-is.
{
"helloWorld": 123
} Reviews
No reviews yet. Be the first.
Issues
We build and maintain this scraper, so a problem with it comes to us. Report it on the Apify listing and the thread stays attached to the scraper where the next person can find it: open an issue.
Broken and urgent, or you would rather not post in public? Write to parseforge@protonmail.com and it reaches the people who wrote it.