ParseForge
Lobsters Scraper
parseforge/lobsters-scraper
News & mediaDeveloper toolsOther
Scrapes Lobsters tech news stories from the hottest or newest feed, optionally filtered by tag. Returns each story as a flat row with title, score, tags, comments, and author.
- Total users
- 2
- Monthly active
- 1
- Total runs
- 100
- Bookmarked
- 0
- Rating
- Not rated yet
- Last modified
- 12 days ago
Overview
Lobsters Scraper
Scrape Lobsters tech news stories from the hottest or newest feed, filtered by tag, up to a million per run. Every story comes with its title, score, tags, comments, and author. No login or API key. Export to CSV, JSON, Excel, or XML.
Lobsters is a focused tech news community where stories are submitted and voted on by developers, and the front page is a tight signal of what engineers care about right now. This Actor reads the public hottest and newest feeds directly, so you can collect stories by tag without building a scraper or dealing with rate limits.
| Who uses it | What they scrape Lobsters for |
|---|---|
| Developer advocates | Which tools and languages the community is discussing this week |
| Market researchers | What topics and tags are trending among technical audiences |
| Content marketers | Story ideas and discussion angles that resonate with developers |
| Data analysts | A clean dataset of tech news stories for trend analysis |
What it does
This Actor collects Lobsters tech news stories from the hottest or newest feed, optionally filtered by tag, and returns each story as a flat row.
- 🔥 Hottest feed: stories ranked by score, the same list you see on the Lobsters front page.
- 🆕 Newest feed: stories in chronological order, for catching submissions as they appear.
- 🏷️ Tag filter: narrow to a single tag like programming, javascript, python, security, linux, web, ask, or show.
- 📦 Flat rows: every story is one row with title, score, tags, comments, and author, ready for CSV or JSON.
Results export to CSV, JSON, Excel, or XML, or straight from the API.
What you can do with Lobsters data
📈 Track trending tech topics.
A developer advocate runs the Actor on the hottest feed each morning to see which tags and stories are gaining traction, then writes a newsletter that matches the community's current interests.
🔍 Monitor a specific tag.
A security researcher filters by the security tag and pulls the newest stories daily to stay on top of disclosed vulnerabilities and discussions before they hit mainstream news.
📊 Build a dataset for analysis.
A data analyst collects a month of stories with scores and tags, then analyzes which topics get the most engagement and how that changes over time.
✍️ Find content ideas.
A content marketer scrapes the ask tag to see what questions developers are asking, then writes blog posts that answer those questions directly.
Why choose this scraper
| What you get | |
|---|---|
| No API key | Reads the public Lobsters feeds directly, no registration or OAuth. |
| Tag filtering | Collect only stories from one tag, like security or ask, instead of the whole feed. |
| Flat schema | Every story is one row with the same fields, so you can append runs and build a history. |
| Scale | Paid users can pull up to a million stories per run for large archives. |
How it compares
This Actor focuses on the hottest and newest feeds with tag filtering, while the other Lobsters scraper also mentions search results.
| Feature | ParseForge | Lobsters Scraper - Tech News & Discussions |
|---|---|---|
| Hottest feed | Yes | Yes |
| Newest feed | Yes | Yes |
| Tag filter | Yes | Not listed |
| Search results | Not listed | Yes |
| Flat row output | Yes | Not listed |
Configure the run
Drive the Actor from the hottest or newest feed, optionally filtered by a single tag, and set a maximum number of stories so only the matches you need reach your dataset. The Input tab lists every parameter.
A first run with the defaults:
{
"maxItems": 10
}
A larger pull:
{
"maxItems": 200
}
Pricing
Pay-per-result: $0.021 per result collected. You pay only for the results written to your dataset.
| Results collected | Approximate cost |
|---|---|
| 100 results | $2.10 |
| 1,000 results | $21.00 |
| 10,000 results | $210.00 |
New Apify accounts start with $5 in free credit.
Free users
Free-plan runs return up to 10 results as a preview. Upgrade your Apify plan to collect up to 1,000,000 results per run.
Run it
- Create a free Apify account with $5 in credit.
- Open the Lobsters Scraper.
- Set your inputs and any filters, then click Start.
- Export the results as CSV, Excel, JSON, or XML from the Dataset tab.
Run it programmatically through the Apify API (run-sync-get-dataset-items) or the ApifyClient for JavaScript and Python.
Use with AI agents (MCP)
Give an AI agent live access to Lobsters through the Model Context Protocol. Add the Actor to Claude, Cursor, or any MCP client:
claude mcp add --transport http apify "https://mcp.apify.com?tools=parseforge/lobsters-scraper"
Then prompt it in plain language to run the scraper and read back the results.
Troubleshooting
Why am I getting no results?
Check your tag filter. If you set a tag that does not exist or has no recent stories, the Actor returns an empty dataset. Try leaving the tag empty or using a common tag like programming.
Why am I only getting 10 items?
Free users are limited to 10 items as a preview. To get more, upgrade to a paid plan and set maxItems to a higher number.
Why are the results different from the Lobsters front page?
The hottest feed is ranked by score, which can change as users vote. If you run the Actor at different times, the order and set of stories may differ.
Can I get more than one tag at a time?
No, the tag input accepts a single tag. To collect multiple tags, run the Actor once per tag and merge the datasets.
Why is the author field empty for some stories?
Some stories may be submitted by users who have deleted their accounts or by anonymous users. In those cases, the author field is empty.
FAQ
| Question | Answer |
|---|---|
| What is Lobsters? | Lobsters is a technology-focused link aggregation site where users submit and vote on stories. It is similar to Hacker News but with a smaller, more curated community and a strong tagging system. |
| Do I need an API key or login? | No. This Actor reads the public Lobsters feeds directly, so you do not need to register an app or provide credentials. |
| Which feeds can I scrape? | You can scrape the hottest feed, which is ranked by score, or the newest feed, which is chronological. Both are available from the feedType input. |
| Can I filter by tag? | Yes. Set the tag input to a single tag like programming, javascript, python, security, linux, web, ask, or show. Leave it empty to get all stories. |
| What data do I get for each story? | Each story is returned as a flat row with its title, score, tags, comments, and author, among other fields. The exact fields are shown in the sample output. |
| How many stories can I scrape? | Free users are limited to 10 items as a preview. Paid users can set maxItems up to 1,000,000 per run. |
| What formats can I export to? | You can export the results to CSV, JSON, Excel, or XML from the Apify dataset. |
| Is this the same as the Hacker News scraper? | No. Hacker News and Lobsters are different sites with different communities and data. This Actor scrapes Lobsters only. |
| Can I scrape search results? | This Actor scrapes the hottest and newest feeds, not the search page. If you need search results, you may need a different tool. |
| How often should I run this? | It depends on your use case. For trending topics, run it daily on the hottest feed. For monitoring, run it hourly on the newest feed with a tag filter. |
Related actors
Browse the full ParseForge collection for more scrapers.
🆘 Need help? Email parseforge@protonmail.com with your run ID, your input, and what you expected.
⚠️ Disclaimer. This Actor is unofficial and is not affiliated with, endorsed by, or sponsored by Lobsters. It collects only publicly available data. You are responsible for using the collected data in compliance with the source's terms of service and applicable data-protection laws, including GDPR, CCPA, and PIPL. Do not use it to collect personal data unlawfully.
Input
| Field | Type | What it does | Default |
|---|---|---|---|
| feedType | string (2 options) | Which feed to scrape | hottest |
| tag | string | Filter by tag (e.g. 'programming', 'javascript', 'python', 'security', 'linux', 'web', 'ask', 'show'). Leave empty for all stories. | not set |
| maxItems | integer | Free users: Limited to 10 items (preview). Paid users: Optional, max 1,000,000 | 10 |
Pricing
from $19.00 per 1,000 results
| Charged for | What it is | Price each |
|---|---|---|
| result | Single result in the default dataset. | $0.019 to $0.021 |
Tiered: the lower figure is the price on a higher Apify plan. Billing and the free credit live on Apify.
API
One POST returns the dataset directly. Same shape for every scraper in the library, so swapping the slug is the only change.
curl -X POST "https://api.apify.com/v2/acts/parseforge~lobsters-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"feedType": "hottest",
"maxItems": 3
}' Examples
Input that runs as-is.
{
"feedType": "hottest",
"maxItems": 3
} Reviews
No reviews yet. Be the first.
Issues
We build and maintain this scraper, so a problem with it comes to us. Report it on the Apify listing and the thread stays attached to the scraper where the next person can find it: open an issue.
Broken and urgent, or you would rather not post in public? Write to parseforge@protonmail.com and it reaches the people who wrote it.