Scraping + Research
Reddit scraping with Claude Code
Crawl any subreddit, pull hundreds of posts and comments, and turn raw community data into actual insight. No coding required.
Why Reddit?
Reddit is one of the best places to understand what real people actually think. Not influencers, not marketers. Real humans complaining, recommending, and asking questions.
Three things you can pull from Reddit:
- The vibe of a community - how do people in r/yourspace actually talk? What language do they use?
- Painpoints - what are people frustrated about, asking for help with, or wishing existed?
- Recommendations + hate - what tools/products/services do people love or trash?
Two things to know about Reddit culture: they hate promo, and there are a lot of bots. These things are related.
The simple version (no setup)
Reddit has a trick: add .json to the end of any Reddit URL and you get structured data back. Claude Code can work with this directly.
You: Go to this subreddit and pull the top posts from
the last week. Summarize what people are talking
about, what painpoints come up, and what tools
people are recommending.
https://www.reddit.com/r/networking/.json
No API keys, no scripts, nothing to install. Claude Code fetches it, parses it, gives you a summary.
The better version (PRAW + Python)
For deeper pulls (hundreds of posts, comments, historical data), set up Reddit's official API. Takes about 5 minutes:
- Go to reddit.com/prefs/apps/ and create an app (type: "script")
- Tell Claude Code what subreddit you want, how many posts, and hand it your credentials
- It writes and runs the scraper for you, saves everything to JSON
You: I want to scrape r/networking for the top 50 posts
this week. Grab title, score, and top 10 comments
for each. Save to JSON.
Claude Code: [writes reddit_praw_scraper.py]
Claude Code: [runs it, saves 50 posts + 312 comments]
Claude Code: Done. Extracted 50 posts and 312 comments
from r/networking. Saved to reddit_data.json
When Reddit blocks you (the Gemini trick)
Reddit blocks a lot of automated requests. If Claude Code gets a 403, there's a workaround: use Gemini CLI as a middleman.
Gemini has its own web access that Reddit doesn't block. You ask Claude Code to spin up a Gemini session, have Gemini fetch the content, then pull the results back for analysis. One AI fetching data for another AI. Weird hack, but it works.
The automated version (Apify + GitHub Actions)
For ongoing monitoring, you can set up a pipeline: Apify scrapes the subreddit on a schedule, Claude API analyzes the data, and Resend emails you the briefing. Alex D has this running for r/ClaudeAI - every 8 hours, a formatted briefing hits his inbox automatically.
Example prompts to try right now
Prompt 1: Fetch the top 25 posts from r/[SUBREDDIT]
this month. Analyze: what are the recurring themes
and painpoints?
Prompt 2: Search Reddit for posts mentioning
[YOUR_PRODUCT]. Summarize the sentiment - what do
people love, hate, wish was different?
Prompt 3: Pull the most upvoted comments from the
top 10 posts in r/[SUBREDDIT] this week. What
language do people in this community use?
Market Research
Live example: Networking painpoints
We ran the Reddit research playbook for real. Here's what we found about professional networking frustrations, done in about 30 minutes.
The process (step by step)
1
Tried the direct route
Asked Claude Code to fetch Reddit's JSON API. Reddit blocked it (403). This happens a lot.
2
Used the Gemini workaround
Spun up Gemini CLI in a tmux session, asked it to search across 5 subreddits. Gemini pulled results and synthesized themes.
3
Ran broader web searches
Found where these conversations actually live and what language people use. This is the part most people skip, and it's the most valuable part.
4
Synthesized into themes
Combined everything into the painpoints, subreddit map, and search term guide below.
Where to look
Not all subreddits are created equal. Here's where people actually talk about networking pain:
r/Entrepreneur
r/sales
r/careerguidance
r/startups
r/smallbusiness
r/introvert
r/freelance
r/LinkedInLunatics
r/EntrepreneurRideAlong
What to search for
Don't search "professional networking painpoints." Nobody talks like that. Here's what real people type:
Emotional (high signal)
"I hate networking"
"networking feels fake"
"networking is exhausting"
"networking sucks"
"waste of time"
Specific frustrations
"LinkedIn spam messages"
"cold outreach annoying"
"business cards useless"
"follow up after networking"
"cold DM" + spam
The 5 big painpoints we found
01
"It feels fake and transactional"
The #1 complaint by a mile. People describe networking as "slimy," "dirty," and "like being a used car salesman." The structure itself turns relationships into transactions.
02
"Events are a waste of time"
Overscheduled, crowded, poorly organized. The recurring image: standing in a room full of strangers, balancing lukewarm wine and stale crackers, stumbling over your elevator pitch.
03
"Follow-up is broken"
Business cards get lost immediately. Only a tiny fraction of event exchanges turn into real conversations. The gap between "we met" and "we have a relationship" is huge.
04
"LinkedIn has become unusable"
64% of users report being annoyed by cold outreach. 35% call it straight-up spam. The tool that was supposed to make networking easier has become the biggest source of frustration.
05
"Introverts are just... left out"
The entire networking system is designed for extroverts. But here's the thing: most humans find this stuff uncomfortable. The system assumes everyone thrives in loud rooms full of strangers.
What people wish existed
- Context-first connections - connecting around shared problems, not just "we're both entrepreneurs"
- Async relationship building - not everything needs to be a live event
- Better follow-up tools - something between business cards and a full CRM
- Smaller, purpose-driven groups - mastermind groups and cohorts over big conferences
- Less performative, more real - AI matching and curated small groups are gaining ground
How to run this for YOUR industry
- Start with search terms people actually use (emotional language, not marketing language)
- Find 5-8 subreddits where your audience hangs out
- Use Claude Code to scrape, or the Gemini workaround when Reddit blocks you
- Don't just read titles. Comments are where the real pain lives.
- Look for recurring phrases. If 20 different people say "networking feels fake," that's your messaging.
The whole process took about 30 minutes.