Scraping 25K Startup Sites & Preparing Them for a Language Model
After a 59-hour scrape pulled almost half a million pages from 25,000 startups sourced from SEC data, the next step is turning that raw mess into a clean dataset a language model can query.
Scraping thousands of company websites is the easy part. The hard part starts after the bot stops running, when you are staring at hundreds of thousands of raw HTML pages and need to turn them into something a language model can actually reason about. That is exactly the stage covered here: what happens on day four of an 80-day project after a scraper finishes pulling data from roughly 25,000 to 26,000 startups sourced from SEC filings.
What the scrape actually produced
The run collected close to half a million HTML pages, averaging around 16 to 17 pages per company, and it took about 59 hours running on and off on a single laptop. That volume matters because it is not just a list of company names. Each company now has a stack of raw pages sitting alongside the structured SEC data that seeded the search in the first place: company name, funding raised, and address. The first move after a run like that is a simple insurance policy: back everything up to Google Drive before touching it with any script, so an accidental delete does not erase 60 hours of collection work.
Turning raw pages into one clean file per company
The real work is converting scattered JSON and web content into something usable. That means combining the SEC-derived JSON data with the scraped web content, stripping out the HTML markup, and merging everything into a single markdown file per company. This step exists specifically so the result can be handed to a language model and queried with plain questions, rather than requiring anyone to manually dig through dozens of raw pages per company.
The questions the dataset needs to answer
Once the data is in a queryable shape, the plan is to annotate every company by asking a language model a consistent set of questions: What does this company actually do? Do they hire recent graduates? Do they hire internationally? Do they have current job openings, and what are those roles? Because most companies have an "about" page describing what they do and who their people are, that page becomes a key source for these answers, layered on top of the roughly 16 pages already gathered per company.
Extracting contact information and signals
Beyond the core questions, the dataset also gets enriched with anything that can be found in the raw pages: emails, addresses, and social links like Twitter, Instagram, or Facebook. On top of that, the plan includes tracking signals like which applicant tracking system (ATS) a company uses and whether it has filed for H-1B sponsorship, since that kind of filing data is publicly available through separate government databases. All of this turns a bare SEC record into a much richer profile of each company.
The endgame: a smaller, higher-quality list
The goal is not to keep all 25,000 companies in play forever. It is to take everything through one full pass and end up with somewhere between 10,000 and 15,000 high-quality leads, prioritizing getting something useful out quickly rather than perfecting domain inference first. Since the initial 25,000 companies were found by guessing at domains, a more sophisticated pass later could likely uncover another 25,000 that were missed the first time around, but that refinement comes after the first useful dataset ships.
Key takeaways
- A 59-hour scrape on a single laptop pulled roughly 25,000 to 26,000 companies from SEC data and close to half a million HTML pages.
- Back up raw scraped data immediately, before writing any processing script that could accidentally delete it.
- Merging JSON and web content into one markdown file per company, with HTML stripped, is what makes the dataset queryable by a language model.
- Planned annotations include what the company does, hiring practices, open jobs, contact info, ATS used, and H-1B filing history.
- The target is 10,000 to 15,000 high-quality leads from a first pass, with room to recover more companies through better domain inference later.
Who this is for
This is for anyone building a lead generation pipeline, working with public SEC filings, or curious about the practical, unglamorous steps between "the scraper finished" and "the dataset is actually useful."
Full transcript(auto-generated, with timestamps)
[0:00]Okay, bear here. The bot finished running. It got 25,000 26,000 companies. It calc it collected uh almost a half a million HTML pages around 16 and a half pages on average per company to 59 59 hours to run. So, what we're going to do now and I'll run this in a bit. So, what I did right now is I took all of this data. I uploaded it to a Google Drive in case I screw up. I can just download it again and you know if I write a script which accidentally delete some stuff, I'm not spending another 60 hours collecting this again. Um again, this
[0:41]Just wrote off ran after a little laptop and you know on and off and D. Um and so what we're going to do now is I'm going to take all of that data and I'm going to make it into one markdown file. So I'm going to combine the JSON data with the web data. Remove all the HTML. And then this is something that we're going to then present to a language model and ask it questions like what does this company do? Do they hire recent graduates? Why? Do they hire international? Why? Do they have current jobs? What are they? Kind of thing. So we can get a summary of every company
[1:17]Based on these sort of 16 pages on average that we have about the company. So that so we have URLs, we have addresses if we can find emails in those documents, then we'll extract those as well. So we're basically annotating that. I'm going to check whether the script that I just wrote which goes to actually generates emails. Not I'll write a specific script to do that. Uh we'll also look for Twitter and other kinds of things, you know, any kind of social media the company might have and add that to the database. So we started with SEC. We have some basic info like name of the company, how much money they
[1:56]Raised, address. Now by finding the URL, we can then calculate whatever information that's sort of easily accessible on their website. It's typically things like do they have jobs? What are the jobs? Most companies have an about page, what do we do, who are we, who are our people, etc. And then uh we're also just going to look for any kind of contact info that we can find like Instagram, Twitter, Facebook, whatever. Okay, so that said, I'm going to run that. Uh but we're already at the point where we have a pretty useful data set. We'll add a bunch of other things including like what kind of ATS they use, how they
[2:38]Filed with the government because there are other databases I've recently find out with that list. Okay, this company has filed for these H1. These are things that's actually publicly listed as well. So, we're going to add that data as well. So, this is day four of 80 days to stay. What we have at this point is around 25,000 companies. We can get more if I go back and remember we just infer the domains. We literally guessed at the domains and found 25,000 of them. More sophisticated methods, we could probably find another 25,000. But because I want to get a useful list to people soon of maybe 15,000 or 10,000
[3:15]High quality leads, we're just going to take everything through one pass and then go back and sort of figure out techniques to pick up what we missed. Okay, that's it. Take care.
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53