Why LinkedIn Failed Us: Building a Real Database for International Student Hiring (80 Days to Stay)
After LinkedIn's API proved useless for identifying visa-sponsoring companies, this update outlines a new plan using SEC filings, domain checks, and Common Crawl data.
The original plan sounded reasonable: use LinkedIn to identify companies likely to sponsor international students, by looking at company pages, then at individual employees, and inferring from their educational and immigration backgrounds whether a company had a track record of hiring internationally. That plan is dead, and this update explains why, and what's replacing it.
Why the LinkedIn approach broke down
LinkedIn does have an API, but it's built for one purpose: allowing outreach at scale, the kind of behavior that looks a lot like spam. It is not built for research access, and there's no way to pay LinkedIn for the kind of data access this project would need, even as a paying customer. Building a crawler to get around that restriction was considered and rejected. Beyond the API restriction, the broader experience of using LinkedIn as a platform reinforced the decision: a feed dominated by irrelevant sponsored messages and generic corporate language, and even a case where LinkedIn removed a post for describing cold outreach accurately as what it is, unsolicited messaging hoping for a small percentage of responses. The frustrating part is that LinkedIn has the underlying data to offer something genuinely useful, tailored job matching for individual users, but has chosen a different, spam-oriented business model instead.
The new approach: filtering SEC Form D data
With LinkedIn ruled out, the new plan starts from a completely different data source: SEC Form D filings, which cover roughly 500,000 companies. The first step is filtering that list down aggressively, removing companies based outside the US, excluding industries unlikely to sponsor international hires, such as dining services and industries requiring security clearances that international candidates typically can't obtain, and narrowing the full set down to a target range of 10,000 to 20,000 high-value prospects.
Verifying companies through domain matching
Once the filtered list exists, the next step is domain verification: inferring a likely website based on each company's name, and checking whether that guess actually resolves to a real site. Where a match is found, the site itself gets crawled, and the text on the site is checked against what's expected for that company, confirming the match before treating the company as verified. This step, cheap and mechanical as it sounds, does real work: pure name-guessing catches a meaningful share of legitimate matches on its own.
Layering in Common Crawl and paid search
Beyond direct domain guessing, the plan brings in Common Crawl data as an additional source, accessed through its API rather than downloading and processing the full raw data files directly, since those files run into the gigabytes per file across thousands of files, a scale that isn't practical to work through by hand at this stage. If free options don't cover a company, the plan allows for falling back to the Google Search API as a paid option, which includes a free daily allotment of searches and a modest per-search cost beyond that, to be used only after the free approaches have been exhausted.
Why crawling company sites pays off twice
Crawling a company's own website isn't just about confirming a name match. It also surfaces the company's actual jobs pages, which is valuable in its own right. Job listings that mention visa sponsorship or openness to international candidates are a direct, strong signal, stronger than an inference based on any individual employee's background, and finding those listings is a natural byproduct of the same crawl used for verification.
Bringing in student verification
The final piece of the plan is human verification, sourced from the existing network of students who've been part of this work in the past. The idea is to send out batches of company URLs and ask students to check and report back on what they find, supported by a purpose-built Vercel website where people can submit a URL along with information about whether they believe that company has hired internationally, using an already-developed outreach message for that request.
Key takeaways
- LinkedIn's API is designed for outreach at scale, not research access, and there's no paid tier that unlocks the kind of data this project needs.
- The replacement approach starts from SEC Form D filings, roughly 500,000 companies, filtered down to a target of 10,000 to 20,000 high-value prospects.
- Domain matching by company name, followed by crawling and text verification, is the first and cheapest verification layer.
- Common Crawl's API and, as a paid fallback, the Google Search API extend coverage beyond direct domain guessing.
- Crawling a verified company's site also surfaces job listings, which offer a stronger direct signal of visa sponsorship than inferring from employee backgrounds.
- A student-network verification step, supported by a purpose-built Vercel site, adds human confirmation on top of the automated pipeline.
Who this is for
This update is part of the 80 Days to Stay initiative, which supports international students navigating visa deadlines around employment. It's a useful look for anyone building data pipelines from public filings on a limited nightly time budget, and it reflects the kind of practical, incremental engineering work Humanitarians AI documents in public as it happens.
Full transcript(auto-generated, with timestamps)
[0:00]K bear here. Uh last time I was thinking of using uh LinkedIn. Basically the idea was to look at find company pages look at people and then once we see people we look at sort of their background and make an inference of of you know are they likely to have hired like if we look at this person here uh we would go down and we would say okay he got a master's from University of Oklahoma but a bachelor's in India so he's very likely international and since he's very likely international it's very likely that visit has hired some international people because it looks like they have
[0:55]Hired an international person but LinkedIn as usual is useless so they have an API but their API is for if you want to spam people like me. So if I even if I want to pay them to get this data, they don't allow me to do that. I could create a crawler which is crawled it anyway, but I'm not going to do that. Um the the LinkedIn is is LinkedIn. I should not be surprised that that LinkedIn is basically useless. Uh my entire LinkedIn experiences sponsored messages with have nothing no clue as what I do. a lot of you know nonsense corporate talk we help sea
[1:38]Level sweet executives do whatever percent RII whatever that's a daily experience on LinkedIn LinkedIn even wants when I post on what they call a cold marketing group I think that's D or cold marketing means a spam uh you're you're sending information asking from somebody who you know nothing about you're just hoping.1% click and do something. That's spam. Uh I mentioned that on this cold marketing group and LinkedIn actually didn't like that. Remove the post etc etc. So LinkedIn as usual is useless. LinkedIn and this a sab is LinkedIn could actually their LinkedIn premium or whatever it's called you what that should be because they have the data to
[2:28]Do this. It should be subscribe to LinkedIn uh premium and show me highly tailored job possibilities for me. LinkedIn can do that, but instead they chosen to be a spam site. So, we're going to move on from LinkedIn. Uh I shouldn't be at least surprised that you know LinkedIn is absolutely useless. Um, so now that we've moved on from LinkedIn, what we are going to do is I'm going to filter down to just 1 million and above companies. remove things from Canada, remove things from the UK, remove things from Europe, remove industries which are highly unlikely like you know um dining service industries, a lot of real estate, oil industries,
[3:16]Some industries which aren't allowed to hire is you have to get sort of US security clearances and get, you know, all that stuff. So, we're going to narrow down. I'm going to do that tonight. My work process is to spend an hour or two at night doing this because I'm busy. I have 50 other things to do. Uh but tonight I'll actually do the programming. But this is an outline of what the new approach is. So the new approaches one to filter down are you know half a million things to maybe 10 20,000 high likelihood high value prospects. Then we are going to um just start checking is there a domain
[3:56]Based on the name just inferring. If we get something that matches i.e. we get a website there we'll then crawl the website. If the website text looks like it matches that company we'll check yes. If not we won't. Um but that should get us some just from the name guessing from the name. Then we're going to look at common crawl. So we're going to take common crawl and you know go that way as well. Chrome and crawl is very very large. So they do have the text there but you know each one is a gigabyte and they have thousands of those files. So that would require downloading a
[4:33]Gigabyte file going through it deleting that doing another one deleting that. We may eventually do that but not for now. We're just going to use their API to see what we can get. And then if we can't get from that, then we'll do the, you know, paid things. We will, uh, use the Google search API, which I think cost, I forgot. I think I have the cost down here somewhere. Um, but I can pay Google to, but at least Google allows me to pay them. They give me a 100 free like searches a day. And so if I wanted to do this over months, I
[5:09]Could do that. Uh but it's not that I think it would cost me maybe $50 to uh search for, you know, 10,000 URLs to see if those websites existed. But we won't do that until we've done the free options first. And then we'll get a basically the idea is to get a uh set of uh domains, crawl those domains to see if all of the text matches what we think they are. That actually has a big advantage for us because that will allow us to find the people pages, the job pages, all those things as well. So we can actually develop a better model because we can
[5:49]Look at their job listings. their job listings say, you know, we're open to whatever visa stuff, then that's another strong indication that they're open to hiring visa stuff. It also gets us, you know, their actual job listings. What I'm also going to do is put up a website for people to put in URLs and verify. So, I've had thousands of students in the past. So, I'm going to reach out and say, "Hey, uh, and for the current students, hey, here's some URLs. check them, you know, is here's 50 euros, please check them. I'll create a Versel website for that. So, what the Versel website will do is
[6:29]It'll basically, you know, ask people to put in a URL and give us some information about a URL. You know, what you know, do you think they hired? You know, they hired I I'll send that form letter. We already have that form worked out. Um so the the basic idea ba here is basically to crawl the entire sort of web space of companies likely to to hire international. So this may take more than today. I don't know. I only have a hour or two a day to spend on this. Um but that's the new approach. So again to summarize LinkedIn is useless. And so what we're going to do is just
[7:14]Create a whole fresh new data set of you know basically crawling a subspace of the internet which is going to be the the companies that that are likely to hire international students. Okay, that's it. Take care.
More from 80 Days to Stay
10:53Building a Visa Sponsorship Database from SEC Data | 80 Days to Stay Project
2:19LLMs Are Like Puppies: They'll Always Fetch, But Sometimes The Wrong Thing | 80 Days to Stay
14:1880 Days to Stay
4:15Building a AI-Powered Web Scraper to Help International Students Find Visa Sponsors | 80 Days 2 Stay
3:00From 500,000 URLs to Real Jobs: Inside the 80 Days to Stay Engine
2:08