Ultimate Web Scraping with Crawlee.dev: JavaScript Framework Deep Dive | Humanitarians AI Fellows
A code-first tour of Crawlee.dev, the JavaScript web scraping framework from Apify, covering a working PlaywrightCrawler script, its use cases, and how it stacks up against Scrapy, Selenium, and BeautifulSoup.
Most people trying to scrape a JavaScript-heavy website end up stitching together two or three tools: something for HTTP requests, something for headless browser control, and their own hand-rolled logic for retries, queues, and rate limits. Crawlee.dev, an open-source framework built by Apify, tries to fold all of that into one interface. This walkthrough covers what it actually does, reviews a real script that crawls a Wikipedia page, and lays out when Crawlee is the right tool versus when something simpler will do the job better.
What Crawlee actually is
Crawlee supports three different crawler types under one interface: CheerioCrawler for fast, simple HTML parsing, PuppeteerCrawler, and PlaywrightCrawler for full headless browser automation. Because all three live inside the same framework, switching between a lightweight HTTP-based approach and a full browser-driven approach for JavaScript-heavy sites is mostly a matter of swapping which crawler class you import, rather than rewriting your scraping logic from scratch.
Walking through a working PlaywrightCrawler script
The demonstration script imports PlaywrightCrawler from Crawlee along with Playwright itself, plus Chromium from Playwright. An ensureBrowser step checks whether Chromium is installed and installs it if needed, so the script can guarantee it has a working browser before crawling starts. From there, a configuration object sets the crawl parameters, including maxRequestsPerCrawl, which caps how many pages the crawler will visit (the example is set to crawl 50 pages by default, though it's dialed down to 10 for a faster demo run), and launch options like headless. With headless mode on, the crawl runs invisibly in the background; setting it to false pops open an actual browser window so you can watch pages load and get crawled in real time.
The core logic sits inside the request handler. For each page, it waits for the page to load fully, then extracts the title, metadata, description, paragraphs, and links, and saves that data to a dataset. Any links found on the page get pushed onto a queue, which the crawler keeps pulling from until it hits the request cap, here capped at 10 for the demo. A simple error handler logs anything that goes wrong during the crawl. Starting the crawler is a single command, running the script file with Node, and in the demo it visits Wikipedia's page on web scraping, following internal links, saving each page's title, URL, paragraphs, and links to a growing dataset, along with session statistics for the run.
One practical note that comes up before the demo starts: some websites enforce rate limits, and exceeding them can get your IP address banned outright, which creates real headaches. The suggestion is to run scraping traffic through a VPN, with the free tier of ProtonVPN mentioned as one option, specifically to avoid that risk.
Where Crawlee shines
Crawlee's use cases span data collection, market research, academic studies, and training data for AI systems. It's also useful for SEO monitoring, tracking keyword rankings and competitor strategies, for e-commerce price tracking, and for lead generation and content aggregation, things like pulling in news, blog posts, and social media content at scale.
The advantages that stand out: built-in anti-bot handling that manages rate limits, retries, and CAPTCHA-solving; high scalability through its queue-based architecture, which supports running multiple crawlers against a shared backlog of URLs; full headless browser support for sites that require real JavaScript execution; flexible middleware for custom data processing logic; and solid documentation and community support backed by Apify.
Where it falls short
No tool is a fit for every job. Crawlee has a steeper learning curve than simpler libraries, which can be a real barrier for beginners. It's also more resource-intensive, doing large-scale crawls comfortably requires decent CPU and RAM. And it's not the right choice when a site already offers a clean API, since hitting that API directly will almost always be more efficient and reliable than scraping rendered pages for the same data.
Crawlee versus the alternatives
Compared to Scrapy, which handles static HTML well but struggles with JavaScript-rendered content, Crawlee's browser automation gives it a real edge on modern, JavaScript-heavy sites. Compared to Selenium, which is built primarily for testing, Crawlee is designed from the ground up for large-scale scraping. Compared to BeautifulSoup, which is excellent at parsing HTML but has no built-in automation or crawling logic of its own, Crawlee handles the entire pipeline: fetching, rendering, queueing, and extraction. That said, each of those tools still has its place: BeautifulSoup is faster and simpler for basic static HTML scraping, a site's own API beats scraping it entirely when one is available, and Selenium or Playwright directly still make sense when a task specifically requires interacting with a page's GUI.
Key takeaways
- Crawlee.dev is an open-source JavaScript scraping framework from Apify that unifies CheerioCrawler, PuppeteerCrawler, and PlaywrightCrawler under one interface.
- The demo script uses PlaywrightCrawler with a request handler that extracts page title, metadata, paragraphs, and links, then queues discovered links for further crawling.
maxRequestsPerCrawlcaps how many pages a run will visit, and togglingheadlesslets you watch the crawl happen in a real browser window instead of running it invisibly.- A VPN, ProtonVPN's free tier was suggested, helps avoid IP bans from sites that enforce rate limits.
- Crawlee's biggest strengths are anti-bot handling, queue-based scalability, and full headless browser support; its costs are a steeper learning curve and heavier resource use.
- When a target site has its own API, use the API. Scraping is the fallback, not the default.
Who this is for
This tutorial was produced as part of the Humanitarians AI Fellows program and is aimed at both beginners and experienced developers who need to build large-scale data extraction pipelines. If you already know basic JavaScript and want a single framework that handles headless browsers, queues, and anti-bot protection without hand-building each piece yourself, this walkthrough gives you a working starting point.
Full transcript(auto-generated, with timestamps)
[0:00]Hello everyone, welcome back to the channel. Today we are taking a quick look at crawly.dev. This is the ultimate web scraping and automation framework. Okay, web scraping and automation are crucial for data collection, competitive analysis and AIdriven applications. Today we are diving into crawly.dev, a powerful framework designed for efficient web uh scraping. We'll explore its use cases, advantages, disadvantages, and alternatives. And we will also review a small piece of code which uh which I used to scrape Wikipedia like a Wikipedia page. Okay. Crawl is a open-source web scraping and automation framework by Ampify and it supports three different types of uh crawlers. Those are Cheeriocrawler, Puppeteer crawler and
[0:51]Playright crawler. You might know some of these but and these are in integrated into crawly and all these are integrated uh in the same interface. So when you're switching between these three uh crawlers inside crawly it's really simple and due to all these options it is really easy to scrape JavaScript heavy sites and also HTTP based simple crawl uh sites. Okay. Now let's review a small piece of code and let's see how crawly actually works. We are using the playright crawler. So we have imported playright crawler from crawly and playright itself. Uh we have actually chromium from playright. Okay. Now let's check the first function which
[1:43]Is ensure browser. This this ensures that chromium is installed and if needed it installs it. This also ensures that we have the necessary browser for scraping. Let's check out the configuration object for crawler. This basically sets the parameters and the browser launch options. We have parameters like max requests per crawl. So it will right now it will crawl through like 50 pages and uh get its data. We can just decrease it to 10 for now. You can change it to any amount. And in the launch options, we have headless. So with headless, you basically the crawling happens in the background and you don't see anything on
[2:30]Screen. If you set the headless to false, you will see a brow browser getting simulated and you will see what pages are getting crawled um in like in action. Next is we initialize the playright crawler. In this we uh take a let's take a look at the request handler. What does this do? It waits for the page to load fully. Then it extracts the page title, metadata, description, paragraphs and links. Saves the data to a data set. This is what it does. It saves the results to a data data set. Then we check for links in the page and we save those links in a que
[3:20]And so when so we have so the queue length will max go till 10. Basically we keep on checking uh links from the queue and uh keep adding more links to the back of the queue. After that we have a very simple um error handling where it will just log out the errors if it faced any during the crawling process and like that is all. So we can just start the crawling uh start crawler function and it will crawl through the uh website. Right now we will just crawl through Wikipedia and the page on web scraping. I think that would be good. Okay. So before starting the crawler I
[4:09]Have like a small suggestion on some websites there are like rate limits and if you exceed those limits they can give you a like a IP ban. So it's better to use a uh like a VPN while doing the scraping. Um it avoids the IP bands and those are kind like really bad. If you get some IP ban from some website you will have a lot like lots of trouble regarding that. So now you can get this VPN. This is free ProtonVPN. You can just download it and you can use the free tier as much as you want. Okay. To start the crawler, we'll just
[4:52]Write nodecrawler.js. That's it. As you can see, the crawler is starting. It went to the website. And you can just you can see all the links that it is going through. As you can see it reached max request per crawl that is 10 which we just set to 10 over here. Yes. And it has gone through 10 pages. We can take a look at those pages over here where we have the data set of the all the data all the links the paragraphs the title URL in in the first page then we can go on to the second page there's Wikipedia main page Wikipedia contents
[5:46]All the data we have got from these pages and there are a few others um files which it has generated This is regarding the crawling sessions and the statistics. So that's it. That's like simply really simple like 50 like around 50-ish lines of code and it does all that so easily. I'll just do a small showcase with headless false. I will I will start it with js. Okay, right there. Now you can see the website is getting crawled. As you can see the first website was scraper. So on web scraping and it is going through all the pages all the different pages and it just scraped through it. It just did the
[6:48]Same thing but we just saw like how it was scraping through all the pages which is kind of interesting. We will go through some of the use cases for crawly like where does crawly shine. Crawley can be used for data collection, market research, academic studies, AI training. It also helps with SEO monitoring like tracking rankings, keywords, competitive strategies. It it helps with e-commerce and price tracking and it also helps with leads generations and content aggregation like gathering news, blogs and social media insights. Al what are like the advantages? Why does Crawley stand out? It has an antibot feature which handles rate limits, retries and capture solving. It
[7:34]Is highly scalable and supports multiple crawlers with Q-based architecture. It also supports headless browser support which I just showed right now in the like just now and with the code and it is very flexible. It has different middleware which allows custom logic for data processing. It has a strong community and documentation backed by ampify with solid support. But no tool is perfect. So here are some limitations of Crawley. It has a steeper learning curve for beginners. It might be a bit complex for beginners. it is a bit resource intensive. So if you're doing uh like huge amounts of crawling, you better have like a good CPU and lots of
[8:19]RAM and it's not ideal for API scraping as there are uh lots of API which is readily available and it will and those might really help you to get the data in a more efficient way than using a crawl crawler like crawly. Now let's take a look at some of the alternatives which um which are there and how crawly is better than these alternatives. So first one we'll take a look is uh at scrapey. It's better for static HTML but Crawley handles JavaScript better. Next let's take a look at Selenium. Selenium is for testing and Crawley excels in large scale scraping. Then we have beautiful beautiful soup
[9:11]Which is great for parsing but lacks the automation capabilities of crawly. But there are also some places that these tools excel like for simple HTML scraping beautiful soup is a lot better and it's much more efficient then just like I told you some websites have their own APIs where you can get the data from. So that is lot better instead of scraping the website. And in websites where you have to play with the GUI, Selenium or Playright comes on the top. So let's conclude this. Crawley.dev is a powerful tool for scalable web scraping and automation. If you found this video helpful, like, subscribe, and hit the
[9:59]Bell icon for more tech insights. See you in the next one.
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53