Git Stars Git Stars

#crawling Repositories

Discover 39 active open source GitHub repositories tagged with #crawling, accumulating 335,438 total stars (averaging 8,600 stars per project). Leading this topic is D4Vinci/Scrapling with 104,760 stars.

39
Total Repositories
335,438
Total Stars
8,600
Avg Stars/Repo
1
D4Vinci/Scrapling

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโ€”join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

D4Vinci
โญ 104,760 Python ๐Ÿ‘ค D4Vinci User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2024-10-13
2
scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy
โญ 65,519 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2010-02-22
3
apify/crawlee

Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
โญ 26,510 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-26
4
gocolly/colly

Elegant Scraper and Crawler Framework for Golang

gocolly
โญ 25,608 Go ๐Ÿ‘ค gocolly Organization ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2017-09-29
5
getmaxun/maxun

Turn any website into a structured API. Extract, automate, search and monitor the web.

getmaxun
โญ 18,538 TypeScript ๐Ÿ‘ค getmaxun Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-23
6
codelucas/newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

codelucas
โญ 15,192 Python ๐Ÿ‘ค codelucas User ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2013-11-25
7
apify/crawlee-python

Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
โญ 9,679 Python ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-01-10
8
lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

lorien
โญ 8,242 JavaScript ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-08-12
9
go-rod/rod

A Chrome DevTools Protocol driver for web automation and scraping.

go-rod
โญ 7,160 Go ๐Ÿ‘ค go-rod Organization ๐Ÿ• Updated 2026-08-11 ๐Ÿ“… Created 2020-01-21
10
exa-labs/exa-mcp-server

Exa MCP for web search and web crawling!

exa-labs
โญ 5,250 TypeScript ๐Ÿ‘ค exa-labs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-11-27
11
hakluke/hakrawler

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

hakluke
โญ 5,171 Go ๐Ÿ‘ค hakluke User ๐Ÿ• Updated 2026-08-05 ๐Ÿ“… Created 2019-12-15
12
ai-robots-txt/ai.robots.txt

A list of AI agents and robots to block.

ai-robots-txt
โญ 4,265 Python ๐Ÿ‘ค ai-robots-txt Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2024-03-27
13
hardkoded/puppeteer-sharp

Headless Chrome .NET API

hardkoded
โญ 3,936 C# ๐Ÿ‘ค hardkoded Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2017-09-30
14
edoardottt/cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

edoardottt
โญ 3,817 Go ๐Ÿ‘ค edoardottt User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2021-04-27
15
apache/nutch

Apache Nutch is an extensible and scalable web crawler

apache
โญ 3,311 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2009-05-21
16
transitive-bullshit/awesome-puppeteer

A curated list of awesome puppeteer resources.

transitive-bullshit
โญ 2,528 ๐Ÿ‘ค transitive-bullshit User ๐Ÿ• Updated 2024-07-19 ๐Ÿ“… Created 2018-05-07
17
lorien/grab

Web Scraping Framework

lorien
โญ 2,461 Python ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2025-09-19 ๐Ÿ“… Created 2013-05-01
18
NateScarlet/holiday-cn

๐Ÿ“…๐Ÿ‡จ๐Ÿ‡ณ China Statutory Holidays Data Automatically Fetched Daily from State Council Announcements

NateScarlet
โญ 2,293 Python ๐Ÿ‘ค NateScarlet User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-03-05
19
zorlan/skycaiji

Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.

zorlan
โญ 2,098 PHP ๐Ÿ‘ค zorlan User ๐Ÿ• Updated 2026-10-07 ๐Ÿ“… Created 2018-02-26
20
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

rebrowser
โญ 1,468 JavaScript ๐Ÿ‘ค rebrowser Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2024-08-06
21
roach-php/core

The complete web scraping toolkit for PHP.

roach-php
โญ 1,456 PHP ๐Ÿ‘ค roach-php Organization ๐Ÿ• Updated 2025-10-27 ๐Ÿ“… Created 2021-08-17
22
lorey/mlscraper

๐Ÿค– Scrape data from HTML websites automatically by just providing examples

lorey
โญ 1,370 Python ๐Ÿ‘ค lorey User ๐Ÿ• Updated 2024-03-17 ๐Ÿ“… Created 2020-07-30
23
webrecorder/browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

webrecorder
โญ 1,199 TypeScript ๐Ÿ‘ค webrecorder Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2020-11-02
24
needleworm/bhban_rpa

This is the sample code from <6๊ฐœ์›” ์น˜ ์—…๋ฌด๋ฅผ ํ•˜๋ฃจ ๋งŒ์— ๋๋‚ด๋Š” ์—…๋ฌด ์ž๋™ํ™”(์ƒ๋Šฅ์ถœํŒ์‚ฌ, 2020)>. It is an example designed for people who have never learned Python, and it provides examples in various fields related to task automation, ranging from Excel to design, macros, and crawling.

needleworm
โญ 1,153 Python ๐Ÿ‘ค needleworm User ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2020-02-11
25
dondai44423/donsetch

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

dondai44423
โญ 1,138 Rust ๐Ÿ‘ค dondai44423 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-08-07
26
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
27
elixir-crawly/crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

elixir-crawly
โญ 1,129 Elixir ๐Ÿ‘ค elixir-crawly Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-09
28
janreges/siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโ€”ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

janreges
โญ 1,019 Rust ๐Ÿ‘ค janreges User ๐Ÿ• Updated 2026-09-27 ๐Ÿ“… Created 2023-10-04
29
clemfromspace/scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

clemfromspace
โญ 950 Python ๐Ÿ‘ค clemfromspace User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2018-02-11
30
mishakorzik/AdminHack

today we will hack the admin panel of the site.

mishakorzik
โญ 932 Shell ๐Ÿ‘ค mishakorzik User ๐Ÿ• Updated 2026-09-10 ๐Ÿ“… Created 2021-05-16
31
scrapinghub/scrapyrt

HTTP API for Scrapy spiders

scrapinghub
โญ 884 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2026-10-13 ๐Ÿ“… Created 2015-01-06
32
StJudeWasHere/seonaut

Open source SEO audit tool.

StJudeWasHere
โญ 873 Go ๐Ÿ‘ค StJudeWasHere User ๐Ÿ• Updated 2026-10-08 ๐Ÿ“… Created 2022-03-02
33
MorvanZhou/easy-scraping-tutorial

Simple but useful Python web scraping tutorial code.

MorvanZhou
โญ 814 Jupyter Notebook ๐Ÿ‘ค MorvanZhou User ๐Ÿ• Updated 2024-04-07 ๐Ÿ“… Created 2017-12-29
34
josephlimtech/linkedin-profile-scraper-api

๐Ÿ•ต๏ธโ€โ™‚๏ธ LinkedIn profile scraper returning structured profile data in JSON.

josephlimtech
โญ 725 TypeScript ๐Ÿ‘ค josephlimtech User ๐Ÿ• Updated 2024-04-05 ๐Ÿ“… Created 2018-12-21
35
lumpinif/deepcrawl

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

lumpinif
โญ 704 TypeScript ๐Ÿ‘ค lumpinif User ๐Ÿ• Updated 2026-08-24 ๐Ÿ“… Created 2025-05-22
36
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17
37
l4rm4nd/LinkedInDumper

Python 3 script to dump/scrape/extract company employees from LinkedIn API

l4rm4nd
โญ 624 Python ๐Ÿ‘ค l4rm4nd User ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2022-10-17
38
scrapinghub/spidermon

Scrapy Extension for monitoring spiders execution.

scrapinghub
โญ 563 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2015-11-17
39
zhuyingda/webster

a reliable high-level web crawling & scraping framework for Node.js.

zhuyingda
โญ 560 JavaScript ๐Ÿ‘ค zhuyingda User ๐Ÿ• Updated 2025-03-12 ๐Ÿ“… Created 2017-11-04

Frequently Asked Questions

1

What is the most popular #crawling repository on GitHub?

The top ranked open source #crawling project on GitHub is D4Vinci/Scrapling with 104,760 stars. ๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโ€”join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

2

Which programming languages are most used in #crawling repositories?

Among the top ranked #crawling projects on GitHub, the most common programming languages are Python, TypeScript, Go, JavaScript.

3

How is the #crawling repository ranking calculated?

We track 39 active open source repositories tagged with #crawling (500+ stars), totaling 335,438 stars with an average of 8,600 stars per repository. Rankings update daily from the GitHub API.