Git Stars Git Stars

#crawler Repositories

Discover 162 active open source GitHub repositories tagged with #crawler, accumulating 1,111,814 total stars (averaging 6,863 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.

162
Total Repositories
1.1M
Total Stars
6,863
Avg Stars/Repo

162 repositories total

1
firecrawl/firecrawl

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

firecrawl
⭐ 205,758 TypeScript 👤 firecrawl Organization 🕐 Updated 2026-11-21 📅 Created 2024-04-15
2
D4Vinci/Scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy—join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

D4Vinci
⭐ 104,760 Python 👤 D4Vinci User 🕐 Updated 2026-11-17 📅 Created 2024-10-13
3
unclecode/crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

unclecode
⭐ 87,627 Python 👤 unclecode User 🕐 Updated 2026-11-20 📅 Created 2024-05-09
4
scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy
⭐ 65,519 Python 👤 scrapy Organization 🕐 Updated 2026-11-19 📅 Created 2010-02-22
5
NaiboWang/EasySpider

A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.

NaiboWang
⭐ 44,801 JavaScript 👤 NaiboWang User 🕐 Updated 2026-11-16 📅 Created 2020-07-18
6
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

ScrapeGraphAI
⭐ 32,877 Python 👤 ScrapeGraphAI Organization 🕐 Updated 2026-10-06 📅 Created 2024-01-27
7
iawia002/lux

👾 Fast and simple video download library and CLI tool written in Go

iawia002
⭐ 31,854 Go 👤 iawia002 User 🕐 Updated 2026-03-29 📅 Created 2018-02-24
8
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
⭐ 26,510 TypeScript 👤 apify Organization 🕐 Updated 2026-11-21 📅 Created 2016-08-26
9
gocolly/colly

Elegant Scraper and Crawler Framework for Golang

gocolly
⭐ 25,608 Go 👤 gocolly Organization 🕐 Updated 2026-10-16 📅 Created 2017-09-29
10
jhao104/proxy_pool

Python ProxyPool for web spider

jhao104
⭐ 23,798 Python 👤 jhao104 User 🕐 Updated 2026-06-15 📅 Created 2016-11-25
11
Evil0ctal/Douyin_TikTok_Download_API

🚀 Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader — asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.

Evil0ctal
⭐ 21,507 Python 👤 Evil0ctal User 🕐 Updated 2026-11-20 📅 Created 2021-11-07
12
getmaxun/maxun

Turn any website into a structured API. Extract, automate, search and monitor the web.

getmaxun
⭐ 18,538 TypeScript 👤 getmaxun Organization 🕐 Updated 2026-11-20 📅 Created 2023-10-23
13
projectdiscovery/katana

A next-generation crawling and spidering framework.

projectdiscovery
⭐ 17,878 Go 👤 projectdiscovery Organization 🕐 Updated 2026-11-21 📅 Created 2021-01-02
14
codelucas/newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

codelucas
⭐ 15,192 Python 👤 codelucas User 🕐 Updated 2026-10-26 📅 Created 2013-11-25
15
shengqiangzhang/examples-of-web-crawlers

Some very interesting Python crawler examples, friendly to beginners, mainly crawling websites like Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ, etc. (Some interesting examples of python crawlers that are friendly to beginners.)

shengqiangzhang
⭐ 14,654 HTML 👤 shengqiangzhang User 🕐 Updated 2025-06-28 📅 Created 2019-03-13
16
s0md3v/Photon

Incredibly fast crawler designed for OSINT.

s0md3v
⭐ 13,377 Python 👤 s0md3v User 🕐 Updated 2026-10-17 📅 Created 2018-03-30
17
crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

crawlab-team
⭐ 12,301 Go 👤 crawlab-team Organization 🕐 Updated 2026-02-10 📅 Created 2019-02-10
18
code4craft/webmagic

A scalable web crawler framework for Java.

code4craft
⭐ 11,660 Java 👤 code4craft User 🕐 Updated 2025-12-20 📅 Created 2013-04-23
19
injetlee/Python

Python script. Simulate login to Zhihu, web crawling, manipulate Excel, WeChat public account, remote power‑on.

injetlee
⭐ 10,920 Python 👤 injetlee User 🕐 Updated 2026-05-09 📅 Created 2016-06-23
20
guyueyingmu/avbook

AV Movie Management System, avmoo, javbus, javlibrary Crawler, Online AV Film Library, AV Magnet Link Database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database

guyueyingmu
⭐ 9,894 PHP 👤 guyueyingmu User 🕐 Updated 2024-06-01 📅 Created 2017-02-19
21
apify/crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
⭐ 9,679 Python 👤 apify Organization 🕐 Updated 2026-11-20 📅 Created 2024-01-10
22
TeamWiseFlow/xiaobei

Self-media customer acquisition agent tailored for OPCs and SMEs

TeamWiseFlow
⭐ 8,743 Python 👤 TeamWiseFlow Organization 🕐 Updated 2026-11-20 📅 Created 2024-04-24
23
lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

lorien
⭐ 8,242 JavaScript 👤 lorien User 🕐 Updated 2026-11-21 📅 Created 2015-08-12
24
hect0x7/JMComic-Crawler-Python

Python API for JMComic | Provides Python API access to JinManTiantang, supporting both web and mobile versions | JinManTiantang GitHub Actions Downloader🚀

hect0x7
⭐ 8,148 Python 👤 hect0x7 User 🕐 Updated 2026-11-13 📅 Created 2023-04-02
25
alirezamika/autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

alirezamika
⭐ 8,102 Python 👤 alirezamika User 🕐 Updated 2026-07-29 📅 Created 2020-08-31
26
mvdctop/Movie_Data_Capture

The AI local media metadata crawler

mvdctop
⭐ 7,449 Python 👤 mvdctop User 🕐 Updated 2026-11-17 📅 Created 2019-05-26
27
adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

adbar
⭐ 7,157 Python 👤 adbar User 🕐 Updated 2026-11-20 📅 Created 2019-04-08
28
BruceDone/awesome-crawler

A collection of awesome web crawler,spider in different languages

BruceDone
⭐ 7,051 👤 BruceDone User 🕐 Updated 2024-06-16 📅 Created 2016-10-10
29
bda-research/node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

bda-research
⭐ 6,797 TypeScript 👤 bda-research Organization 🕐 Updated 2026-06-18 📅 Created 2010-11-25
30
chyroc/WechatSogou

WeChat Official Account Crawler API based on Sogou WeChat Search

chyroc
⭐ 6,411 Python 👤 chyroc User 🕐 Updated 2026-10-23 📅 Created 2016-05-09
31
rmax/scrapy-redis

Redis-based components for Scrapy.

rmax
⭐ 5,648 Python 👤 rmax User 🕐 Updated 2026-11-21 📅 Created 2011-08-29
32
DropsDevopsOrg/ECommerceCrawlers

Practical 🐍 multiple websites, e-commerce data crawlers 🕷. Includes 🕸: Taobao products, WeChat public accounts, Dianping, Qichacha, job websites, Xianyu, Ali tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu.com, Quan jing wang, Douban music, some province's drug supervision bureau, Sohu news, machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword inclusion count, spider general directory, Toutiao, Douban movie reviews, Ctrip, Xiaomi app store, Anjuke, Tujia homestay ❤️❤️❤️. WeChat crawler demo project:

DropsDevopsOrg
⭐ 5,355 Python 👤 DropsDevopsOrg Organization 🕐 Updated 2024-05-22 📅 Created 2019-03-29
33
DedSecInside/TorBot

Dark Web OSINT Tool

DedSecInside
⭐ 5,220 Python 👤 DedSecInside Organization 🕐 Updated 2026-11-12 📅 Created 2017-05-17
34
niespodd/browser-fingerprinting

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

niespodd
⭐ 5,171 JavaScript 👤 niespodd User 🕐 Updated 2026-07-27 📅 Created 2021-01-23
35
hiddendevj/Crawler_Illegal_Cases_In_China

This project is designed to compile all news, documentation, and legal regulations related to lawsuits and violations involving web crawler developers in mainland China. It aims to help professionals in the web crawler industry working in mainland China understand relevant laws and avoid infringing data compliance boundaries.

hiddendevj
⭐ 4,759 HTML 👤 hiddendevj User 🕐 Updated 2026-03-12 📅 Created 2019-09-29
36
dataabc/weibo-crawler

Sina Weibo Crawler, using Python to crawl Sina Weibo data, and download Weibo images and videos

dataabc
⭐ 4,705 Python 👤 dataabc User 🕐 Updated 2026-11-06 📅 Created 2019-06-25
37
myreader-io/myGPTReader

A community-driven way to read and chat with AI bots - powered by chatGPT.

myreader-io
⭐ 4,416 Python 👤 myreader-io Organization 🕐 Updated 2026-02-05 📅 Created 2023-03-08
38
kanasimi/work_crawler

Download comics novels tool:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.

kanasimi
⭐ 4,391 JavaScript 👤 kanasimi User 🕐 Updated 2026-10-12 📅 Created 2016-11-02
39
dotnetcore/DotnetSpider

DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

dotnetcore
⭐ 4,142 C# 👤 dotnetcore Organization 🕐 Updated 2026-04-03 📅 Created 2016-03-21
40
constverum/ProxyBroker

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

constverum
⭐ 4,084 Python 👤 constverum User 🕐 Updated 2024-03-18 📅 Created 2015-10-11
41
MikeChongCan/scylla

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

MikeChongCan
⭐ 4,021 Python 👤 MikeChongCan User 🕐 Updated 2026-11-03 📅 Created 2018-04-10
42
Arachni/arachni

Web Application Security Scanner Framework

Arachni
⭐ 4,020 Ruby 👤 Arachni Organization 🕐 Updated 2026-04-22 📅 Created 2010-08-25
43
hardkoded/puppeteer-sharp

Headless Chrome .NET API

hardkoded
⭐ 3,936 C# 👤 hardkoded Organization 🕐 Updated 2026-11-21 📅 Created 2017-09-30
44
edoardottt/cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

edoardottt
⭐ 3,817 Go 👤 edoardottt User 🕐 Updated 2026-11-19 📅 Created 2021-04-27
45
3441293738/creatorhub

Multi-platform content monitoring · collection · reposting. One web dashboard to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok / Bilibili / Instagram / Xiaohongshu overseas (rednote)

3441293738
⭐ 3,774 Python 👤 3441293738 User 🕐 Updated 2026-11-21 📅 Created 2026-07-03
46
Boris-code/feapder

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.

Boris-code
⭐ 3,732 Python 👤 Boris-code User 🕐 Updated 2026-08-21 📅 Created 2021-02-08
47
sqzw-x/mdcx

Movie metadata scraper

sqzw-x
⭐ 3,727 Python 👤 sqzw-x User 🕐 Updated 2026-08-23 📅 Created 2023-11-06
48
elliotgao2/toapi

Every web site provides APIs.

elliotgao2
⭐ 3,557 Python 👤 elliotgao2 User 🕐 Updated 2026-08-27 📅 Created 2017-11-27
49
Tuhinshubhra/RED_HAWK

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

Tuhinshubhra
⭐ 3,491 PHP 👤 Tuhinshubhra User 🕐 Updated 2024-07-31 📅 Created 2017-06-11
50
CharlesPikachu/videodl

Videodl: A lightweight video downloader written in pure Python. (A lightweight video downloader that prioritizes high-definition, watermark‑free downloads and supports platforms such as Douyin, Kuaishou, Xiaohongshu, Bilibili, TikTok, YouTube, FIFA+, Youku, Tencent Video, iQIYI, 1905 Movie Network, LeTV, Mango TV, Migu, PPTV, Sohu, Facebook, Twitter, Sina Weibo, Today’s Headlines, NetEase Open Classes, 全民K歌, CCTV, Kugou MV, New Film, Zhihu, Baidu Tieba, TED, and other vast streaming platforms.)

CharlesPikachu
⭐ 3,294 Python 👤 CharlesPikachu User 🕐 Updated 2026-11-21 📅 Created 2018-07-10

Frequently Asked Questions

1

What is the most popular #crawler repository on GitHub?

The top ranked open source #crawler project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

2

Which programming languages are most used in #crawler repositories?

Among the top ranked #crawler projects on GitHub, the most common programming languages are Python, Go, TypeScript, JavaScript.

3

How is the #crawler repository ranking calculated?

We track 162 active open source repositories tagged with #crawler (500+ stars), totaling 1,111,814 stars with an average of 6,863 stars per repository. Rankings update daily from the GitHub API.