Git Stars Git Stars

#crawler Repositories

162
Total Repositories
1.1M
Total Stars
6,863
Avg Stars/Repo
51
thewhiteh4t/FinalRecon

All In One Web Recon

thewhiteh4t
โญ 3,058 Python ๐Ÿ‘ค thewhiteh4t User ๐Ÿ• Updated 2026-10-31 ๐Ÿ“… Created 2019-03-28
52
Qianlitp/crawlergo

A powerful browser crawler for web vulnerability scanners

Qianlitp
โญ 3,034 Go ๐Ÿ‘ค Qianlitp User ๐Ÿ• Updated 2025-03-11 ๐Ÿ“… Created 2019-12-25
53
facundoolano/google-play-scraper

Node.js scraper to get data from Google Play

facundoolano
โญ 3,005 JavaScript ๐Ÿ‘ค facundoolano User ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2015-04-07
54
howie6879/owllook

owllook-Novel Search Engine

howie6879
โญ 2,900 Python ๐Ÿ‘ค howie6879 User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2017-02-18
55
CharlesPikachu/DecryptLogin

DecryptLogin: APIs for loginning some websites by using requests.

CharlesPikachu
โญ 2,871 Python ๐Ÿ‘ค CharlesPikachu User ๐Ÿ• Updated 2024-08-09 ๐Ÿ“… Created 2019-02-25
56
spider-rs/spider

Foundational low-latency web data collection in Rust

spider-rs
โญ 2,856 Rust ๐Ÿ‘ค spider-rs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-07
57
jaeles-project/gospider

Gospider - Fast web spider written in Go

jaeles-project
โญ 2,851 Go ๐Ÿ‘ค jaeles-project Organization ๐Ÿ• Updated 2024-04-21 ๐Ÿ“… Created 2020-01-22
58
spatie/crawler

https://spatie.be/docs/crawler

spatie
โญ 2,827 PHP ๐Ÿ‘ค spatie Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2015-11-02
59
geziyor/geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

geziyor
โญ 2,781 Go ๐Ÿ‘ค geziyor Organization ๐Ÿ• Updated 2026-07-02 ๐Ÿ“… Created 2019-06-06
60
jae-jae/QueryList

:spider: The progressive PHP crawler framework! ไผ˜้›…็š„ๆธ่ฟ›ๅผPHP้‡‡้›†ๆก†ๆžถใ€‚

jae-jae
โญ 2,689 PHP ๐Ÿ‘ค jae-jae User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2015-12-20
61
Autumn-27/ScopeSentry

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

Autumn-27
โญ 2,597 Go ๐Ÿ‘ค Autumn-27 User ๐Ÿ• Updated 2026-10-11 ๐Ÿ“… Created 2024-02-27
62
xtuhcy/gecco

Easy to use lightweight web crawler

xtuhcy
โญ 2,510 Java ๐Ÿ‘ค xtuhcy User ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2015-12-12
63
fhamborg/news-please

news-please - an integrated web crawler and information extractor for news that just works

fhamborg
โญ 2,494 Python ๐Ÿ‘ค fhamborg User ๐Ÿ• Updated 2026-04-14 ๐Ÿ“… Created 2016-12-18
64
lorien/grab

Web Scraping Framework

lorien
โญ 2,461 Python ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2025-09-19 ๐Ÿ“… Created 2013-05-01
65
JayBizzle/Crawler-Detect

๐Ÿ•ท CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

JayBizzle
โญ 2,410 PHP ๐Ÿ‘ค JayBizzle User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2015-03-23
66
friuns2/Leaked-GPTs

Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.

friuns2
โญ 2,396 Python ๐Ÿ‘ค friuns2 User ๐Ÿ• Updated 2024-01-17 ๐Ÿ“… Created 2023-11-27
67
sjdirect/abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

sjdirect
โญ 2,308 C# ๐Ÿ‘ค sjdirect User ๐Ÿ• Updated 2026-07-17 ๐Ÿ“… Created 2013-08-08
68
goclone-dev/goclone

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

goclone-dev
โญ 2,303 Go ๐Ÿ‘ค goclone-dev Organization ๐Ÿ• Updated 2026-06-02 ๐Ÿ“… Created 2020-03-23
69
AAndyProgram/SCrawler

Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.

AAndyProgram
โญ 2,263 Visual Basic .NET ๐Ÿ‘ค AAndyProgram User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2021-12-07
70
watercrawl/WaterCrawl

Transform Web Content into LLM-Ready Data

watercrawl
โญ 2,244 TypeScript ๐Ÿ‘ค watercrawl Organization ๐Ÿ• Updated 2026-08-17 ๐Ÿ“… Created 2024-12-15
71
zorlan/skycaiji

Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.

zorlan
โญ 2,098 PHP ๐Ÿ‘ค zorlan User ๐Ÿ• Updated 2026-10-07 ๐Ÿ“… Created 2018-02-26
72
TeamNewPipe/NewPipeExtractor

NewPipe's core library for extracting data from streaming sites

TeamNewPipe
โญ 2,070 Java ๐Ÿ‘ค TeamNewPipe Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2017-03-01
73
elliotgao2/gain

Web crawling framework based on asyncio.

elliotgao2
โญ 2,017 Python ๐Ÿ‘ค elliotgao2 User ๐Ÿ• Updated 2026-08-09 ๐Ÿ“… Created 2017-05-31
74
coder-hxl/x-crawl

Flexible Node.js crawler library

coder-hxl
โญ 1,888 TypeScript ๐Ÿ‘ค coder-hxl User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-01-22
75
JSREI/ast-hook-for-js-RE

Browser memory roaming solution (exploring...)

JSREI
โญ 1,882 JavaScript ๐Ÿ‘ค JSREI Organization ๐Ÿ• Updated 2024-05-07 ๐Ÿ“… Created 2021-01-15
76
lixi5338619/lxSpider

Collection of crawler cases. Including but not limited to "Taobao, JD.com, Tmall, Douban, Douyin, Kuaishou, Weibo, WeChat, Alibaba, Toutiao, PDD, Youku, iQIYI, Ctrip, 12306, 58, Sohu, various indexes, VIP, Wanfang, ZLibrary, Oalib, Novels, bidding websites, procurement websites, Xiaohongshu, Dianping, Twitter, Maimai, Zhihu"

lixi5338619
โญ 1,876 Python ๐Ÿ‘ค lixi5338619 User ๐Ÿ• Updated 2024-05-09 ๐Ÿ“… Created 2021-01-29
77
diskoverdata/diskover-community

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

diskoverdata
โญ 1,831 PHP ๐Ÿ‘ค diskoverdata Organization ๐Ÿ• Updated 2026-03-25 ๐Ÿ“… Created 2017-04-19
78
eatmoreduck/boss-zhipin-scraper

BOSS Zhipin scraper / BOSS Zhipin job data crawler, based on Chrome CDP protocol to reuse real login sessions, bypass font anti-scraping, and output plaintext salary JSON/CSV + salary-skill analysis.

eatmoreduck
โญ 1,781 Python ๐Ÿ‘ค eatmoreduck User ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-06-11
79
u3c3/BT-btt

Introduction to the magnetic website U3C3 and domain update

u3c3
โญ 1,779 ๐Ÿ‘ค u3c3 User ๐Ÿ• Updated 2025-01-01 ๐Ÿ“… Created 2020-04-28
80
howie6879/ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

howie6879
โญ 1,738 Python ๐Ÿ‘ค howie6879 User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2018-07-10
81
rushter/selectolax

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

rushter
โญ 1,717 Python ๐Ÿ‘ค rushter User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2017-11-26
82
gildas-lormeau/single-file-cli

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

gildas-lormeau
โญ 1,702 JavaScript ๐Ÿ‘ค gildas-lormeau User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-05-31
83
YoongiKim/AutoCrawler

Google, Naver multiprocess image web crawler (Selenium)

YoongiKim
โญ 1,700 Python ๐Ÿ‘ค YoongiKim User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2018-11-21
84
orangecoding/fredy

โค๏ธ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

orangecoding
โญ 1,648 JavaScript ๐Ÿ‘ค orangecoding User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-01-15
85
erma0/douyin

Douyin Crawler - Collect public data such as account homepage, likes, collections, music original sound, topics, search, collections, works, following, followers, etc.

erma0
โญ 1,625 ๐Ÿ‘ค erma0 User ๐Ÿ• Updated 2026-08-14 ๐Ÿ“… Created 2022-05-29
86
ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

ArchiveTeam
โญ 1,572 Python ๐Ÿ‘ค ArchiveTeam Organization ๐Ÿ• Updated 2025-05-23 ๐Ÿ“… Created 2015-02-05
87
srx-2000/spider_collection

Python crawlers, current inventory: NetEase Cloud Music song crawling, Bilibili video crawling, Zhihu Q&A crawling, wallpaper crawling, xvideos video crawling, audiobook crawling, Weibo crawler, Anjuke information crawling + data visualization, Bilibili video cover extractor, IP proxy pool encapsulation, Zhihu million-level user crawler + data analysis, GitHub user crawler

srx-2000
โญ 1,554 Python ๐Ÿ‘ค srx-2000 User ๐Ÿ• Updated 2024-04-23 ๐Ÿ“… Created 2020-02-04
88
0x676e67/wreq-python

An ergonomic, privacy-aware Python HTTP Client

0x676e67
โญ 1,526 Python ๐Ÿ‘ค 0x676e67 User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2025-02-10
89
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

rebrowser
โญ 1,468 JavaScript ๐Ÿ‘ค rebrowser Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2024-08-06
90
dadoonet/fscrawler

Elasticsearch File System Crawler (FS Crawler)

dadoonet
โญ 1,459 Java ๐Ÿ‘ค dadoonet User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2012-06-08
91
xisuo67/XHS-Spider

Xiaohongshu data collection, website image, video resource batch download tool, super high appearance data collection tool (batch download, video extraction, images) Telegram:https://t.me/+ZtLSwuIKTo44MDY1

xisuo67
โญ 1,456 ๐Ÿ‘ค xisuo67 User ๐Ÿ• Updated 2025-12-16 ๐Ÿ“… Created 2023-04-25
92
openwpm/OpenWPM

A web privacy measurement framework

openwpm
โญ 1,429 Python ๐Ÿ‘ค openwpm Organization ๐Ÿ• Updated 2026-10-13 ๐Ÿ“… Created 2014-06-10
93
darbra/sperm

A summary of excellent reverse engineering articles, worth a look

darbra
โญ 1,427 ๐Ÿ‘ค darbra User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2022-03-02
94
kgspider/crawler

Brother K Web Scraping Code Sharing, JS Reverse Engineering, Advanced Web Scraping. Follow the WeChat official account: Brother K Web Scraping

kgspider
โญ 1,371 JavaScript ๐Ÿ‘ค kgspider User ๐Ÿ• Updated 2025-05-12 ๐Ÿ“… Created 2021-08-24
95
lorey/mlscraper

๐Ÿค– Scrape data from HTML websites automatically by just providing examples

lorey
โญ 1,370 Python ๐Ÿ‘ค lorey User ๐Ÿ• Updated 2024-03-17 ๐Ÿ“… Created 2020-07-30
96
felipecsl/wombat

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

felipecsl
โญ 1,360 Ruby ๐Ÿ‘ค felipecsl User ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2011-11-14
97
0xInfection/XSRFProbe

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

0xInfection
โญ 1,310 Python ๐Ÿ‘ค 0xInfection User ๐Ÿ• Updated 2026-07-12 ๐Ÿ“… Created 2018-08-21
98
fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

fastcrw
โญ 1,284 Rust ๐Ÿ‘ค fastcrw Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-02
99
dwisiswant0/go-dork

The fastest dork scanner written in Go.

dwisiswant0
โญ 1,266 Go ๐Ÿ‘ค dwisiswant0 User ๐Ÿ• Updated 2024-02-04 ๐Ÿ“… Created 2020-07-13
100
kiddyuchina/Beanbun

Beanbun is a multi-process web crawling framework written in PHP, with good openness and high scalability, based on Workerman.

kiddyuchina
โญ 1,258 PHP ๐Ÿ‘ค kiddyuchina User ๐Ÿ• Updated 2026-05-26 ๐Ÿ“… Created 2017-04-13