Git Stars Git Stars

#crawler Repositories

162
Total Repositories
1.1M
Total Stars
6,863
Avg Stars/Repo
151
fengzhizi715/NetDiscovery

NetDiscovery is a general-purpose crawler framework/middleware implemented based on frameworks such as Vert.x and RxJava 2.

fengzhizi715
โญ 643 Java ๐Ÿ‘ค fengzhizi715 User ๐Ÿ• Updated 2026-06-05 ๐Ÿ“… Created 2017-11-26
152
cv-cat/catbus

One command to operate Xiaohongshu, Douyin, TikTok, Bilibili, Kuaishou, Weibo, Xianyu, Taobao, JD.com, and X: unified commands and JSON output for people and AI agents

cv-cat
โญ 641 TypeScript ๐Ÿ‘ค cv-cat User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-27
153
rajatomar788/pywebcopy

Locally saves webpages to your hard disk with images, css, js & links as is.

rajatomar788
โญ 639 Python ๐Ÿ‘ค rajatomar788 User ๐Ÿ• Updated 2025-05-13 ๐Ÿ“… Created 2018-07-23
154
scrapfly/Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

scrapfly
โญ 636 JavaScript ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2025-09-03
155
yhy0/Jie

Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features encompasses vulnerability scanning, information gathering, and exploitation, elevating it to an indispensable toolkit for both security professionals and penetration testers. ๆŒ–ๆดž่พ…ๅŠฉๅทฅๅ…ท(ๆผๆดžๆ‰ซๆใ€ไฟกๆฏๆ”ถ้›†)

yhy0
โญ 599 Go ๐Ÿ‘ค yhy0 User ๐Ÿ• Updated 2024-07-05 ๐Ÿ“… Created 2023-01-30
156
CharlesPikachu/freeproxy

FreeProxy: Collecting free proxies from the internet. (Massive high-quality free proxies worldwide, supporting crawling from dozens of free proxy sharing sources, custom rule-based proxy filtering, essential for web scraping and data analysis, updated daily with a vast number of free proxies.)

CharlesPikachu
โญ 594 Python ๐Ÿ‘ค CharlesPikachu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-10-29
157
vakra-dev/reader

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

vakra-dev
โญ 577 TypeScript ๐Ÿ‘ค vakra-dev Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-01-07
158
zhuyingda/webster

a reliable high-level web crawling & scraping framework for Node.js.

zhuyingda
โญ 560 JavaScript ๐Ÿ‘ค zhuyingda User ๐Ÿ• Updated 2025-03-12 ๐Ÿ“… Created 2017-11-04
159
MikeMeliz/TorCrawl.py

Crawl and extract (regular or onion) webpages through TOR network

MikeMeliz
โญ 550 Python ๐Ÿ‘ค MikeMeliz User ๐Ÿ• Updated 2026-06-29 ๐Ÿ“… Created 2016-12-05
160
dirtyfilthy/freshonions-torscraper

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

dirtyfilthy
โญ 543 Python ๐Ÿ‘ค dirtyfilthy User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2017-02-15
161
dengmeiluan/ticket-monitoring

โœˆ๏ธ Self-hosted, multi-channel airfare monitoring: collect detailed flight data โ†’ send text-and-image alerts via DingTalk โ†’ trigger strong, four-tier alerts when targets are met. Domestic and international routes (destinations across five continents) use the same collectโ€“evaluateโ€“notify workflow. Data stays on your machine, all settings are managed on the web, and itโ€™s open source and free.

dengmeiluan
โญ 528 Python ๐Ÿ‘ค dengmeiluan User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-09-15
162
AlexMathew/scrapple

A framework for creating semi-automatic web content extractors

AlexMathew
โญ 506 Python ๐Ÿ‘ค AlexMathew User ๐Ÿ• Updated 2026-01-16 ๐Ÿ“… Created 2014-12-13