Git Stars Git Stars

#crawl Repositories

Discover 11 active open source GitHub repositories tagged with #crawl, accumulating 27,381 total stars (averaging 2,489 stars per project). Leading this topic is kangvcar/InfoSpider with 8,275 stars.

11
Total Repositories
27,381
Total Stars
2,489
Avg Stars/Repo
1
kangvcar/InfoSpider

INFO-SPIDER is a crawler toolbox๐Ÿงฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.

kangvcar
โญ 8,275 Python ๐Ÿ‘ค kangvcar User ๐Ÿ• Updated 2026-04-21 ๐Ÿ“… Created 2020-07-11
2
201206030/novel-plus

novel-plus is a feature-rich, multi-platform (PC, WAP, app, and mini program) novel CMS. It includes features such as novel recommendations, search, rankings, reading, bookshelves, reviews, reading lists, web scraping, a member center, a writersโ€™ section, top-ups and subscriptions, and news publishing.

201206030
โญ 4,822 Java ๐Ÿ‘ค 201206030 User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2020-05-02
3
any4ai/AnyCrawl

AnyCrawl ๐Ÿš€: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

any4ai
โญ 3,580 TypeScript ๐Ÿ‘ค any4ai User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2025-03-31
4
ReaJason/xhs

Request wrapper based on the Xiaohongshu Web client. https://reajason.github.io/xhs/

ReaJason
โญ 2,177 Python ๐Ÿ‘ค ReaJason User ๐Ÿ• Updated 2025-07-01 ๐Ÿ“… Created 2023-04-03
5
coder-hxl/x-crawl

Flexible Node.js crawler library

coder-hxl
โญ 1,888 TypeScript ๐Ÿ‘ค coder-hxl User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-01-22
6
ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

ArchiveTeam
โญ 1,572 Python ๐Ÿ‘ค ArchiveTeam Organization ๐Ÿ• Updated 2025-05-23 ๐Ÿ“… Created 2015-02-05
7
tavily-ai/tavily-python

The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.

tavily-ai
โญ 1,432 Python ๐Ÿ‘ค tavily-ai Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2023-09-01
8
darbra/sperm

A summary of excellent reverse engineering articles, worth a look

darbra
โญ 1,427 ๐Ÿ‘ค darbra User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2022-03-02
9
LoseNine/Crack-JS-Spider

JS reverse engineering, cracking JS anti-scraping encryption parameters, already cracked Geetest slider w (2022.2.19), QQ Music sign (2022.2.13), Pinduoduo anti_content, Boss Zhipin zp_token, Zhihu x-zse-96, Kugou kg_mid/dfid, Vipshop mars_cid, China Judgments Online (updated 2020-06-30), Taobao password, Tianan Insurance login, Bilibili login, Fang.com login, WPS login, Weibo login, Youdao Translate, Netease login, WeChat Official Account login, Air China Network login, Jmubiao login, Student Information Management System login, Gengying Finance login, Chongqing Science and Technology Resource Sharing Platform login, Netease Cloud Music download, one-click video link parsing, Cailianshe login.

LoseNine
โญ 974 JavaScript ๐Ÿ‘ค LoseNine User ๐Ÿ• Updated 2024-05-03 ๐Ÿ“… Created 2020-02-09
10
scrapfly/Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

scrapfly
โญ 636 JavaScript ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2025-09-03
11
philschmid/clipper.js

HTML to Markdown converter and crawler.

philschmid
โญ 606 TypeScript ๐Ÿ‘ค philschmid User ๐Ÿ• Updated 2024-01-09 ๐Ÿ“… Created 2023-09-27

Frequently Asked Questions

1

What is the most popular #crawl repository on GitHub?

The top ranked open source #crawl project on GitHub is kangvcar/InfoSpider with 8,275 stars. INFO-SPIDER is a crawler toolbox๐Ÿงฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.

2

Which programming languages are most used in #crawl repositories?

Among the top ranked #crawl projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Java.

3

How is the #crawl repository ranking calculated?

We track 11 active open source repositories tagged with #crawl (500+ stars), totaling 27,381 stars with an average of 2,489 stars per repository. Rankings update daily from the GitHub API.