Git Stars Git Stars

#spider Repositories

92
Total Repositories
325,814
Total Stars
3,541
Avg Stars/Repo
51
kiddyuchina/Beanbun

Beanbun is a multi-process web crawling framework written in PHP, with good openness and high scalability, based on Workerman.

kiddyuchina
โญ 1,258 PHP ๐Ÿ‘ค kiddyuchina User ๐Ÿ• Updated 2026-05-26 ๐Ÿ“… Created 2017-04-13
52
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
53
BaiduSpider/BaiduSpider

BaiduSpider, a crawler that crawls Baidu search results, currently supports Baidu web search, Baidu image search, Baidu Knows search, Baidu video search, Baidu news search, Baidu Wenku search, Baidu experience search, and Baidu Baike search.

BaiduSpider
โญ 1,130 Python ๐Ÿ‘ค BaiduSpider Organization ๐Ÿ• Updated 2024-06-14 ๐Ÿ“… Created 2020-10-03
54
elixir-crawly/crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

elixir-crawly
โญ 1,129 Elixir ๐Ÿ‘ค elixir-crawly Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-09
55
skygazer42/DL-Hub

LLMs Large Models Notes 50 Articles This repository contains practical projects in fields such as machine learning, deep learning, computer vision, natural language processing, large models, web scraping, etc.

skygazer42
โญ 1,121 Python ๐Ÿ‘ค skygazer42 User ๐Ÿ• Updated 2026-08-30 ๐Ÿ“… Created 2022-04-05
56
wujunwei928/parse-video

Golang Short Video Watermark Removal: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, Quanmin Xiaoshipin, Pipixia Gaoqiao, Xigua Video, Huya, Li Video, AcFun, Haokan Video...

wujunwei928
โญ 1,104 Go ๐Ÿ‘ค wujunwei928 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2022-07-12
57
zhangyd-c/OneBlog

:alien: OneBlog, a simple, beautiful, powerful, and responsive Java blog

zhangyd-c
โญ 1,103 Java ๐Ÿ‘ค zhangyd-c User ๐Ÿ• Updated 2025-03-16 ๐Ÿ“… Created 2018-04-10
58
1061700625/WeChat_Article

Crawl WeChat Official Account Articles

1061700625
โญ 1,089 Python ๐Ÿ‘ค 1061700625 User ๐Ÿ• Updated 2026-06-17 ๐Ÿ“… Created 2019-02-21
59
gsh199449/spider

A configurable web spider with a easy-to-use web console

gsh199449
โญ 996 Java ๐Ÿ‘ค gsh199449 User ๐Ÿ• Updated 2026-06-03 ๐Ÿ“… Created 2016-11-24
60
spidersuite/SpiderSuite

SpiderSuite (web security crawler) releases, wiki and roadmap

spidersuite
โญ 990 ๐Ÿ‘ค spidersuite Organization ๐Ÿ• Updated 2026-08-13 ๐Ÿ“… Created 2023-04-05
61
QIN2DIM/V2RSS

:rocket: Collect|Free|High-quality|Subscription?

QIN2DIM
โญ 978 Python ๐Ÿ‘ค QIN2DIM User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2020-06-02
62
LoseNine/Crack-JS-Spider

JS reverse engineering, cracking JS anti-scraping encryption parameters, already cracked Geetest slider w (2022.2.19), QQ Music sign (2022.2.13), Pinduoduo anti_content, Boss Zhipin zp_token, Zhihu x-zse-96, Kugou kg_mid/dfid, Vipshop mars_cid, China Judgments Online (updated 2020-06-30), Taobao password, Tianan Insurance login, Bilibili login, Fang.com login, WPS login, Weibo login, Youdao Translate, Netease login, WeChat Official Account login, Air China Network login, Jmubiao login, Student Information Management System login, Gengying Finance login, Chongqing Science and Technology Resource Sharing Platform login, Netease Cloud Music download, one-click video link parsing, Cailianshe login.

LoseNine
โญ 974 JavaScript ๐Ÿ‘ค LoseNine User ๐Ÿ• Updated 2024-05-03 ๐Ÿ“… Created 2020-02-09
63
fredwu/crawler

A high performance web crawler / scraper in Elixir.

fredwu
โญ 963 Elixir ๐Ÿ‘ค fredwu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-08
64
ma6254/FictionDown

Novel Download|Novel Scraping|่ตท็‚น|็ฌ”่ถฃ้˜|Export Markdown|Export txt|Convert epub|Ad Filtering|Auto Proofreading

ma6254
โญ 955 Go ๐Ÿ‘ค ma6254 User ๐Ÿ• Updated 2025-01-21 ๐Ÿ“… Created 2019-03-03
65
0xAllenChen/spider_reverse

Crawler reverse engineering cases, completed: TLS fingerprint | Ruishu | Zhenkunxing | NetEase Yudun | WeChat Mini Program Decompilation Reverse Engineering (Baida Xingxi) | Tonghuashun | rpc decryption | Jiasule | Jueyan slider CAPTCHA | Julang Sanshu | Boss Zhipin | Qichacha | China Minmetals | QQ Music | Industrial Policy Big Data Platform | Qizhidiao | Xueqiu.com (acw_sc__v2) | 1688 | Qimai Data | whggzy | Qiming Keji | mohurd | Ejoin Data | Ouke Yunlian

0xAllenChen
โญ 946 Python ๐Ÿ‘ค 0xAllenChen User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-06-02
66
hellock/icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

hellock
โญ 936 Python ๐Ÿ‘ค hellock User ๐Ÿ• Updated 2025-12-10 ๐Ÿ“… Created 2016-03-19
67
alltheplaces/alltheplaces

A set of spiders and scrapers to extract location information from places that post their location on the internet.

alltheplaces
โญ 851 Python ๐Ÿ‘ค alltheplaces Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-06-15
68
BlessedRebuS/Krawl

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

BlessedRebuS
โญ 844 Python ๐Ÿ‘ค BlessedRebuS User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-12-10
69
kingschan1204/istock

:point_right: A Java stock crawler based on Spring Boot (supports only A-shares), if you :heart: please :star: . V2 upgrade version is under development!

kingschan1204
โญ 836 Java ๐Ÿ‘ค kingschan1204 User ๐Ÿ• Updated 2024-12-27 ๐Ÿ“… Created 2017-03-09
70
postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

postmodern
โญ 835 Ruby ๐Ÿ‘ค postmodern User ๐Ÿ• Updated 2026-01-12 ๐Ÿ“… Created 2009-03-08
71
DoiiarX/NLCISBNPlugin

A calibre source/metadata plugin based on the National Library of China's ISBN search. https://doiiars.com/article/NLCISBNPlugin

DoiiarX
โญ 812 Python ๐Ÿ‘ค DoiiarX User ๐Ÿ• Updated 2026-01-07 ๐Ÿ“… Created 2023-12-14
72
bahaabdelwahed/killshot

A Penetration Testing Framework, Information gathering tool & Website Vulnerability Scanner

bahaabdelwahed
โญ 791 Ruby ๐Ÿ‘ค bahaabdelwahed User ๐Ÿ• Updated 2025-12-07 ๐Ÿ“… Created 2018-10-27
73
montaro2017/bili_novel_packer

Light Novel Packer, which scrapes content from Bili Light Novel websites (bilinovel.com linovelib.com) and packages it into EPUB format. It supports covers, illustrations, tables of contents, and merging separate volumes.

montaro2017
โญ 787 Dart ๐Ÿ‘ค montaro2017 User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-02-20
74
xuxueli/xxl-crawler

A lightweight web crawler framework. (Java crawler framework)

xuxueli
โญ 756 Java ๐Ÿ‘ค xuxueli User ๐Ÿ• Updated 2026-08-08 ๐Ÿ“… Created 2017-10-17
75
polyrabbit/hacker-news-digest

๐Ÿ“ฐ Let LLM Summarize Hacker News for You

polyrabbit
โญ 754 Python ๐Ÿ‘ค polyrabbit User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-09-17
76
signal18/replication-manager

Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server

signal18
โญ 747 Go ๐Ÿ‘ค signal18 Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-10-14
77
josephlimtech/linkedin-profile-scraper-api

๐Ÿ•ต๏ธโ€โ™‚๏ธ LinkedIn profile scraper returning structured profile data in JSON.

josephlimtech
โญ 725 TypeScript ๐Ÿ‘ค josephlimtech User ๐Ÿ• Updated 2024-04-05 ๐Ÿ“… Created 2018-12-21
78
wujunwei928/parse-video-py

Python Short Video Watermark Removal Scraper: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, National Short Video, Pipi Funny, Xigua Video, Huya, Li Video, AcFun, Haokan Video...

wujunwei928
โญ 721 Python ๐Ÿ‘ค wujunwei928 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2024-04-03
79
xinlingqudongX/TSDK

Taobao Crawler SDK, used for Taobao Open Platform or logging in and crawling Taobao, Tmall, Alibaba

xinlingqudongX
โญ 716 Python ๐Ÿ‘ค xinlingqudongX User ๐Ÿ• Updated 2026-03-26 ๐Ÿ“… Created 2018-12-06
80
casual-silva/NewsCrawl

Enterprise-grade open source public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers; crawler monitoring visualization; configuration of cluster crawler allocation strategy; ๐Ÿ‘‰ The ready-made Docker one-click deployment documentation has already solved the pitfalls for you.

casual-silva
โญ 688 Python ๐Ÿ‘ค casual-silva User ๐Ÿ• Updated 2026-05-23 ๐Ÿ“… Created 2022-07-08
81
darbra/sign

Approaches and experience for reverse engineering app parameters, Android reverse engineering

darbra
โญ 667 ๐Ÿ‘ค darbra User ๐Ÿ• Updated 2024-06-13 ๐Ÿ“… Created 2020-05-26
82
spiritLHLS/Hang-up-items

Questionnaire project, cloud server recommendation, hanging project, free proxy, various script collection. Welcome to the upper right corner of the point bell to receive timely updates. (Do not fork, low profile)

spiritLHLS
โญ 646 ๐Ÿ‘ค spiritLHLS User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2022-04-11
83
fengzhizi715/NetDiscovery

NetDiscovery is a general-purpose crawler framework/middleware implemented based on frameworks such as Vert.x and RxJava 2.

fengzhizi715
โญ 643 Java ๐Ÿ‘ค fengzhizi715 User ๐Ÿ• Updated 2026-06-05 ๐Ÿ“… Created 2017-11-26
84
cv-cat/catbus

One command to operate Xiaohongshu, Douyin, TikTok, Bilibili, Kuaishou, Weibo, Xianyu, Taobao, JD.com, and X: unified commands and JSON output for people and AI agents

cv-cat
โญ 641 TypeScript ๐Ÿ‘ค cv-cat User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-27
85
scrapfly/Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

scrapfly
โญ 636 JavaScript ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2025-09-03
86
l4rm4nd/LinkedInDumper

Python 3 script to dump/scrape/extract company employees from LinkedIn API

l4rm4nd
โญ 624 Python ๐Ÿ‘ค l4rm4nd User ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2022-10-17
87
mouday/spider-admin-pro

spider-admin-pro A visual management tool for viewing Scrapy+Scrapyd crawler projects and scheduled timing of crawler tasks, the upgraded version of SpiderAdmin

mouday
โญ 614 Python ๐Ÿ‘ค mouday User ๐Ÿ• Updated 2026-06-15 ๐Ÿ“… Created 2021-01-25
88
jadehh/TVSpider

Movie and Cat Movie Crawler Repository

jadehh
โญ 604 JavaScript ๐Ÿ‘ค jadehh User ๐Ÿ• Updated 2024-09-18 ๐Ÿ“… Created 2023-02-16
89
CharlesPikachu/freeproxy

FreeProxy: Collecting free proxies from the internet. (Massive high-quality free proxies worldwide, supporting crawling from dozens of free proxy sharing sources, custom rule-based proxy filtering, essential for web scraping and data analysis, updated daily with a vast number of free proxies.)

CharlesPikachu
โญ 594 Python ๐Ÿ‘ค CharlesPikachu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-10-29
90
zhuyingda/webster

a reliable high-level web crawling & scraping framework for Node.js.

zhuyingda
โญ 560 JavaScript ๐Ÿ‘ค zhuyingda User ๐Ÿ• Updated 2025-03-12 ๐Ÿ“… Created 2017-11-04
91
dirtyfilthy/freshonions-torscraper

Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

dirtyfilthy
โญ 543 Python ๐Ÿ‘ค dirtyfilthy User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2017-02-15
92
sansan0/bilibili-comment-analyzer

๐ŸŽฏ Bilibili comment data visualization analysis software -- content creators can use it to guide their topic selection and understand their fan base.

sansan0
โญ 503 Python ๐Ÿ‘ค sansan0 User ๐Ÿ• Updated 2025-06-27 ๐Ÿ“… Created 2025-06-03