Git Stars Git Stars

#spider Repositories

Discover 92 active open source GitHub repositories tagged with #spider, accumulating 325,814 total stars (averaging 3,541 stars per project). Leading this topic is NaiboWang/EasySpider with 44,801 stars.

92
Total Repositories
325,814
Total Stars
3,541
Avg Stars/Repo
1
NaiboWang/EasySpider

A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.

NaiboWang
โญ 44,801 JavaScript ๐Ÿ‘ค NaiboWang User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2020-07-18
2
gocolly/colly

Elegant Scraper and Crawler Framework for Golang

gocolly
โญ 25,608 Go ๐Ÿ‘ค gocolly Organization ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2017-09-29
3
jhao104/proxy_pool

Python ProxyPool for web spider

jhao104
โญ 23,798 Python ๐Ÿ‘ค jhao104 User ๐Ÿ• Updated 2026-06-15 ๐Ÿ“… Created 2016-11-25
4
shengqiangzhang/examples-of-web-crawlers

Some very interesting Python crawler examples, friendly to beginners, mainly crawling websites like Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ, etc. (Some interesting examples of python crawlers that are friendly to beginners.)

shengqiangzhang
โญ 14,654 HTML ๐Ÿ‘ค shengqiangzhang User ๐Ÿ• Updated 2025-06-28 ๐Ÿ“… Created 2019-03-13
5
s0md3v/Photon

Incredibly fast crawler designed for OSINT.

s0md3v
โญ 13,377 Python ๐Ÿ‘ค s0md3v User ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2018-03-30
6
crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

crawlab-team
โญ 12,301 Go ๐Ÿ‘ค crawlab-team Organization ๐Ÿ• Updated 2026-02-10 ๐Ÿ“… Created 2019-02-10
7
ihmily/DouyinLiveRecorder

A live recording software with cyclic duty and multiplayer recording, supporting live recording of 40+ platforms such as Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, Shopee, etc.

ihmily
โญ 11,278 Python ๐Ÿ‘ค ihmily User ๐Ÿ• Updated 2026-04-01 ๐Ÿ“… Created 2023-07-17
8
guyueyingmu/avbook

AV Movie Management System, avmoo, javbus, javlibrary Crawler, Online AV Film Library, AV Magnet Link Database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database

guyueyingmu
โญ 9,894 PHP ๐Ÿ‘ค guyueyingmu User ๐Ÿ• Updated 2024-06-01 ๐Ÿ“… Created 2017-02-19
9
kangvcar/InfoSpider

INFO-SPIDER is a crawler toolbox๐Ÿงฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.

kangvcar
โญ 8,275 Python ๐Ÿ‘ค kangvcar User ๐Ÿ• Updated 2026-04-21 ๐Ÿ“… Created 2020-07-11
10
lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

lorien
โญ 8,242 JavaScript ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-08-12
11
andeya/pholcus

Pholcus is a distributed high-concurrency crawler software written in pure golang

andeya
โญ 7,583 Go ๐Ÿ‘ค andeya User ๐Ÿ• Updated 2026-08-24 ๐Ÿ“… Created 2015-07-03
12
BruceDone/awesome-crawler

A collection of awesome web crawler,spider in different languages

BruceDone
โญ 7,051 ๐Ÿ‘ค BruceDone User ๐Ÿ• Updated 2024-06-16 ๐Ÿ“… Created 2016-10-10
13
bda-research/node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

bda-research
โญ 6,797 TypeScript ๐Ÿ‘ค bda-research Organization ๐Ÿ• Updated 2026-06-18 ๐Ÿ“… Created 2010-11-25
14
wechatsync/Wechatsync

One-click synchronization of articles to multiple content platforms, supports Toutiao, WordPress, Zhihu, JianShu, Juejin, CSDN, Typecho and other major platforms, publish once, sync to multiple platforms. Free up personal productivity.

wechatsync
โญ 6,663 TypeScript ๐Ÿ‘ค wechatsync Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2020-09-01
15
DedSecInside/TorBot

Dark Web OSINT Tool

DedSecInside
โญ 5,220 Python ๐Ÿ‘ค DedSecInside Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2017-05-17
16
201206030/novel-plus

novel-plus is a feature-rich, multi-platform (PC, WAP, app, and mini program) novel CMS. It includes features such as novel recommendations, search, rankings, reading, bookshelves, reviews, reading lists, web scraping, a member center, a writersโ€™ section, top-ups and subscriptions, and news publishing.

201206030
โญ 4,822 Java ๐Ÿ‘ค 201206030 User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2020-05-02
17
Nemo2011/bilibili-api

Common Bilibili API calls. Supports functions such as videos, animes, users, channels, audio, etc. Original repository address: https://github.com/MoyuScript/bilibili-api

Nemo2011
โญ 4,197 Python ๐Ÿ‘ค Nemo2011 User ๐Ÿ• Updated 2026-07-03 ๐Ÿ“… Created 2022-04-27
18
Boris-code/feapder

๐Ÿš€๐Ÿš€๐Ÿš€feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.

Boris-code
โญ 3,732 Python ๐Ÿ‘ค Boris-code User ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2021-02-08
19
elliotgao2/toapi

Every web site provides APIs.

elliotgao2
โญ 3,557 Python ๐Ÿ‘ค elliotgao2 User ๐Ÿ• Updated 2026-08-27 ๐Ÿ“… Created 2017-11-27
20
Gerapy/Gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Gerapy
โญ 3,518 Python ๐Ÿ‘ค Gerapy Organization ๐Ÿ• Updated 2026-07-04 ๐Ÿ“… Created 2017-06-30
21
my8100/scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs ๆ–‡ๆกฃ :point_right:

my8100
โญ 3,403 Python ๐Ÿ‘ค my8100 User ๐Ÿ• Updated 2025-02-19 ๐Ÿ“… Created 2018-09-30
22
wnma3mz/wechat_articles_spider

WeChat Public Account Article Crawler

wnma3mz
โญ 3,326 Python ๐Ÿ‘ค wnma3mz User ๐Ÿ• Updated 2024-04-18 ๐Ÿ“… Created 2018-03-04
23
5ime/video_spider

Short video watermark removal: Douyin, Pipixia, Huoshan, Weishi, Weibo, Oasis, Zuiyou, Qingshipin, Kuaishou, Quanmin Xiaoshipin, Barcelona Film, Momo, Before Bifeng, Kaiyan, Vue Vlog Xiaokuxiu, Pipixia Funny, Quanmin K Ge, Xigua Video, Doupai, Huya, 6 Rooms, Li Video, Xinpianchang, acfun, Meipai...

5ime
โญ 3,001 PHP ๐Ÿ‘ค 5ime User ๐Ÿ• Updated 2025-12-03 ๐Ÿ“… Created 2020-07-22
24
howie6879/owllook

owllook-Novel Search Engine

howie6879
โญ 2,900 Python ๐Ÿ‘ค howie6879 User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2017-02-18
25
CharlesPikachu/DecryptLogin

DecryptLogin: APIs for loginning some websites by using requests.

CharlesPikachu
โญ 2,871 Python ๐Ÿ‘ค CharlesPikachu User ๐Ÿ• Updated 2024-08-09 ๐Ÿ“… Created 2019-02-25
26
spider-rs/spider

Foundational low-latency web data collection in Rust

spider-rs
โญ 2,856 Rust ๐Ÿ‘ค spider-rs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-07
27
jaeles-project/gospider

Gospider - Fast web spider written in Go

jaeles-project
โญ 2,851 Go ๐Ÿ‘ค jaeles-project Organization ๐Ÿ• Updated 2024-04-21 ๐Ÿ“… Created 2020-01-22
28
geziyor/geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

geziyor
โญ 2,781 Go ๐Ÿ‘ค geziyor Organization ๐Ÿ• Updated 2026-07-02 ๐Ÿ“… Created 2019-06-06
29
pibigstar/go-demo

Go Language Example Tutorial from Beginner to Advanced, Including Basic Library Usage, Design Patterns, Common Interview Pitfalls, Utility Classes, Integrating Third-Party Services, etc.

pibigstar
โญ 2,708 Go ๐Ÿ‘ค pibigstar User ๐Ÿ• Updated 2025-06-07 ๐Ÿ“… Created 2018-08-29
30
jae-jae/QueryList

:spider: The progressive PHP crawler framework! ไผ˜้›…็š„ๆธ่ฟ›ๅผPHP้‡‡้›†ๆก†ๆžถใ€‚

jae-jae
โญ 2,689 PHP ๐Ÿ‘ค jae-jae User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2015-12-20
31
lorien/grab

Web Scraping Framework

lorien
โญ 2,461 Python ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2025-09-19 ๐Ÿ“… Created 2013-05-01
32
JayBizzle/Crawler-Detect

๐Ÿ•ท CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

JayBizzle
โญ 2,410 PHP ๐Ÿ‘ค JayBizzle User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2015-03-23
33
QianyanTech/Image-Downloader

Download images from Google, Bing, Baidu. Google, Baidu, Bing image download.

QianyanTech
โญ 2,331 Python ๐Ÿ‘ค QianyanTech Organization ๐Ÿ• Updated 2024-07-07 ๐Ÿ“… Created 2016-07-17
34
sjdirect/abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

sjdirect
โญ 2,308 C# ๐Ÿ‘ค sjdirect User ๐Ÿ• Updated 2026-07-17 ๐Ÿ“… Created 2013-08-08
35
zorlan/skycaiji

Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.

zorlan
โญ 2,098 PHP ๐Ÿ‘ค zorlan User ๐Ÿ• Updated 2026-10-07 ๐Ÿ“… Created 2018-02-26
36
elliotgao2/gain

Web crawling framework based on asyncio.

elliotgao2
โญ 2,017 Python ๐Ÿ‘ค elliotgao2 User ๐Ÿ• Updated 2026-08-09 ๐Ÿ“… Created 2017-05-31
37
ShunCai/QZoneExport

QQ Space Export Assistant, used to back up QQ Space posts, journals, private diaries, albums, videos, message boards, QQ friends, favorites, shares, and recent visitors as files for easy migration and preservation.

ShunCai
โญ 1,927 TypeScript ๐Ÿ‘ค ShunCai User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2019-03-10
38
coder-hxl/x-crawl

Flexible Node.js crawler library

coder-hxl
โญ 1,888 TypeScript ๐Ÿ‘ค coder-hxl User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-01-22
39
1N3/BlackWidow

A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.

1N3
โญ 1,843 Python ๐Ÿ‘ค 1N3 User ๐Ÿ• Updated 2026-04-17 ๐Ÿ“… Created 2018-01-06
40
eatmoreduck/boss-zhipin-scraper

BOSS Zhipin scraper / BOSS Zhipin job data crawler, based on Chrome CDP protocol to reuse real login sessions, bypass font anti-scraping, and output plaintext salary JSON/CSV + salary-skill analysis.

eatmoreduck
โญ 1,781 Python ๐Ÿ‘ค eatmoreduck User ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-06-11
41
u3c3/BT-btt

Introduction to the magnetic website U3C3 and domain update

u3c3
โญ 1,779 ๐Ÿ‘ค u3c3 User ๐Ÿ• Updated 2025-01-01 ๐Ÿ“… Created 2020-04-28
42
howie6879/ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

howie6879
โญ 1,738 Python ๐Ÿ‘ค howie6879 User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2018-07-10
43
erma0/douyin

Douyin Crawler - Collect public data such as account homepage, likes, collections, music original sound, topics, search, collections, works, following, followers, etc.

erma0
โญ 1,625 ๐Ÿ‘ค erma0 User ๐Ÿ• Updated 2026-08-14 ๐Ÿ“… Created 2022-05-29
44
ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

ArchiveTeam
โญ 1,572 Python ๐Ÿ‘ค ArchiveTeam Organization ๐Ÿ• Updated 2025-05-23 ๐Ÿ“… Created 2015-02-05
45
srx-2000/spider_collection

Python crawlers, current inventory: NetEase Cloud Music song crawling, Bilibili video crawling, Zhihu Q&A crawling, wallpaper crawling, xvideos video crawling, audiobook crawling, Weibo crawler, Anjuke information crawling + data visualization, Bilibili video cover extractor, IP proxy pool encapsulation, Zhihu million-level user crawler + data analysis, GitHub user crawler

srx-2000
โญ 1,554 Python ๐Ÿ‘ค srx-2000 User ๐Ÿ• Updated 2024-04-23 ๐Ÿ“… Created 2020-02-04
46
darbra/sperm

A summary of excellent reverse engineering articles, worth a look

darbra
โญ 1,427 ๐Ÿ‘ค darbra User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2022-03-02
47
okfn-brasil/querido-diario

๐Ÿ“ฐ Brazilian official gazettes accessible to all | ๐Ÿ“ฐ Brazilian government gazettes, accessible to everyone.

okfn-brasil
โญ 1,424 Python ๐Ÿ‘ค okfn-brasil Organization ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2018-04-01
48
kgspider/crawler

Brother K Web Scraping Code Sharing, JS Reverse Engineering, Advanced Web Scraping. Follow the WeChat official account: Brother K Web Scraping

kgspider
โญ 1,371 JavaScript ๐Ÿ‘ค kgspider User ๐Ÿ• Updated 2025-05-12 ๐Ÿ“… Created 2021-08-24
49
0xInfection/XSRFProbe

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

0xInfection
โญ 1,310 Python ๐Ÿ‘ค 0xInfection User ๐Ÿ• Updated 2026-07-12 ๐Ÿ“… Created 2018-08-21
50
bookstairs/bookhunter

A download tools for clawing the ebooks from internets.

bookstairs
โญ 1,305 Go ๐Ÿ‘ค bookstairs Organization ๐Ÿ• Updated 2026-02-28 ๐Ÿ“… Created 2022-04-08

Frequently Asked Questions

1

What is the most popular #spider repository on GitHub?

The top ranked open source #spider project on GitHub is NaiboWang/EasySpider with 44,801 stars. A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.

2

Which programming languages are most used in #spider repositories?

Among the top ranked #spider projects on GitHub, the most common programming languages are Python, Go, PHP, TypeScript.

3

How is the #spider repository ranking calculated?

We track 92 active open source repositories tagged with #spider (500+ stars), totaling 325,814 stars with an average of 3,541 stars per repository. Rankings update daily from the GitHub API.