Git Stars Git Stars

#scrapy Repositories

Discover 38 active open source GitHub repositories tagged with #scrapy, accumulating 84,191 total stars (averaging 2,215 stars per project). Leading this topic is crawlab-team/crawlab with 12,301 stars.

38
Total Repositories
84,191
Total Stars
2,215
Avg Stars/Repo
1
crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

crawlab-team
โญ 12,301 Go ๐Ÿ‘ค crawlab-team Organization ๐Ÿ• Updated 2026-02-10 ๐Ÿ“… Created 2019-02-10
2
chyroc/WechatSogou

WeChat Official Account Crawler API based on Sogou WeChat Search

chyroc
โญ 6,411 Python ๐Ÿ‘ค chyroc User ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2016-05-09
3
rmax/scrapy-redis

Redis-based components for Scrapy.

rmax
โญ 5,648 Python ๐Ÿ‘ค rmax User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2011-08-29
4
DropsDevopsOrg/ECommerceCrawlers

Practical ๐Ÿ multiple websites, e-commerce data crawlers ๐Ÿ•ท. Includes ๐Ÿ•ธ: Taobao products, WeChat public accounts, Dianping, Qichacha, job websites, Xianyu, Ali tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu.com, Quan jing wang, Douban music, some province's drug supervision bureau, Sohu news, machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword inclusion count, spider general directory, Toutiao, Douban movie reviews, Ctrip, Xiaomi app store, Anjuke, Tujia homestay โค๏ธโค๏ธโค๏ธ. WeChat crawler demo project:

DropsDevopsOrg
โญ 5,355 Python ๐Ÿ‘ค DropsDevopsOrg Organization ๐Ÿ• Updated 2024-05-22 ๐Ÿ“… Created 2019-03-29
5
nghuyong/WeiboSpider

Continuously maintained Sina Weibo collection tool๐Ÿš€๐Ÿš€๐Ÿš€

nghuyong
โญ 4,125 Python ๐Ÿ‘ค nghuyong User ๐Ÿ• Updated 2026-06-30 ๐Ÿ“… Created 2017-10-29
6
Boris-code/feapder

๐Ÿš€๐Ÿš€๐Ÿš€feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.

Boris-code
โญ 3,732 Python ๐Ÿ‘ค Boris-code User ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2021-02-08
7
Gerapy/Gerapy

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Gerapy
โญ 3,518 Python ๐Ÿ‘ค Gerapy Organization ๐Ÿ• Updated 2026-07-04 ๐Ÿ“… Created 2017-06-30
8
my8100/scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs ๆ–‡ๆกฃ :point_right:

my8100
โญ 3,403 Python ๐Ÿ‘ค my8100 User ๐Ÿ• Updated 2025-02-19 ๐Ÿ“… Created 2018-09-30
9
scrapy-plugins/scrapy-splash

Scrapy+Splash for JavaScript integration

scrapy-plugins
โญ 3,230 Python ๐Ÿ‘ค scrapy-plugins Organization ๐Ÿ• Updated 2025-02-11 ๐Ÿ“… Created 2013-05-13
10
scrapy/scrapyd

A service daemon to run Scrapy spiders

scrapy
โญ 3,100 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2013-02-01
11
RayXBuilds/Movie_Recommend

Historical learning project: Spark movie recommendations, web scraping, web app, and backend. Ongoing development of the new version: cinematch-recommender.

RayXBuilds
โญ 3,007 Java ๐Ÿ‘ค RayXBuilds User ๐Ÿ• Updated 2026-10-21 ๐Ÿ“… Created 2018-04-18
12
dataabc/weibo-search

Get Weibo search result information, the search can be either Weibo keyword search or Weibo topic search

dataabc
โญ 2,340 Python ๐Ÿ‘ค dataabc User ๐Ÿ• Updated 2026-06-05 ๐Ÿ“… Created 2020-03-25
13
QianyanTech/Image-Downloader

Download images from Google, Bing, Baidu. Google, Baidu, Bing image download.

QianyanTech
โญ 2,331 Python ๐Ÿ‘ค QianyanTech Organization ๐Ÿ• Updated 2024-07-07 ๐Ÿ“… Created 2016-07-17
14
ictar/python-doc

translate python documents to Chinese for convenient reference In short, this is where those dear Python documents are kept, and we will try our best to translate them into Chinese ~~

ictar
โญ 1,942 ๐Ÿ‘ค ictar User ๐Ÿ• Updated 2024-05-17 ๐Ÿ“… Created 2015-12-18
15
DanMcInerney/xsscrapy

XSS spider - 66/66 wavsep XSS detected

DanMcInerney
โญ 1,731 Python ๐Ÿ‘ค DanMcInerney User ๐Ÿ• Updated 2024-06-13 ๐Ÿ“… Created 2014-06-23
16
TheWebScrapingClub/webscraping-from-0-to-hero

The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

TheWebScrapingClub
โญ 1,695 ๐Ÿ‘ค TheWebScrapingClub Organization ๐Ÿ• Updated 2024-05-27 ๐Ÿ“… Created 2022-05-26
17
eliasdabbas/advertools

advertools - online marketing productivity and analysis tools

eliasdabbas
โญ 1,508 Python ๐Ÿ‘ค eliasdabbas User ๐Ÿ• Updated 2026-06-30 ๐Ÿ“… Created 2017-05-14
18
scrapy-plugins/scrapy-playwright

Playwright integration for Scrapy

scrapy-plugins
โญ 1,469 Python ๐Ÿ‘ค scrapy-plugins Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2020-11-10
19
tb0hdan/domains

Worldโ€™s single largest Internet domains dataset

tb0hdan
โญ 1,181 Shell ๐Ÿ‘ค tb0hdan User ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2020-01-12
20
xingag/spider_python

Python crawler

xingag
โญ 1,165 Python ๐Ÿ‘ค xingag User ๐Ÿ• Updated 2026-08-07 ๐Ÿ“… Created 2018-09-12
21
juancarlospaco/faster-than-requests

Faster requests on Python 3

juancarlospaco
โญ 1,131 Nim ๐Ÿ‘ค juancarlospaco User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-11-13
22
vifreefly/kimuraframework

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.

vifreefly
โญ 1,102 Ruby ๐Ÿ‘ค vifreefly User ๐Ÿ• Updated 2026-01-27 ๐Ÿ“… Created 2018-08-23
23
moyada/stealer

Douyin, Kuaishou, Huoshan, Pipixia, video watermark remover program

moyada
โญ 1,094 Python ๐Ÿ‘ค moyada User ๐Ÿ• Updated 2024-05-21 ๐Ÿ“… Created 2020-04-07
24
clemfromspace/scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

clemfromspace
โญ 950 Python ๐Ÿ‘ค clemfromspace User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2018-02-11
25
TikHub/TikHub-API-Python-SDK

High-performance asynchronous Captcha Solver(verification code resolver) for Douyin(ๆŠ–้Ÿณ), TikTok, Xiaohongshu(ๅฐ็บขไนฆ), Kuaishou(ๅฟซๆ‰‹), Weibo(ๅพฎๅš), Instagram, YouTube(ๆฒน็ฎก), Twitter(X) with Temp Mail(temporary email) API(interface).

TikHub
โญ 943 Python ๐Ÿ‘ค TikHub Organization ๐Ÿ• Updated 2026-10-03 ๐Ÿ“… Created 2022-12-17
26
hellock/icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

hellock
โญ 936 Python ๐Ÿ‘ค hellock User ๐Ÿ• Updated 2025-12-10 ๐Ÿ“… Created 2016-03-19
27
superboyyy/xianyu_spider

Xianyu scraper, developed based on the Xianyu web client. It provides a product search API built with FastAPI, supporting asynchronous concurrent requests and automated deduplication and storage of data. Login is now supported.

superboyyy
โญ 922 Python ๐Ÿ‘ค superboyyy User ๐Ÿ• Updated 2026-09-18 ๐Ÿ“… Created 2025-06-04
28
scrapinghub/scrapyrt

HTTP API for Scrapy spiders

scrapinghub
โญ 884 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2026-10-13 ๐Ÿ“… Created 2015-01-06
29
alltheplaces/alltheplaces

A set of spiders and scrapers to extract location information from places that post their location on the internet.

alltheplaces
โญ 851 Python ๐Ÿ‘ค alltheplaces Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-06-15
30
MorvanZhou/easy-scraping-tutorial

Simple but useful Python web scraping tutorial code.

MorvanZhou
โญ 814 Jupyter Notebook ๐Ÿ‘ค MorvanZhou User ๐Ÿ• Updated 2024-04-07 ๐Ÿ“… Created 2017-12-29
31
unnohwn/telegram-scraper

A powerful Python script that allows you to scrape messages and media from Telegram channels using the Telethon library. Features include real-time continuous scraping, media downloading, and data export capabilities.

unnohwn
โญ 797 Python ๐Ÿ‘ค unnohwn User ๐Ÿ• Updated 2026-04-11 ๐Ÿ“… Created 2024-11-04
32
scrapy/scrapyd-client

Command line client for Scrapyd server

scrapy
โญ 773 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-02-27 ๐Ÿ“… Created 2015-03-27
33
TeamHG-Memex/scrapy-rotating-proxies

use multiple proxies with Scrapy

TeamHG-Memex
โญ 772 Python ๐Ÿ‘ค TeamHG-Memex Organization ๐Ÿ• Updated 2026-04-08 ๐Ÿ“… Created 2017-02-01
34
casual-silva/NewsCrawl

Enterprise-grade open source public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers; crawler monitoring visualization; configuration of cluster crawler allocation strategy; ๐Ÿ‘‰ The ready-made Docker one-click deployment documentation has already solved the pitfalls for you.

casual-silva
โญ 688 Python ๐Ÿ‘ค casual-silva User ๐Ÿ• Updated 2026-05-23 ๐Ÿ“… Created 2022-07-08
35
lzjqsdd/NewsSpider

Crawl news from Toutiao, NetEase, Tencent, etc., and build a simple search engine

lzjqsdd
โญ 638 Python ๐Ÿ‘ค lzjqsdd User ๐Ÿ• Updated 2024-05-14 ๐Ÿ“… Created 2016-04-19
36
mouday/spider-admin-pro

spider-admin-pro A visual management tool for viewing Scrapy+Scrapyd crawler projects and scheduled timing of crawler tasks, the upgraded version of SpiderAdmin

mouday
โญ 614 Python ๐Ÿ‘ค mouday User ๐Ÿ• Updated 2026-06-15 ๐Ÿ“… Created 2021-01-25
37
spekulatius/PHPScraper

A universal web-util for PHP.

spekulatius
โญ 588 PHP ๐Ÿ‘ค spekulatius User ๐Ÿ• Updated 2026-09-08 ๐Ÿ“… Created 2020-04-21
38
AlexMathew/scrapple

A framework for creating semi-automatic web content extractors

AlexMathew
โญ 506 Python ๐Ÿ‘ค AlexMathew User ๐Ÿ• Updated 2026-01-16 ๐Ÿ“… Created 2014-12-13

Frequently Asked Questions

1

What is the most popular #scrapy repository on GitHub?

The top ranked open source #scrapy project on GitHub is crawlab-team/crawlab with 12,301 stars. Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

2

Which programming languages are most used in #scrapy repositories?

Among the top ranked #scrapy projects on GitHub, the most common programming languages are Python, Go, Java, Shell.

3

How is the #scrapy repository ranking calculated?

We track 38 active open source repositories tagged with #scrapy (500+ stars), totaling 84,191 stars with an average of 2,215 stars per repository. Rankings update daily from the GitHub API.