#scrapy Repositories
Discover 38 active open source GitHub repositories tagged with #scrapy, accumulating 84,191 total stars (averaging 2,215 stars per project). Leading this topic is crawlab-team/crawlab with 12,301 stars.
38 repositories total
Distributed web crawler admin platform for spiders management regardless of languages and frameworks.
Practical ๐ multiple websites, e-commerce data crawlers ๐ท. Includes ๐ธ: Taobao products, WeChat public accounts, Dianping, Qichacha, job websites, Xianyu, Ali tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu.com, Quan jing wang, Douban music, some province's drug supervision bureau, Sohu news, machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword inclusion count, spider general directory, Toutiao, Douban movie reviews, Ctrip, Xiaomi app store, Anjuke, Tujia homestay โค๏ธโค๏ธโค๏ธ. WeChat crawler demo project:
๐๐๐feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs ๆๆกฃ :point_right:
Historical learning project: Spark movie recommendations, web scraping, web app, and backend. Ongoing development of the new version: cinematch-recommender.
Get Weibo search result information, the search can be either Weibo keyword search or Weibo topic search
Download images from Google, Bing, Baidu. Google, Baidu, Bing image download.
translate python documents to Chinese for convenient reference In short, this is where those dear Python documents are kept, and we will try our best to translate them into Chinese ~~
The web scraping open project repository aims to share knowledge and experiences about web scraping with Python
Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.
High-performance asynchronous Captcha Solver(verification code resolver) for Douyin(ๆ้ณ), TikTok, Xiaohongshu(ๅฐ็บขไนฆ), Kuaishou(ๅฟซๆ), Weibo(ๅพฎๅ), Instagram, YouTube(ๆฒน็ฎก), Twitter(X) with Temp Mail(temporary email) API(interface).
Xianyu scraper, developed based on the Xianyu web client. It provides a product search API built with FastAPI, supporting asynchronous concurrent requests and automated deduplication and storage of data. Login is now supported.
A set of spiders and scrapers to extract location information from places that post their location on the internet.
A powerful Python script that allows you to scrape messages and media from Telegram channels using the Telethon library. Features include real-time continuous scraping, media downloading, and data export capabilities.
Enterprise-grade open source public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers; crawler monitoring visualization; configuration of cluster crawler allocation strategy; ๐ The ready-made Docker one-click deployment documentation has already solved the pitfalls for you.
Crawl news from Toutiao, NetEase, Tencent, etc., and build a simple search engine
spider-admin-pro A visual management tool for viewing Scrapy+Scrapyd crawler projects and scheduled timing of crawler tasks, the upgraded version of SpiderAdmin
Frequently Asked Questions
What is the most popular #scrapy repository on GitHub?
The top ranked open source #scrapy project on GitHub is crawlab-team/crawlab with 12,301 stars. Distributed web crawler admin platform for spiders management regardless of languages and frameworks.
Which programming languages are most used in #scrapy repositories?
Among the top ranked #scrapy projects on GitHub, the most common programming languages are Python, Go, Java, Shell.
How is the #scrapy repository ranking calculated?
We track 38 active open source repositories tagged with #scrapy (500+ stars), totaling 84,191 stars with an average of 2,215 stars per repository. Rankings update daily from the GitHub API.