#spider Repositories
Discover 92 active open source GitHub repositories tagged with #spider, accumulating 325,814 total stars (averaging 3,541 stars per project). Leading this topic is NaiboWang/EasySpider with 44,801 stars.
92 repositories total
A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.
Some very interesting Python crawler examples, friendly to beginners, mainly crawling websites like Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ, etc. (Some interesting examples of python crawlers that are friendly to beginners.)
Distributed web crawler admin platform for spiders management regardless of languages and frameworks.
A live recording software with cyclic duty and multiplayer recording, supporting live recording of 40+ platforms such as Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, Shopee, etc.
AV Movie Management System, avmoo, javbus, javlibrary Crawler, Online AV Film Library, AV Magnet Link Database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
INFO-SPIDER is a crawler toolbox๐งฐ integrating numerous data sources, aiming to help users retrieve their data safely and quickly. The tool's code is open source, and the process is transparent. Supported data sources include GitHub, QQ Mail, Netease Mail, Ali Mail, Sina Mail, Hotmail, Outlook, JD, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, Netease Cloud Music, QQ Friends, QQ Groups, Generating WeChat Moments albums, Browser browsing history, 12306, Blog Garden, CSDN Blog, OSCHINA Blog, JianShu.
List of libraries, tools and APIs for web scraping and data processing.
One-click synchronization of articles to multiple content platforms, supports Toutiao, WordPress, Zhihu, JianShu, Juejin, CSDN, Typecho and other major platforms, publish once, sync to multiple platforms. Free up personal productivity.
novel-plus is a feature-rich, multi-platform (PC, WAP, app, and mini program) novel CMS. It includes features such as novel recommendations, search, rankings, reading, bookshelves, reviews, reading lists, web scraping, a member center, a writersโ section, top-ups and subscriptions, and news publishing.
Common Bilibili API calls. Supports functions such as videos, animes, users, channels, audio, etc. Original repository address: https://github.com/MoyuScript/bilibili-api
๐๐๐feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs ๆๆกฃ :point_right:
Short video watermark removal: Douyin, Pipixia, Huoshan, Weishi, Weibo, Oasis, Zuiyou, Qingshipin, Kuaishou, Quanmin Xiaoshipin, Barcelona Film, Momo, Before Bifeng, Kaiyan, Vue Vlog Xiaokuxiu, Pipixia Funny, Quanmin K Ge, Xigua Video, Doupai, Huya, 6 Rooms, Li Video, Xinpianchang, acfun, Meipai...
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Go Language Example Tutorial from Beginner to Advanced, Including Basic Library Usage, Design Patterns, Common Interview Pitfalls, Utility Classes, Integrating Third-Party Services, etc.
:spider: The progressive PHP crawler framework! ไผ้ ็ๆธ่ฟๅผPHP้้ๆกๆถใ
๐ท CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Download images from Google, Bing, Baidu. Google, Baidu, Bing image download.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.
QQ Space Export Assistant, used to back up QQ Space posts, journals, private diaries, albums, videos, message boards, QQ friends, favorites, shares, and recent visitors as files for easy migration and preservation.
A Python based web application scanner to gather OSINT and fuzz for OWASP vulnerabilities on a target website.
BOSS Zhipin scraper / BOSS Zhipin job data crawler, based on Chrome CDP protocol to reuse real login sessions, bypass font anti-scraping, and output plaintext salary JSON/CSV + salary-skill analysis.
Douyin Crawler - Collect public data such as account homepage, likes, collections, music original sound, topics, search, collections, works, following, followers, etc.
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Python crawlers, current inventory: NetEase Cloud Music song crawling, Bilibili video crawling, Zhihu Q&A crawling, wallpaper crawling, xvideos video crawling, audiobook crawling, Weibo crawler, Anjuke information crawling + data visualization, Bilibili video cover extractor, IP proxy pool encapsulation, Zhihu million-level user crawler + data analysis, GitHub user crawler
๐ฐ Brazilian official gazettes accessible to all | ๐ฐ Brazilian government gazettes, accessible to everyone.
Brother K Web Scraping Code Sharing, JS Reverse Engineering, Advanced Web Scraping. Follow the WeChat official account: Brother K Web Scraping
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Frequently Asked Questions
What is the most popular #spider repository on GitHub?
The top ranked open source #spider project on GitHub is NaiboWang/EasySpider with 44,801 stars. A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.
Which programming languages are most used in #spider repositories?
Among the top ranked #spider projects on GitHub, the most common programming languages are Python, Go, PHP, TypeScript.
How is the #spider repository ranking calculated?
We track 92 active open source repositories tagged with #spider (500+ stars), totaling 325,814 stars with an average of 3,541 stars per repository. Rankings update daily from the GitHub API.