#spider Repositories
92 repositories total
Beanbun is a multi-process web crawling framework written in PHP, with good openness and high scalability, based on Workerman.
BaiduSpider, a crawler that crawls Baidu search results, currently supports Baidu web search, Baidu image search, Baidu Knows search, Baidu video search, Baidu news search, Baidu Wenku search, Baidu experience search, and Baidu Baike search.
LLMs Large Models Notes 50 Articles This repository contains practical projects in fields such as machine learning, deep learning, computer vision, natural language processing, large models, web scraping, etc.
Golang Short Video Watermark Removal: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, Quanmin Xiaoshipin, Pipixia Gaoqiao, Xigua Video, Huya, Li Video, AcFun, Haokan Video...
JS reverse engineering, cracking JS anti-scraping encryption parameters, already cracked Geetest slider w (2022.2.19), QQ Music sign (2022.2.13), Pinduoduo anti_content, Boss Zhipin zp_token, Zhihu x-zse-96, Kugou kg_mid/dfid, Vipshop mars_cid, China Judgments Online (updated 2020-06-30), Taobao password, Tianan Insurance login, Bilibili login, Fang.com login, WPS login, Weibo login, Youdao Translate, Netease login, WeChat Official Account login, Air China Network login, Jmubiao login, Student Information Management System login, Gengying Finance login, Chongqing Science and Technology Resource Sharing Platform login, Netease Cloud Music download, one-click video link parsing, Cailianshe login.
Novel Download|Novel Scraping|่ตท็น|็ฌ่ถฃ้|Export Markdown|Export txt|Convert epub|Ad Filtering|Auto Proofreading
Crawler reverse engineering cases, completed: TLS fingerprint | Ruishu | Zhenkunxing | NetEase Yudun | WeChat Mini Program Decompilation Reverse Engineering (Baida Xingxi) | Tonghuashun | rpc decryption | Jiasule | Jueyan slider CAPTCHA | Julang Sanshu | Boss Zhipin | Qichacha | China Minmetals | QQ Music | Industrial Policy Big Data Platform | Qizhidiao | Xueqiu.com (acw_sc__v2) | 1688 | Qimai Data | whggzy | Qiming Keji | mohurd | Ejoin Data | Ouke Yunlian
A set of spiders and scrapers to extract location information from places that post their location on the internet.
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
:point_right: A Java stock crawler based on Spring Boot (supports only A-shares), if you :heart: please :star: . V2 upgrade version is under development!
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
A calibre source/metadata plugin based on the National Library of China's ISBN search. https://doiiars.com/article/NLCISBNPlugin
A Penetration Testing Framework, Information gathering tool & Website Vulnerability Scanner
Light Novel Packer, which scrapes content from Bili Light Novel websites (bilinovel.com linovelib.com) and packages it into EPUB format. It supports covers, illustrations, tables of contents, and merging separate volumes.
Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server
๐ต๏ธโโ๏ธ LinkedIn profile scraper returning structured profile data in JSON.
Python Short Video Watermark Removal Scraper: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, National Short Video, Pipi Funny, Xigua Video, Huya, Li Video, AcFun, Haokan Video...
Taobao Crawler SDK, used for Taobao Open Platform or logging in and crawling Taobao, Tmall, Alibaba
Enterprise-grade open source public opinion news crawler project: supports one-click operation of any number of crawlers, scheduled crawler tasks, batch deletion of crawlers; one-click deployment of crawlers; crawler monitoring visualization; configuration of cluster crawler allocation strategy; ๐ The ready-made Docker one-click deployment documentation has already solved the pitfalls for you.
Approaches and experience for reverse engineering app parameters, Android reverse engineering
Questionnaire project, cloud server recommendation, hanging project, free proxy, various script collection. Welcome to the upper right corner of the point bell to receive timely updates. (Do not fork, low profile)
NetDiscovery is a general-purpose crawler framework/middleware implemented based on frameworks such as Vert.x and RxJava 2.
One command to operate Xiaohongshu, Douyin, TikTok, Bilibili, Kuaishou, Weibo, Xianyu, Taobao, JD.com, and X: unified commands and JSON output for people and AI agents
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
Python 3 script to dump/scrape/extract company employees from LinkedIn API
spider-admin-pro A visual management tool for viewing Scrapy+Scrapyd crawler projects and scheduled timing of crawler tasks, the upgraded version of SpiderAdmin
FreeProxy: Collecting free proxies from the internet. (Massive high-quality free proxies worldwide, supporting crawling from dozens of free proxy sharing sources, custom rule-based proxy filtering, essential for web scraping and data analysis, updated daily with a vast number of free proxies.)
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
๐ฏ Bilibili comment data visualization analysis software -- content creators can use it to guide their topic selection and understand their fan base.