#crawler Repositories
162 repositories total
A user-friendly Bilibili comic downloader with a graphical interface, supporting keyword search for comics and QR code login, black technology for downloading unlocked chapters, multi-threaded downloads, various saving formats, local comic management, and one-click update checking!
๐ quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
Run a high-fidelity browser-based web archiving crawler in a single Docker container
๐ฐ Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.
Easily download all the photos/videos from tumblr blogs. Download pictures and videos from specified Tumblr blogs
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
BaiduSpider, a crawler that crawls Baidu search results, currently supports Baidu web search, Baidu image search, Baidu Knows search, Baidu video search, Baidu news search, Baidu Wenku search, Baidu experience search, and Baidu Baike search.
Golang Short Video Watermark Removal: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, Quanmin Xiaoshipin, Pipixia Gaoqiao, Xigua Video, Huya, Li Video, AcFun, Haokan Video...
Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.
Kernel-level antidetect browser with the Playwright API you already write. Python + Node SDKs, MCP-ready, unlimited local profiles. Linux x64 + arm64, macOS Intel + Apple Silicon, Windows x64.
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
Novel Download|Novel Scraping|่ตท็น|็ฌ่ถฃ้|Export Markdown|Export txt|Convert epub|Ad Filtering|Auto Proofreading
Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets
Crawler reverse engineering cases, completed: TLS fingerprint | Ruishu | Zhenkunxing | NetEase Yudun | WeChat Mini Program Decompilation Reverse Engineering (Baida Xingxi) | Tonghuashun | rpc decryption | Jiasule | Jueyan slider CAPTCHA | Julang Sanshu | Boss Zhipin | Qichacha | China Minmetals | QQ Music | Industrial Policy Big Data Platform | Qizhidiao | Xueqiu.com (acw_sc__v2) | 1688 | Qimai Data | whggzy | Qiming Keji | mohurd | Ejoin Data | Ouke Yunlian
High-performance asynchronous Captcha Solver(verification code resolver) for Douyin(ๆ้ณ), TikTok, Xiaohongshu(ๅฐ็บขไนฆ), Kuaishou(ๅฟซๆ), Weibo(ๅพฎๅ), Instagram, YouTube(ๆฒน็ฎก), Twitter(X) with Temp Mail(temporary email) API(interface).
Google play scraper for Python inspired by <facundoolano/google-play-scraper>
ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
The Big Brother is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, crypto wallet analysis, SSL intelligence, digital footprint reconstruction, EXIF extraction, advanced dorking, and network reconnaissance.
JS Cookie Reverse Engineering Power Tool: JS Cookie Change Monitoring Visualization Tool & JS Cookie Hook for Conditional Breakpoints
Code repository for the book "Advanced Practice of Web Scraping Reverse Engineering"
๐ต๏ธโโ๏ธ LinkedIn profile scraper returning structured profile data in JSON.
Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.
Search google, bing, yahoo, and other search engines with python