#crawler Repositories
162 repositories total
NetDiscovery is a general-purpose crawler framework/middleware implemented based on frameworks such as Vert.x and RxJava 2.
One command to operate Xiaohongshu, Douyin, TikTok, Bilibili, Kuaishou, Weibo, Xianyu, Taobao, JD.com, and X: unified commands and JSON output for people and AI agents
Locally saves webpages to your hard disk with images, css, js & links as is.
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features encompasses vulnerability scanning, information gathering, and exploitation, elevating it to an indispensable toolkit for both security professionals and penetration testers. ๆๆด่พ ๅฉๅทฅๅ ท(ๆผๆดๆซๆใไฟกๆฏๆถ้)
FreeProxy: Collecting free proxies from the internet. (Massive high-quality free proxies worldwide, supporting crawling from dozens of free proxy sharing sources, custom rule-based proxy filtering, essential for web scraping and data analysis, updated daily with a vast number of free proxies.)
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
โ๏ธ Self-hosted, multi-channel airfare monitoring: collect detailed flight data โ send text-and-image alerts via DingTalk โ trigger strong, four-tier alerts when targets are met. Domestic and international routes (destinations across five continents) use the same collectโevaluateโnotify workflow. Data stays on your machine, all settings are managed on the web, and itโs open source and free.