#crawler Repositories
162 repositories total
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
:spider: The progressive PHP crawler framework! ไผ้ ็ๆธ่ฟๅผPHP้้ๆกๆถใ
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
news-please - an integrated web crawler and information extractor for news that just works
๐ท CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
Blue Sky Collector is an open-source and free crawler system that only requires clicking and editing rules to collect data. It can run locally, on a virtual host, or on a cloud server, and can crawl almost all types of web pages. It seamlessly integrates with various CMS website building programs, publishes data in real-time without login, and is fully automated without human intervention! It is a completely cross-platform cloud crawler system in web big data collection software.
Collection of crawler cases. Including but not limited to "Taobao, JD.com, Tmall, Douban, Douyin, Kuaishou, Weibo, WeChat, Alibaba, Toutiao, PDD, Youku, iQIYI, Ctrip, 12306, 58, Sohu, various indexes, VIP, Wanfang, ZLibrary, Oalib, Novels, bidding websites, procurement websites, Xiaohongshu, Dianping, Twitter, Maimai, Zhihu"
Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch
BOSS Zhipin scraper / BOSS Zhipin job data crawler, based on Chrome CDP protocol to reuse real login sessions, bypass font anti-scraping, and output plaintext salary JSON/CSV + salary-skill analysis.
Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
โค๏ธ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)
Douyin Crawler - Collect public data such as account homepage, likes, collections, music original sound, topics, search, collections, works, following, followers, etc.
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Python crawlers, current inventory: NetEase Cloud Music song crawling, Bilibili video crawling, Zhihu Q&A crawling, wallpaper crawling, xvideos video crawling, audiobook crawling, Weibo crawler, Anjuke information crawling + data visualization, Bilibili video cover extractor, IP proxy pool encapsulation, Zhihu million-level user crawler + data analysis, GitHub user crawler
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
Xiaohongshu data collection, website image, video resource batch download tool, super high appearance data collection tool (batch download, video extraction, images) Telegram:https://t.me/+ZtLSwuIKTo44MDY1
Brother K Web Scraping Code Sharing, JS Reverse Engineering, Advanced Web Scraping. Follow the WeChat official account: Brother K Web Scraping
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Beanbun is a multi-process web crawling framework written in PHP, with good openness and high scalability, based on Workerman.