#crawler Repositories
Discover 162 active open source GitHub repositories tagged with #crawler, accumulating 1,111,814 total stars (averaging 6,863 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.
162 repositories total
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy—join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
🚀 Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader — asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.
Turn any website into a structured API. Extract, automate, search and monitor the web.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Some very interesting Python crawler examples, friendly to beginners, mainly crawling websites like Taobao, Tmall, WeChat, WeChat Reading, Douban, QQ, etc. (Some interesting examples of python crawlers that are friendly to beginners.)
Distributed web crawler admin platform for spiders management regardless of languages and frameworks.
Python script. Simulate login to Zhihu, web crawling, manipulate Excel, WeChat public account, remote power‑on.
AV Movie Management System, avmoo, javbus, javlibrary Crawler, Online AV Film Library, AV Magnet Link Database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
List of libraries, tools and APIs for web scraping and data processing.
Python API for JMComic | Provides Python API access to JinManTiantang, supporting both web and mobile versions | JinManTiantang GitHub Actions Downloader🚀
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Practical 🐍 multiple websites, e-commerce data crawlers 🕷. Includes 🕸: Taobao products, WeChat public accounts, Dianping, Qichacha, job websites, Xianyu, Ali tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu.com, Quan jing wang, Douban music, some province's drug supervision bureau, Sohu news, machine learning text collection, fofa asset collection, Autohome, National Bureau of Statistics, Baidu keyword inclusion count, spider general directory, Toutiao, Douban movie reviews, Ctrip, Xiaomi app store, Anjuke, Tujia homestay ❤️❤️❤️. WeChat crawler demo project:
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
This project is designed to compile all news, documentation, and legal regulations related to lawsuits and violations involving web crawler developers in mainland China. It aims to help professionals in the web crawler industry working in mainland China understand relevant laws and avoid infringing data compliance boundaries.
Sina Weibo Crawler, using Python to crawl Sina Weibo data, and download Weibo images and videos
A community-driven way to read and chat with AI bots - powered by chatGPT.
Download comics novels tool:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Multi-platform content monitoring · collection · reposting. One web dashboard to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok / Bilibili / Instagram / Xiaohongshu overseas (rednote)
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder is an easy to use, powerful Python crawler framework. It has four built-in spider types: AirSpider, Spider, TaskSpider, and BatchSpider to meet the needs of different scenarios. It supports features such as breakpoint resuming, monitoring and alerting, browser rendering, and deduplicating massive amounts of data. Furthermore, the powerful crawler management system feaplat provides convenient deployment and scheduling.
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Videodl: A lightweight video downloader written in pure Python. (A lightweight video downloader that prioritizes high-definition, watermark‑free downloads and supports platforms such as Douyin, Kuaishou, Xiaohongshu, Bilibili, TikTok, YouTube, FIFA+, Youku, Tencent Video, iQIYI, 1905 Movie Network, LeTV, Mango TV, Migu, PPTV, Sohu, Facebook, Twitter, Sina Weibo, Today’s Headlines, NetEase Open Classes, 全民K歌, CCTV, Kugou MV, New Film, Zhihu, Baidu Tieba, TED, and other vast streaming platforms.)
Frequently Asked Questions
What is the most popular #crawler repository on GitHub?
The top ranked open source #crawler project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Which programming languages are most used in #crawler repositories?
Among the top ranked #crawler projects on GitHub, the most common programming languages are Python, Go, TypeScript, JavaScript.
How is the #crawler repository ranking calculated?
We track 162 active open source repositories tagged with #crawler (500+ stars), totaling 1,111,814 stars with an average of 6,863 stars per repository. Rankings update daily from the GitHub API.