#web-crawling Repositories
Discover 38 active open source GitHub repositories tagged with #web-crawling, accumulating 211,837 total stars (averaging 5,574 stars per project). Leading this topic is unclecode/crawl4ai with 87,627 stars.
38 repositories total
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
CrawleeโA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust
CrawleeโA web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Node server that uses Headless Chrome to render a javascript-rendered page as HTML. To be used in conjunction with prerender middleware.
Declarative data automation language and Go runtime for structured extraction workflows.
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Get Weibo search result information, the search can be either Weibo keyword search or Weibo topic search
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
A simple, agile, distributed Java crawler framework supporting SpringBoot; An agile, distributed crawler framework.
๐ฅ Open Source Perplexity like AI search engine with real-time citations, streaming responses, and live data powered by Firecrawl
A python tool used to discover endpoints, potential parameters, a target specific wordlist for a given target and secrets.
Xiaohongshu data, XHS data collection, Xiaohongshu reverse engineering, Xiaohongshu XS reverse engineering, Xiaohongshu scraper, Xiaohongshu data
Open-source API interface server for crawling platforms like Douyin (a_bogus latest version), Kuaishou, Bilibili, Xiaohongshu, Taobao, JD.com, Weibo, etc. One-click quick deployment with Docker.
๐ MCP server for fetch deepwiki.com and get latest knowledge in Cursor and other Code Editors
A community-driven AI automation framework that builds upon the incredible work of the open source community. Our goal is to combine language models with specialized tools for tasks like web search, crawling, and Python code execution, while giving back to the community that made this possible.
โ๏ธ The fastest HTML to markdown convertor on GitHub. Optimized for LLMs and supports streaming.
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
An automated AI system (Python framework) designed to analyze any type of website content and generate structured reports using Claude 3.5 Sonnet API and Firecrawl. While currently configured for enterprise AI case studies, the architecture can be adapted for various content analysis needs.
Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values
๐๐ The โช ๐ง Dev Event Email Subscription Service โซ you receive weekly by email
Frequently Asked Questions
What is the most popular #web-crawling repository on GitHub?
The top ranked open source #web-crawling project on GitHub is unclecode/crawl4ai with 87,627 stars. Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Which programming languages are most used in #web-crawling repositories?
Among the top ranked #web-crawling projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Go.
How is the #web-crawling repository ranking calculated?
We track 38 active open source repositories tagged with #web-crawling (500+ stars), totaling 211,837 stars with an average of 5,574 stars per repository. Rankings update daily from the GitHub API.