#web-crawler Repositories
Discover 39 active open source GitHub repositories tagged with #web-crawler, accumulating 470,326 total stars (averaging 12,059 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.
39 repositories total
Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
CrawleeโA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust
Distributed web crawler admin platform for spiders management regardless of languages and frameworks.
CrawleeโA web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Ingest, parse, and optimize any data format โก๏ธ from documents to multimedia โก๏ธ for enhanced compatibility with GenAI frameworks
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
The go-to web for your AI coding agent โ local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Crawl a website starting from a URL, find relevant pages, and extract data โ all guided by your natural language prompt.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data โ all from Rust. CLI, REST API, and MCP server.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.
Stand on the shoulders of giants โ Create AI skills built on proven methodologies from domain experts.
Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Run a high-fidelity browser-based web archiving crawler in a single Docker container
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
๐ฅ This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
๐ Open-source internet search engine developed in Go, with the tutorial "Build your own internet search engine"
A browser extension designed for page information gathering scenarios in SRC vulnerability discovery, which automatically collects sensitive information and suspicious clues from pages and related resources, supporting basic scanning, deep recursive scanning, bulk API testing, result export, and custom regex configuration.
Python script that empowers people with no programming background to generate robust leads at scale. This repository will consist of various versatile techniques used in lead generation.
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files. Presented at WWW 2025 @ Sydney, Australia (https://dl.acm.org/doi/10.1145/3701716.3715289)
Frequently Asked Questions
What is the most popular #web-crawler repository on GitHub?
The top ranked open source #web-crawler project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.
Which programming languages are most used in #web-crawler repositories?
Among the top ranked #web-crawler projects on GitHub, the most common programming languages are Python, TypeScript, Rust, Go.
How is the #web-crawler repository ranking calculated?
We track 39 active open source repositories tagged with #web-crawler (500+ stars), totaling 470,326 stars with an average of 12,059 stars per repository. Rankings update daily from the GitHub API.