Git Stars Git Stars

#web-crawler Repositories

Discover 39 active open source GitHub repositories tagged with #web-crawler, accumulating 470,326 total stars (averaging 12,059 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.

39
Total Repositories
470,326
Total Stars
12,059
Avg Stars/Repo
1
firecrawl/firecrawl

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

firecrawl
โญ 205,758 TypeScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-15
2
unclecode/crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

unclecode
โญ 87,627 Python ๐Ÿ‘ค unclecode User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-09
3
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

ScrapeGraphAI
โญ 32,877 Python ๐Ÿ‘ค ScrapeGraphAI Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-01-27
4
apify/crawlee

Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
โญ 26,510 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-26
5
lexmount/moli

Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust

lexmount
โญ 24,190 Rust ๐Ÿ‘ค lexmount Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-08-10
6
crawlab-team/crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

crawlab-team
โญ 12,301 Go ๐Ÿ‘ค crawlab-team Organization ๐Ÿ• Updated 2026-02-10 ๐Ÿ“… Created 2019-02-10
7
apify/crawlee-python

Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
โญ 9,679 Python ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-01-10
8
adithya-s-k/omniparse

Ingest, parse, and optimize any data format โžก๏ธ from documents to multimedia โžก๏ธ for enhanced compatibility with GenAI frameworks

adithya-s-k
โญ 7,935 Python ๐Ÿ‘ค adithya-s-k User ๐Ÿ• Updated 2025-12-12 ๐Ÿ“… Created 2024-06-04
9
firecrawl/firecrawl-mcp-server

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

firecrawl
โญ 7,871 JavaScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-12-06
10
BruceDone/awesome-crawler

A collection of awesome web crawler,spider in different languages

BruceDone
โญ 7,051 ๐Ÿ‘ค BruceDone User ๐Ÿ• Updated 2024-06-16 ๐Ÿ“… Created 2016-10-10
11
KnockOutEZ/wigolo

The go-to web for your AI coding agent โ€” local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

KnockOutEZ
โญ 5,560 TypeScript ๐Ÿ‘ค KnockOutEZ User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-12
12
divinelabio/Argus

The Ultimate Information Gathering Toolkit

divinelabio
โญ 4,348 Python ๐Ÿ‘ค divinelabio User ๐Ÿ• Updated 2025-12-10 ๐Ÿ“… Created 2024-10-01
13
Anakin-Inc/anakin

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Anakin-Inc
โญ 3,764 Go ๐Ÿ‘ค Anakin-Inc Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2026-03-17
14
apache/nutch

Apache Nutch is an extensible and scalable web crawler

apache
โญ 3,311 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2009-05-21
15
oxylabs/ai-crawler-py

Crawl a website starting from a URL, find relevant pages, and extract data โ€“ all guided by your natural language prompt.

oxylabs
โญ 3,264 ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2025-09-17
16
spider-rs/spider

Foundational low-latency web data collection in Rust

spider-rs
โญ 2,856 Rust ๐Ÿ‘ค spider-rs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-07
17
0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data โ€” all from Rust. CLI, REST API, and MCP server.

0xMassi
โญ 2,440 Rust ๐Ÿ‘ค 0xMassi User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-10
18
sjdirect/abot

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

sjdirect
โญ 2,308 C# ๐Ÿ‘ค sjdirect User ๐Ÿ• Updated 2026-07-17 ๐Ÿ“… Created 2013-08-08
19
MarginaliaSearch/MarginaliaSearch

Internet search engine for text-oriented websites. Indexing the small, old and weird web.

MarginaliaSearch
โญ 2,241 Java ๐Ÿ‘ค MarginaliaSearch Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-03-20
20
gildas-lormeau/single-file-cli

CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

gildas-lormeau
โญ 1,702 JavaScript ๐Ÿ‘ค gildas-lormeau User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-05-31
21
gigablast/open-source-search-engine

Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.

gigablast
โญ 1,599 C++ ๐Ÿ‘ค gigablast User ๐Ÿ• Updated 2024-01-10 ๐Ÿ“… Created 2013-08-02
22
GBSOSS/skill-from-masters

Stand on the shoulders of giants โ€” Create AI skills built on proven methodologies from domain experts.

GBSOSS
โญ 1,592 ๐Ÿ‘ค GBSOSS Organization ๐Ÿ• Updated 2026-02-16 ๐Ÿ“… Created 2026-01-08
23
JustinBeckwith/linkinator

Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.

JustinBeckwith
โญ 1,290 TypeScript ๐Ÿ‘ค JustinBeckwith User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-02-12
24
fastcrw/crw

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

fastcrw
โญ 1,284 Rust ๐Ÿ‘ค fastcrw Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-02
25
webrecorder/browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

webrecorder
โญ 1,199 TypeScript ๐Ÿ‘ค webrecorder Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2020-11-02
26
PhialsBasement/LibreCrawl

Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!

PhialsBasement
โญ 1,133 Python ๐Ÿ‘ค PhialsBasement User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2025-09-27
27
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
28
apache/stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

apache
โญ 1,014 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2013-04-12
29
RimoChan/sese-engine

ใ€sese-engineใ€‘The search engine of a new era!

RimoChan
โญ 952 Python ๐Ÿ‘ค RimoChan User ๐Ÿ• Updated 2025-05-10 ๐Ÿ“… Created 2022-01-09
30
firecrawl/firecrawl-app-examples

๐Ÿ”ฅ This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

firecrawl
โญ 847 Jupyter Notebook ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-01-13 ๐Ÿ“… Created 2024-06-22
31
postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

postmodern
โญ 835 Ruby ๐Ÿ‘ค postmodern User ๐Ÿ• Updated 2026-01-12 ๐Ÿ“… Created 2009-03-08
32
johnlui/DIYSearchEngine

๐Ÿ” Open-source internet search engine developed in Go, with the tutorial "Build your own internet search engine"

johnlui
โญ 706 Go ๐Ÿ‘ค johnlui User ๐Ÿ• Updated 2026-08-27 ๐Ÿ“… Created 2023-07-01
33
Team-intN18-SoybeanSeclab/Phantom

A browser extension designed for page information gathering scenarios in SRC vulnerability discovery, which automatically collects sensitive information and suspicious clues from pages and related resources, supporting basic scanning, deep recursive scanning, bulk API testing, result export, and custom regex configuration.

Team-intN18-SoybeanSeclab
โญ 696 JavaScript ๐Ÿ‘ค Team-intN18-SoybeanSeclab Organization ๐Ÿ• Updated 2026-09-16 ๐Ÿ“… Created 2025-08-21
34
Madi-S/Lead-Generation

Python script that empowers people with no programming background to generate robust leads at scale. This repository will consist of various versatile techniques used in lead generation.

Madi-S
โญ 693 Python ๐Ÿ‘ค Madi-S User ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2020-12-14
35
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17
36
vakra-dev/reader

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

vakra-dev
โญ 577 TypeScript ๐Ÿ‘ค vakra-dev Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-01-07
37
jpjacobpadilla/Stealth-Requests

Undetected web-scraping & seamless HTML parsing in Python!

jpjacobpadilla
โญ 570 Python ๐Ÿ‘ค jpjacobpadilla User ๐Ÿ• Updated 2026-09-19 ๐Ÿ“… Created 2024-08-04
38
lefterisloukas/edgar-crawler

The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files. Presented at WWW 2025 @ Sydney, Australia (https://dl.acm.org/doi/10.1145/3701716.3715289)

lefterisloukas
โญ 535 Python ๐Ÿ‘ค lefterisloukas User ๐Ÿ• Updated 2025-07-18 ๐Ÿ“… Created 2021-09-24
39
erevus-cn/pocscan

Will to be a niubility scan-framework

erevus-cn
โญ 520 Python ๐Ÿ‘ค erevus-cn User ๐Ÿ• Updated 2025-07-01 ๐Ÿ“… Created 2016-02-02

Frequently Asked Questions

1

What is the most popular #web-crawler repository on GitHub?

The top ranked open source #web-crawler project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

2

Which programming languages are most used in #web-crawler repositories?

Among the top ranked #web-crawler projects on GitHub, the most common programming languages are Python, TypeScript, Rust, Go.

3

How is the #web-crawler repository ranking calculated?

We track 39 active open source repositories tagged with #web-crawler (500+ stars), totaling 470,326 stars with an average of 12,059 stars per repository. Rankings update daily from the GitHub API.