Git Stars Git Stars

#web-crawling Repositories

Discover 38 active open source GitHub repositories tagged with #web-crawling, accumulating 211,837 total stars (averaging 5,574 stars per project). Leading this topic is unclecode/crawl4ai with 87,627 stars.

38
Total Repositories
211,837
Total Stars
5,574
Avg Stars/Repo
1
unclecode/crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

unclecode
โญ 87,627 Python ๐Ÿ‘ค unclecode User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-09
2
apify/crawlee

Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
โญ 26,510 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-26
3
lexmount/moli

Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust

lexmount
โญ 24,190 Rust ๐Ÿ‘ค lexmount Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-08-10
4
apify/crawlee-python

Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
โญ 9,679 Python ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-01-10
5
prerender/prerender

Node server that uses Headless Chrome to render a javascript-rendered page as HTML. To be used in conjunction with prerender middleware.

prerender
โญ 6,577 JavaScript ๐Ÿ‘ค prerender Organization ๐Ÿ• Updated 2026-05-05 ๐Ÿ“… Created 2013-09-26
6
MontFerret/ferret

Declarative data automation language and Go runtime for structured extraction workflows.

MontFerret
โญ 6,025 Go ๐Ÿ‘ค MontFerret Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-08-23
7
omkarcloud/botasaurus

The All in One Framework to Build Undefeatable Scrapers

omkarcloud
โญ 5,857 TypeScript ๐Ÿ‘ค omkarcloud Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-05-09
8
google/robotstxt

The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).

google
โญ 3,478 C++ ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-04-01 ๐Ÿ“… Created 2019-06-11
9
CrawlScript/WebCollector

WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.

CrawlScript
โญ 3,073 Java ๐Ÿ‘ค CrawlScript Organization ๐Ÿ• Updated 2026-02-10 ๐Ÿ“… Created 2014-07-12
10
digininja/CeWL

CeWL is a Custom Word List Generator

digininja
โญ 2,896 Ruby ๐Ÿ‘ค digininja User ๐Ÿ• Updated 2026-06-24 ๐Ÿ“… Created 2016-04-22
11
brightdata/brightdata-mcp

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

brightdata
โญ 2,733 JavaScript ๐Ÿ‘ค brightdata Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-04-15
12
tavily-ai/tavily-mcp

Production ready MCP server with real-time search, extract, map & crawl.

tavily-ai
โญ 2,499 TypeScript ๐Ÿ‘ค tavily-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-01-27
13
dataabc/weibo-search

Get Weibo search result information, the search can be either Weibo keyword search or Weibo topic search

dataabc
โญ 2,340 Python ๐Ÿ‘ค dataabc User ๐Ÿ• Updated 2026-06-05 ๐Ÿ“… Created 2020-03-25
14
coleam00/mcp-crawl4ai-rag

Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants

coleam00
โญ 2,224 Python ๐Ÿ‘ค coleam00 User ๐Ÿ• Updated 2025-07-25 ๐Ÿ“… Created 2025-05-03
15
zhegexiaohuozi/SeimiCrawler

A simple, agile, distributed Java crawler framework supporting SpringBoot; An agile, distributed crawler framework.

zhegexiaohuozi
โญ 1,984 Java ๐Ÿ‘ค zhegexiaohuozi User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2015-11-06
16
firecrawl/fireplexity

๐Ÿ”ฅ Open Source Perplexity like AI search engine with real-time citations, streaming responses, and live data powered by Firecrawl

firecrawl
โญ 1,956 TypeScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2025-08-22 ๐Ÿ“… Created 2025-06-19
17
DanMcInerney/xsscrapy

XSS spider - 66/66 wavsep XSS detected

DanMcInerney
โญ 1,731 Python ๐Ÿ‘ค DanMcInerney User ๐Ÿ• Updated 2024-06-13 ๐Ÿ“… Created 2014-06-23
18
xnl-h4ck3r/xnLinkFinder

A python tool used to discover endpoints, potential parameters, a target specific wordlist for a given target and secrets.

xnl-h4ck3r
โญ 1,599 Python ๐Ÿ‘ค xnl-h4ck3r User ๐Ÿ• Updated 2026-03-08 ๐Ÿ“… Created 2022-04-14
19
submato/xhscrawl

Xiaohongshu data, XHS data collection, Xiaohongshu reverse engineering, Xiaohongshu XS reverse engineering, Xiaohongshu scraper, Xiaohongshu data

submato
โญ 1,533 ๐Ÿ‘ค submato User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2023-07-05
20
ShilongLee/Crawler

Open-source API interface server for crawling platforms like Douyin (a_bogus latest version), Kuaishou, Bilibili, Xiaohongshu, Taobao, JD.com, Weibo, etc. One-click quick deployment with Docker.

ShilongLee
โญ 1,490 JavaScript ๐Ÿ‘ค ShilongLee User ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2024-05-24
21
wummel/linkchecker

check links in web documents or full websites

wummel
โญ 1,433 Python ๐Ÿ‘ค wummel User ๐Ÿ• Updated 2024-11-07 ๐Ÿ“… Created 2012-12-19
22
regenrek/deepwiki-mcp

๐Ÿ“– MCP server for fetch deepwiki.com and get latest knowledge in Cursor and other Code Editors

regenrek
โญ 1,393 TypeScript ๐Ÿ‘ค regenrek User ๐Ÿ• Updated 2026-03-20 ๐Ÿ“… Created 2025-04-28
23
mvdbos/php-spider

A configurable and extensible PHP web spider

mvdbos
โญ 1,338 PHP ๐Ÿ‘ค mvdbos User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2013-02-25
24
scrapinghub/frontera

A scalable frontier for web crawlers

scrapinghub
โญ 1,329 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2025-06-06 ๐Ÿ“… Created 2014-11-22
25
Darwin-lfl/langmanus

A community-driven AI automation framework that builds upon the incredible work of the open source community. Our goal is to combine language models with specialized tools for tasks like web search, crawling, and Python code execution, while giving back to the community that made this possible.

Darwin-lfl
โญ 1,317 ๐Ÿ‘ค Darwin-lfl User ๐Ÿ• Updated 2025-03-17 ๐Ÿ“… Created 2025-03-18
26
harlan-zw/mdream

โ˜๏ธ The fastest HTML to markdown convertor on GitHub. Optimized for LLMs and supports streaming.

harlan-zw
โญ 1,032 TypeScript ๐Ÿ‘ค harlan-zw User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-16
27
changfengoss/pub

Scientific internet access tools

changfengoss
โญ 1,007 ๐Ÿ‘ค changfengoss User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2021-08-30
28
tiliondev/fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

tiliondev
โญ 817 Python ๐Ÿ‘ค tiliondev Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-07-02
29
hyperbrowserai/mcp

A MCP server implementation for hyperbrowser

hyperbrowserai
โญ 795 TypeScript ๐Ÿ‘ค hyperbrowserai Organization ๐Ÿ• Updated 2025-11-20 ๐Ÿ“… Created 2025-02-25
30
rubycdp/vessel

Fast high-level web crawling Ruby framework

rubycdp
โญ 725 Ruby ๐Ÿ‘ค rubycdp Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2019-09-17
31
spatie/laravel-export

Create a static site bundle from a Laravel app

spatie
โญ 701 PHP ๐Ÿ‘ค spatie Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2019-03-18
32
muratcankoylan/AI-Investigator

An automated AI system (Python framework) designed to analyze any type of website content and generate structured reports using Claude 3.5 Sonnet API and Firecrawl. While currently configured for enterprise AI case studies, the architecture can be adapted for various content analysis needs.

muratcankoylan
โญ 679 Python ๐Ÿ‘ค muratcankoylan User ๐Ÿ• Updated 2024-11-05 ๐Ÿ“… Created 2024-11-04
33
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17
34
ArchiveTeam/wpull

Wget-compatible web downloader and crawler.

ArchiveTeam
โญ 595 HTML ๐Ÿ‘ค ArchiveTeam Organization ๐Ÿ• Updated 2024-04-29 ๐Ÿ“… Created 2013-12-07
35
unconv/gpt4v-browsing

Web Scraping with GPT-4 Vision API and Puppeteer

unconv
โญ 563 JavaScript ๐Ÿ‘ค unconv User ๐Ÿ• Updated 2024-01-31 ๐Ÿ“… Created 2023-11-24
36
detectify/page-fetch

Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values

detectify
โญ 530 Go ๐Ÿ‘ค detectify Organization ๐Ÿ• Updated 2026-07-03 ๐Ÿ“… Created 2021-06-03
37
firecrawl/llmstxt-generator

๐Ÿ”ฅ Generate llms.txt and llms-full.txt files for any website!

firecrawl
โญ 528 TypeScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2025-06-17 ๐Ÿ“… Created 2024-11-19
38
brave-people/Dev-Event-Subscribe

๐ŸŽ‰๐ŸŽˆ The โ‰ช ๐Ÿ“ง Dev Event Email Subscription Service โ‰ซ you receive weekly by email

brave-people
โญ 512 ๐Ÿ‘ค brave-people Organization ๐Ÿ• Updated 2024-06-07 ๐Ÿ“… Created 2021-02-08

Frequently Asked Questions

1

What is the most popular #web-crawling repository on GitHub?

The top ranked open source #web-crawling project on GitHub is unclecode/crawl4ai with 87,627 stars. Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

2

Which programming languages are most used in #web-crawling repositories?

Among the top ranked #web-crawling projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Go.

3

How is the #web-crawling repository ranking calculated?

We track 38 active open source repositories tagged with #web-crawling (500+ stars), totaling 211,837 stars with an average of 5,574 stars per repository. Rankings update daily from the GitHub API.