Git Stars Git Stars

#web-scraping Repositories

Discover 381 active open source GitHub repositories tagged with #web-scraping, accumulating 2,147,590 total stars (averaging 5,636 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.

381
Total Repositories
2.1M
Total Stars
5,636
Avg Stars/Repo
1
firecrawl/firecrawl

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

firecrawl
โญ 205,758 TypeScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-15
2
jackfrued/Python-100-Days

Python - 100 Days from Beginner to Master

jackfrued
โญ 187,961 Jupyter Notebook ๐Ÿ‘ค jackfrued User ๐Ÿ• Updated 2026-07-29 ๐Ÿ“… Created 2018-03-01
3
ytdl-org/youtube-dl

Command-line program to download videos from YouTube.com and other video sites

ytdl-org
โญ 141,627 Python ๐Ÿ‘ค ytdl-org Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2010-10-31
4
browser-use/browser-use

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

browser-use
โญ 121,654 Python ๐Ÿ‘ค browser-use Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-10-31
5
D4Vinci/Scrapling

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโ€”join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

D4Vinci
โญ 104,760 Python ๐Ÿ‘ค D4Vinci User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2024-10-13
6
unclecode/crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

unclecode
โญ 87,627 Python ๐Ÿ‘ค unclecode User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-09
7
NanmiCoder/MediaCrawler

Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝœComment Crawler

NanmiCoder
โญ 71,528 Python ๐Ÿ‘ค NanmiCoder User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-06-09
8
scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy
โญ 65,519 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2010-02-22
9
soimort/you-get

:arrow_double_down: Dumb downloader that scrapes the web

soimort
โญ 56,851 Python ๐Ÿ‘ค soimort User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2012-08-20
10
gedoor/legado

Legado 3.0 Book Reader with powerful controls & full functions โค๏ธ Reading 3.0, Reading is a tool that allows custom sources to read online content, providing network literature enthusiasts with a convenient, fast, and comfortable trial reading experience.

gedoor
โญ 47,268 Kotlin ๐Ÿ‘ค gedoor User ๐Ÿ• Updated 2026-05-27 ๐Ÿ“… Created 2019-05-22
11
JCodesMore/ai-website-cloner-template

Clone any website with one command using AI coding agents

JCodesMore
โญ 39,582 TypeScript ๐Ÿ‘ค JCodesMore User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-03-13
12
dgtlmoon/changedetection.io

The best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoringโ€”with MCP included! All for free, or enjoy our SaaS plan!

dgtlmoon
โญ 35,939 Python ๐Ÿ‘ค dgtlmoon User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-01-27
13
CloakHQ/CloakBrowser

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.

CloakHQ
โญ 33,007 Python ๐Ÿ‘ค CloakHQ Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-02-22
14
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

ScrapeGraphAI
โญ 32,877 Python ๐Ÿ‘ค ScrapeGraphAI Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-01-27
15
apify/crawlee

Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
โญ 26,510 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-26
16
browserbase/stagehand

The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.

browserbase
โญ 26,109 TypeScript ๐Ÿ‘ค browserbase Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-03-24
17
lexmount/moli

Best headless browser for AI agents. Lightweight, Fast, Highly Compatible. Built in Rust

lexmount
โญ 24,190 Rust ๐Ÿ‘ค lexmount Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-08-10
18
Evil0ctal/Douyin_TikTok_Download_API

๐Ÿš€ Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader โ€” asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.

Evil0ctal
โญ 21,507 Python ๐Ÿ‘ค Evil0ctal User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-11-07
19
getmaxun/maxun

Turn any website into a structured API. Extract, automate, search and monitor the web.

getmaxun
โญ 18,538 TypeScript ๐Ÿ‘ค getmaxun Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-23
20
MODSetter/SurfSense

Air-gapped, privacy-focused, open-source NotebookLM alternative. An AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9

MODSetter
โญ 16,907 Python ๐Ÿ‘ค MODSetter User ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2024-07-30
21
yusufkaraaslan/Skill_Seekers

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

yusufkaraaslan
โญ 15,351 Python ๐Ÿ‘ค yusufkaraaslan User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2025-10-17
22
pjialin/py12306

๐Ÿš‚ 12306 Ticket Buying Assistant, supports clusters, multiple accounts, multi-task ticket buying, and Web page management

pjialin
โญ 14,960 Python ๐Ÿ‘ค pjialin User ๐Ÿ• Updated 2026-10-01 ๐Ÿ“… Created 2019-01-07
23
nanobrowser/nanobrowser

Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.

nanobrowser
โญ 14,359 TypeScript ๐Ÿ‘ค nanobrowser Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2024-12-31
24
seleniumbase/SeleniumBase

Fast and reliable browser automation for end-to-end testing, scraping, and stealth. Bypasses bot detection with CDP Mode. Integrates with pytest, Playwright, and MCP.

seleniumbase
โญ 13,272 Python ๐Ÿ‘ค seleniumbase Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-03-04
25
jiji262/douyin-downloader

A practical Douyin downloader for both single-item and profile batch downloads, with progress display, retries, SQLite deduplication, and browser fallback support. A Douyin batch download tool supporting videos, image albums, collections, and music (original audio).

jiji262
โญ 12,644 Python ๐Ÿ‘ค jiji262 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-25
26
jo-inc/camofox-browser

Stealth headless browser for AI agents โ€” bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

jo-inc
โญ 12,437 JavaScript ๐Ÿ‘ค jo-inc Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-01-26
27
jhy/jsoup

jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

jhy
โญ 11,418 Java ๐Ÿ‘ค jhy User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2009-12-19
28
NopeCHALLC/nopecha-extension

Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.

NopeCHALLC
โญ 11,140 ๐Ÿ‘ค NopeCHALLC User ๐Ÿ• Updated 2026-10-02 ๐Ÿ“… Created 2023-03-23
29
pinchtab/pinchtab

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

pinchtab
โญ 10,599 Go ๐Ÿ‘ค pinchtab Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-02-15
30
apify/crawlee-python

Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
โญ 9,679 Python ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-01-10
31
scrapinghub/portia

Visual scraping for Scrapy

scrapinghub
โญ 9,479 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2024-06-26 ๐Ÿ“… Created 2014-03-21
32
nashsu/FreeAskInternet

FreeAskInternet is a completely free, PRIVATE and LOCALLY running search aggregator & answer generate using MULTI LLMs, without GPU needed. The user can ask a question and the system will make a multi engine search and combine the search result to LLM and generate the answer based on search results. It's all FREE to use.

nashsu
โญ 8,734 Python ๐Ÿ‘ค nashsu User ๐Ÿ• Updated 2024-04-18 ๐Ÿ“… Created 2024-04-05
33
cv-cat/Spider_XHS

Xiaohongshu (RED) crawler data collection, Xiaohongshu reverse engineering, private messaging, live streaming, and full-domain Xiaohongshu operation solutions

cv-cat
โญ 8,542 Python ๐Ÿ‘ค cv-cat User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2023-08-08
34
cporter202/API-mega-list

This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHubโ€”period. ๐Ÿ’ช

cporter202
โญ 8,406 JavaScript ๐Ÿ‘ค cporter202 User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-12-09
35
ericchiang/pup

Parsing HTML at the command line

ericchiang
โญ 8,372 HTML ๐Ÿ‘ค ericchiang User ๐Ÿ• Updated 2024-05-02 ๐Ÿ“… Created 2014-09-01
36
mherrmann/helium

Lighter web automation with Python

mherrmann
โญ 8,333 Python ๐Ÿ‘ค mherrmann User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2019-11-26
37
gosom/google-maps-scraper

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews, email and more for each place

gosom
โญ 8,291 Go ๐Ÿ‘ค gosom User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-04-22
38
lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

lorien
โญ 8,242 JavaScript ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-08-12
39
alirezamika/autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

alirezamika
โญ 8,102 Python ๐Ÿ‘ค alirezamika User ๐Ÿ• Updated 2026-07-29 ๐Ÿ“… Created 2020-08-31
40
firecrawl/firecrawl-mcp-server

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

firecrawl
โญ 7,871 JavaScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-12-06
41
WECENG/ticket-purchase

Barley automatic ticket grabbing, supports selection of people, cities, date sessions, and prices

WECENG
โญ 7,844 Python ๐Ÿ‘ค WECENG User ๐Ÿ• Updated 2026-05-14 ๐Ÿ“… Created 2023-10-12
42
mgdm/htmlq

Like jq, but for HTML.

mgdm
โญ 7,591 Rust ๐Ÿ‘ค mgdm User ๐Ÿ• Updated 2026-05-27 ๐Ÿ“… Created 2019-05-07
43
go-rod/rod

A Chrome DevTools Protocol driver for web automation and scraping.

go-rod
โญ 7,160 Go ๐Ÿ‘ค go-rod Organization ๐Ÿ• Updated 2026-08-11 ๐Ÿ“… Created 2020-01-21
44
adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

adbar
โญ 7,157 Python ๐Ÿ‘ค adbar User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-04-08
45
autoscrape-labs/pydoll

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

autoscrape-labs
โญ 7,145 HTML ๐Ÿ‘ค autoscrape-labs Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2024-10-27
46
lexiforest/curl_cffi

Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

lexiforest
โญ 6,909 Python ๐Ÿ‘ค lexiforest User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-03-10
47
browser-act/skills

Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing.

browser-act
โญ 6,385 Python ๐Ÿ‘ค browser-act Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2026-02-06
48
nickscamara/open-deep-research

An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl

nickscamara
โญ 6,242 TypeScript ๐Ÿ‘ค nickscamara User ๐Ÿ• Updated 2025-05-07 ๐Ÿ“… Created 2025-02-03
49
MontFerret/ferret

Declarative data automation language and Go runtime for structured extraction workflows.

MontFerret
โญ 6,025 Go ๐Ÿ‘ค MontFerret Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-08-23
50
RedSiege/EyeWitness

EyeWitness is designed to take screenshots of websites, provide some server header info, and identify default credentials if possible.

RedSiege
โญ 5,906 Python ๐Ÿ‘ค RedSiege Organization ๐Ÿ• Updated 2026-01-05 ๐Ÿ“… Created 2014-02-26

Frequently Asked Questions

1

What is the most popular #web-scraping repository on GitHub?

The top ranked open source #web-scraping project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

2

Which programming languages are most used in #web-scraping repositories?

Among the top ranked #web-scraping projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Go.

3

How is the #web-scraping repository ranking calculated?

We track 381 active open source repositories tagged with #web-scraping (500+ stars), totaling 2,147,590 stars with an average of 5,636 stars per repository. Rankings update daily from the GitHub API.