Git Stars Git Stars

#web-scraping Repositories

381
Total Repositories
2.1M
Total Stars
5,636
Avg Stars/Repo
51
matthewmueller/x-ray

The next web scraper. See through the <html> noise.

matthewmueller
โญ 5,899 JavaScript ๐Ÿ‘ค matthewmueller User ๐Ÿ• Updated 2026-08-31 ๐Ÿ“… Created 2015-02-05
52
omkarcloud/botasaurus

The All in One Framework to Build Undefeatable Scrapers

omkarcloud
โญ 5,857 TypeScript ๐Ÿ‘ค omkarcloud Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-05-09
53
executeautomation/mcp-playwright

Playwright Model Context Protocol Server - Tool to automate Browsers and APIs in Claude Desktop, Cline, Cursor IDE and More ๐Ÿ”Œ

executeautomation
โญ 5,681 TypeScript ๐Ÿ‘ค executeautomation User ๐Ÿ• Updated 2025-12-13 ๐Ÿ“… Created 2024-12-03
54
KnockOutEZ/wigolo

The go-to web for your AI coding agent โ€” local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

KnockOutEZ
โญ 5,560 TypeScript ๐Ÿ‘ค KnockOutEZ User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-12
55
Kaliiiiiiiiii-Vinyzu/patchright

Undetected version of the Playwright testing and automation library.

Kaliiiiiiiiii-Vinyzu
โญ 5,227 TypeScript ๐Ÿ‘ค Kaliiiiiiiiii-Vinyzu Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-11-07
56
Manavarya09/design-extract

Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

Manavarya09
โญ 5,180 HTML ๐Ÿ‘ค Manavarya09 User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-04-15
57
ultrafunkamsterdam/nodriver

Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha

ultrafunkamsterdam
โญ 4,950 Python ๐Ÿ‘ค ultrafunkamsterdam User ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2024-02-20
58
jaypyles/Scraperr

Self-hosted webscraper.

jaypyles
โญ 4,876 TypeScript ๐Ÿ‘ค jaypyles User ๐Ÿ• Updated 2025-10-12 ๐Ÿ“… Created 2024-07-06
59
NanmiCoder/CrawlerTutorial

Beginner, Intermediate, Advanced Web Scraping

NanmiCoder
โญ 4,873 Python ๐Ÿ‘ค NanmiCoder User ๐Ÿ• Updated 2026-04-03 ๐Ÿ“… Created 2024-03-24
60
request/request-promise

The simplified HTTP request client 'request' with Promise support. Powered by Bluebird.

request
โญ 4,734 JavaScript ๐Ÿ‘ค request Organization ๐Ÿ• Updated 2024-01-31 ๐Ÿ“… Created 2013-10-04
61
REMitchell/python-scraping

Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do

REMitchell
โญ 4,669 Jupyter Notebook ๐Ÿ‘ค REMitchell User ๐Ÿ• Updated 2024-06-01 ๐Ÿ“… Created 2015-06-14
62
AhmadHassan-BTed/Noria

An agentic extractor for messaging platforms. Noria receives messages and URLs via chat, scrapes web data to evaluate content against custom scoring matrices (e.g., jobs, scholarships), and sends a structured summary with a calculated match score directly to the assigned messenger number.

AhmadHassan-BTed
โญ 4,624 JavaScript ๐Ÿ‘ค AhmadHassan-BTed User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-05-31
63
linsomniac/spotify_to_ytmusic

Copy playlists and liked music from Spotify to YTMusic

linsomniac
โญ 4,588 Python ๐Ÿ‘ค linsomniac User ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2023-12-22
64
0xHJK/music-dl

search and download music from NetEase Cloud Music, QQ Music, KuGou Music, Baidu Music, Xiami Music, MiGu Music, etc.

0xHJK
โญ 4,423 Python ๐Ÿ‘ค 0xHJK User ๐Ÿ• Updated 2025-08-12 ๐Ÿ“… Created 2019-01-08
65
ssut/py-googletrans

(unofficial) Googletrans: Free and Unlimited Google translate API for Python. Translates totally free of charge.

ssut
โญ 4,303 Python ๐Ÿ‘ค ssut User ๐Ÿ• Updated 2026-08-08 ๐Ÿ“… Created 2015-06-05
66
RookieEnough/Orion-Store

The ultimate home for Open Source and Cleaned apps. OrionStore offers instant access to YouTube Morphe, YT Music Morphe, and essential open-source tools without the clutter. No servers, no tracking, just a beautiful, modern gateway to the apps you love.

RookieEnough
โญ 4,285 Kotlin ๐Ÿ‘ค RookieEnough User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-11-28
67
scrapinghub/splash

Lightweight, scriptable browser as a service with an HTTP API

scrapinghub
โญ 4,197 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2024-08-02 ๐Ÿ“… Created 2013-05-20
68
kalil0321/reverse-api-engineer

Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor, and OpenCode.

kalil0321
โญ 4,088 Python ๐Ÿ‘ค kalil0321 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-12-23
69
snooppr/snoop

Snoop โ€” An open-source intelligence (OSINT) tool

snooppr
โญ 4,072 Python ๐Ÿ‘ค snooppr User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-02-14
70
IonicaBizau/scrape-it

๐Ÿ”ฎ A Node.js scraper for humans.

IonicaBizau
โญ 4,069 JavaScript ๐Ÿ‘ค IonicaBizau User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2016-04-28
71
ericciarla/trendFinder

Stay on top of trending topics on social media and the web with AI

ericciarla
โญ 4,016 TypeScript ๐Ÿ‘ค ericciarla User ๐Ÿ• Updated 2025-02-17 ๐Ÿ“… Created 2024-12-31
72
osnr/TabFS

๐Ÿ—„ Mount your browser tabs as a filesystem.

osnr
โญ 3,916 JavaScript ๐Ÿ‘ค osnr User ๐Ÿ• Updated 2024-12-28 ๐Ÿ“… Created 2019-02-26
73
Samueli924/chaoxing

Superstar Learning Cloud / Super Star Er Ya / Pan Ya Superstar Fully Automatic Unattended Task Point Completion

Samueli924
โญ 3,883 Python ๐Ÿ‘ค Samueli924 User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2020-11-20
74
sentient-agi/OpenDeepSearch

SOTA search powered LLM

sentient-agi
โญ 3,824 Python ๐Ÿ‘ค sentient-agi Organization ๐Ÿ• Updated 2025-04-04 ๐Ÿ“… Created 2025-03-20
75
edoardottt/cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

edoardottt
โญ 3,817 Go ๐Ÿ‘ค edoardottt User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2021-04-27
76
3441293738/creatorhub

Multi-platform content monitoring ยท collection ยท reposting. One web dashboard to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok / Bilibili / Instagram / Xiaohongshu overseas (rednote)

3441293738
โญ 3,774 Python ๐Ÿ‘ค 3441293738 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-07-03
77
Anakin-Inc/anakin

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Anakin-Inc
โญ 3,764 Go ๐Ÿ‘ค Anakin-Inc Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2026-03-17
78
cv-cat/DouYin_Spider

TikTok reverse engineering, TikTok crawler, all TikTok APIs, private messages, live room monitoring

cv-cat
โญ 3,711 Python ๐Ÿ‘ค cv-cat User ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2023-10-17
79
oxylabs/amazon-scraper

Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.

oxylabs
โญ 3,417 Python ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-11-29
81
oxylabs/oxylabs-ai-studio-py

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.

oxylabs
โญ 3,394 Python ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2025-06-17
82
vibheksoni/stealth-browser-mcp

Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.

vibheksoni
โญ 3,384 Python ๐Ÿ‘ค vibheksoni User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-08-09
83
php-curl-class/php-curl-class

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

php-curl-class
โญ 3,293 PHP ๐Ÿ‘ค php-curl-class Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2013-08-16
84
Virtual-Browser/VirtualBrowser

Free anti fingerprint browser, fingerprint browser, privacy browser, anti recognition browser, anti identification browser, anti association browser, free web3 airdrop dedicated fingerprint browser https://virtualbrowser.cc/?src=github

Virtual-Browser
โญ 3,282 Vue ๐Ÿ‘ค Virtual-Browser User ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2023-05-20
85
hydrusnetwork/hydrus

A personal booru-style media tagger that can import files and tags from your hard drive and popular websites. Content can be shared with other users via user-run servers.

hydrusnetwork
โญ 3,276 Python ๐Ÿ‘ค hydrusnetwork User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2013-02-19
86
oxylabs/how-to-scrape-amazon-product-data

The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.

oxylabs
โญ 3,248 Python ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-09-03
87
mitsuhiko/agent-stuff

These are commands I use with agents, mostly Claude

mitsuhiko
โญ 3,223 TypeScript ๐Ÿ‘ค mitsuhiko User ๐Ÿ• Updated 2026-09-27 ๐Ÿ“… Created 2025-11-02
88
scrapy/scrapyd

A service daemon to run Scrapy spiders

scrapy
โญ 3,100 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2013-02-01
โญ 3,074 ๐Ÿ‘ค bpc-clone User ๐Ÿ• Updated 2025-05-18 ๐Ÿ“… Created 2024-04-21
90
langchain-ai/social-media-agent

๐Ÿ“ฒ An agent for sourcing, curating, and scheduling social media posts with human-in-the-loop.

langchain-ai
โญ 3,045 TypeScript ๐Ÿ‘ค langchain-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-11-21
91
TheBlewish/Automated-AI-Web-Researcher-Ollama

A python program that turns an LLM, running on Ollama, into an automated researcher, which will with a single query determine focus areas to investigate, do websearches and scrape content from various relevant websites and do research for you all on its own! And more, not limited to but including saving the findings for you!

TheBlewish
โญ 3,019 Python ๐Ÿ‘ค TheBlewish User ๐Ÿ• Updated 2026-09-02 ๐Ÿ“… Created 2024-11-20
92
x4nth055/pythoncode-tutorials

The Python Code Tutorials

x4nth055
โญ 3,003 Jupyter Notebook ๐Ÿ‘ค x4nth055 User ๐Ÿ• Updated 2026-09-30 ๐Ÿ“… Created 2019-07-29
93
vladkens/twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

vladkens
โญ 2,980 Python ๐Ÿ‘ค vladkens User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-05-05
94
rust-headless-chrome/rust-headless-chrome

A high-level API to control headless Chrome or Chromium over the DevTools Protocol. It is the Rust equivalent of Puppeteer, a Node library maintained by the Chrome DevTools team.

rust-headless-chrome
โญ 2,960 Rust ๐Ÿ‘ค rust-headless-chrome Organization ๐Ÿ• Updated 2026-06-11 ๐Ÿ“… Created 2018-09-18
95
oxylabs/how-to-scrape-google-trends

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.

oxylabs
โญ 2,907 Python ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-08-06
96
digininja/CeWL

CeWL is a Custom Word List Generator

digininja
โญ 2,896 Ruby ๐Ÿ‘ค digininja User ๐Ÿ• Updated 2026-06-24 ๐Ÿ“… Created 2016-04-22
97
spider-rs/spider

Foundational low-latency web data collection in Rust

spider-rs
โญ 2,856 Rust ๐Ÿ‘ค spider-rs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-07
98
Geniusay/ChopperBot

Huya, Douyu, Douyin, BiliBili, TikTok, Twitch๐Ÿ”ฅHot๐Ÿ”ฅSmart Live Video Editing and Publishing AI Robot, Automation๐Ÿค–, Fully Intelligentโš™(Smart Slice Generation, Titles, Covers, Descriptions), Visualization๐Ÿ‘“, Platform Hotness Monitoring๐ŸŒก, Rich Plugins with Free Expansion๐Ÿ•น, Fast Deploymentโšก, Automated Video Account Creation and Publishing๐ŸŒŸ, DIY Support๐ŸŽฎ

Geniusay
โญ 2,758 Java ๐Ÿ‘ค Geniusay User ๐Ÿ• Updated 2025-08-04 ๐Ÿ“… Created 2023-04-27
99
levywang/avhub

R18 Resource Search & Management Tool

levywang
โญ 2,742 Vue ๐Ÿ‘ค levywang User ๐Ÿ• Updated 2026-04-18 ๐Ÿ“… Created 2025-03-07
100
brightdata/brightdata-mcp

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

brightdata
โญ 2,733 JavaScript ๐Ÿ‘ค brightdata Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-04-15