Git Stars Git Stars

#web-scraping Repositories

381
Total Repositories
2.1M
Total Stars
5,636
Avg Stars/Repo
201
germondai/trawl

Self-hosted scraping engine โ€” bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

germondai
โญ 1,209 TypeScript ๐Ÿ‘ค germondai User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-06-26
202
intoli/user-agents

A JavaScript library for generating random user agents with data that's updated daily.

intoli
โญ 1,194 TypeScript ๐Ÿ‘ค intoli Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-08-06
203
Evolution0/bandcamp-dl

Simple python script to download Bandcamp albums

Evolution0
โญ 1,189 Python ๐Ÿ‘ค Evolution0 User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2013-04-29
204
goodreasonai/ScrapeServ

A self-hosted API that takes a URL and returns a file with browser screenshots.

goodreasonai
โญ 1,184 Python ๐Ÿ‘ค goodreasonai Organization ๐Ÿ• Updated 2025-03-09 ๐Ÿ“… Created 2025-02-05
205
eshaham/israeli-bank-scrapers

Provide scrapers for all major Israeli banks and credit card companies

eshaham
โญ 1,181 TypeScript ๐Ÿ‘ค eshaham User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2017-07-29
206
dchrastil/ScrapedIn

A tool to scrape LinkedIn without API restrictions for data reconnaissance

dchrastil
โญ 1,168 Python ๐Ÿ‘ค dchrastil User ๐Ÿ• Updated 2024-04-11 ๐Ÿ“… Created 2018-03-29
207
ccd0/4chan-x

Adds various features to anonymous imageboards.

ccd0
โญ 1,163 CoffeeScript ๐Ÿ‘ค ccd0 User ๐Ÿ• Updated 2026-08-27 ๐Ÿ“… Created 2014-04-01
208
yize8888/maotai

iMoutai, HuLuWa reservation script

yize8888
โญ 1,135 JavaScript ๐Ÿ‘ค yize8888 User ๐Ÿ• Updated 2025-03-08 ๐Ÿ“… Created 2023-06-20
209
fwonggh/Bthub

Bthub Latest Address Release Page

fwonggh
โญ 1,134 HTML ๐Ÿ‘ค fwonggh User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-01-11
210
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
211
juancarlospaco/faster-than-requests

Faster requests on Python 3

juancarlospaco
โญ 1,131 Nim ๐Ÿ‘ค juancarlospaco User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-11-13
212
cipher387/python-for-OSINT-21-days

In this repository you will find sample code files for each day of the course "Python for OSINT. A 21-day course for beginners".

cipher387
โญ 1,129 Python ๐Ÿ‘ค cipher387 User ๐Ÿ• Updated 2026-01-06 ๐Ÿ“… Created 2023-03-26
213
Cloxl/xhshow

Xiaohongshu xs pure calculation, reverse engineering of pure calculation logic for fields such as Xiaohongshu x-s, x-s-common, xsc.

Cloxl
โญ 1,124 Python ๐Ÿ‘ค Cloxl User ๐Ÿ• Updated 2026-06-11 ๐Ÿ“… Created 2024-09-11
215
xiaoguyu/wechatDownload

WeChat Official Account Article Download Tool

xiaoguyu
โญ 1,107 TypeScript ๐Ÿ‘ค xiaoguyu User ๐Ÿ• Updated 2025-11-28 ๐Ÿ“… Created 2023-01-03
216
vifreefly/kimuraframework

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.

vifreefly
โญ 1,102 Ruby ๐Ÿ‘ค vifreefly User ๐Ÿ• Updated 2026-01-27 ๐Ÿ“… Created 2018-08-23
217
requireCool/stealth.min.js

Automatically generate the newest stealth.min.js.

requireCool
โญ 1,096 ๐Ÿ‘ค requireCool User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2022-04-10
218
caolvchong-top/twitter_download

Twitter Images Videos Crawler; One-click Download

caolvchong-top
โญ 1,087 Python ๐Ÿ‘ค caolvchong-top User ๐Ÿ• Updated 2026-07-28 ๐Ÿ“… Created 2022-08-23
219
YaoZeyuan/stablog

Stable Tribe. Professional backup and export of Weibo records, stable!

YaoZeyuan
โญ 1,085 JavaScript ๐Ÿ‘ค YaoZeyuan User ๐Ÿ• Updated 2026-08-15 ๐Ÿ“… Created 2019-09-11
220
Anil-matcha/awesome-agent-apis

660+ muapi-hosted generative media models plus community-submitted third-party API tools (SEO, enrichment, social, scraping) โ€” one YAML file per entry, browsable by capability.

Anil-matcha
โญ 1,070 Python ๐Ÿ‘ค Anil-matcha User ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2023-05-06
221
antibrow/antibrow

Kernel-level antidetect browser with the Playwright API you already write. Python + Node SDKs, MCP-ready, unlimited local profiles. Linux x64 + arm64, macOS Intel + Apple Silicon, Windows x64.

antibrow
โญ 1,068 TypeScript ๐Ÿ‘ค antibrow User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2026-07-29
222
IcarusRyy/NewJob

The last modification time of this position can be seen at a glance: green is within 2 weeks, dark orange is within 1.5 months, and red is over 1.5 months

IcarusRyy
โญ 1,058 JavaScript ๐Ÿ‘ค IcarusRyy User ๐Ÿ• Updated 2024-06-10 ๐Ÿ“… Created 2024-05-03
223
stay-leave/weibo-public-opinion-analysis

A Weibo data-based public opinion analysis project, including Weibo crawler, LDA topic analysis, and sentiment analysis.

stay-leave
โญ 1,058 Python ๐Ÿ‘ค stay-leave User ๐Ÿ• Updated 2026-10-24 ๐Ÿ“… Created 2021-08-01
224
lm-rebooter
โญ 1,054 HTML ๐Ÿ‘ค lm-rebooter User ๐Ÿ• Updated 2026-05-06 ๐Ÿ“… Created 2023-05-31
225
mandatoryprogrammer/thermoptic

A next-generation HTTP stealth proxy which perfectly cloaks requests as the Chrome browser across all layers of the stack.

mandatoryprogrammer
โญ 1,053 JavaScript ๐Ÿ‘ค mandatoryprogrammer User ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2025-09-02
โญ 1,041 JavaScript ๐Ÿ‘ค badlogic User ๐Ÿ• Updated 2026-02-17 ๐Ÿ“… Created 2023-05-08
227
FR13ND8/BruteFb

Hack Facebook with the Brute Force Method in Termux

FR13ND8
โญ 1,026 Python ๐Ÿ‘ค FR13ND8 User ๐Ÿ• Updated 2024-07-17 ๐Ÿ“… Created 2019-01-10
228
janreges/siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโ€”ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

janreges
โญ 1,019 Rust ๐Ÿ‘ค janreges User ๐Ÿ• Updated 2026-09-27 ๐Ÿ“… Created 2023-10-04
229
utkarshkukreti/select.rs

A Rust library to extract useful data from HTML documents, suitable for web scraping.

utkarshkukreti
โญ 1,018 Rust ๐Ÿ‘ค utkarshkukreti User ๐Ÿ• Updated 2025-03-19 ๐Ÿ“… Created 2015-07-15
230
thepycoach/automation

Automation with Python (Excel, scripts, etc)

thepycoach
โญ 1,010 Jupyter Notebook ๐Ÿ‘ค thepycoach User ๐Ÿ• Updated 2024-05-19 ๐Ÿ“… Created 2021-05-02
231
fferrin/free-medium

An extension to read Medium posts for free

fferrin
โญ 1,008 JavaScript ๐Ÿ‘ค fferrin User ๐Ÿ• Updated 2025-11-02 ๐Ÿ“… Created 2024-02-13
232
WittmannF/sort-google-scholar

Ranking Google Scholar search results based on the number of citations

WittmannF
โญ 999 Python ๐Ÿ‘ค WittmannF User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2016-04-30
233
maldevel/EmailHarvester

Email addresses harvester

maldevel
โญ 999 Python ๐Ÿ‘ค maldevel User ๐Ÿ• Updated 2026-06-07 ๐Ÿ“… Created 2016-04-16
234
runesleo/x-reader

Universal content reader MCP Server for 10+ platforms

runesleo
โญ 992 Python ๐Ÿ‘ค runesleo User ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2026-02-22
235
borisbabic/browser_cookie3

This is a fork of browser_cookie

borisbabic
โญ 989 Python ๐Ÿ‘ค borisbabic User ๐Ÿ• Updated 2024-12-20 ๐Ÿ“… Created 2015-04-26
236
MinorJerry/WebVoyager

Code for "WebVoyager: WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models"

MinorJerry
โญ 986 Python ๐Ÿ‘ค MinorJerry User ๐Ÿ• Updated 2024-03-04 ๐Ÿ“… Created 2024-01-24
237
YaoZeyuan/zhihuhelp

Rewriting Zhihu Assistant based on Node & TypeScript

YaoZeyuan
โญ 985 TypeScript ๐Ÿ‘ค YaoZeyuan User ๐Ÿ• Updated 2026-10-24 ๐Ÿ“… Created 2019-02-04
238
nicoxiang/geektime-downloader

Geektime Course Downloader, supports downloading Geektime Columns/Video Courses/Daily Lessons/Big Factory Practicals/Bootcamp Videos

nicoxiang
โญ 982 Go ๐Ÿ‘ค nicoxiang User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2022-03-12
239
romansky/dom-to-semantic-markdown

DOM to Semantic-Markdown for use with LLMs

romansky
โญ 976 TypeScript ๐Ÿ‘ค romansky User ๐Ÿ• Updated 2025-05-21 ๐Ÿ“… Created 2024-07-22
โญ 970 Python ๐Ÿ‘ค Saik0s User ๐Ÿ• Updated 2026-02-11 ๐Ÿ“… Created 2025-01-17
241
cutetailbud/ff-ruyijs

This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios.

cutetailbud
โญ 968 C++ ๐Ÿ‘ค cutetailbud User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-10-28
242
treetrum/amazon-kindle-bulk-downloader

Allows you to bulk download all your Kindle eBook in a more automated fashion. This tool allows you to create backup copies of the books you've already purchased.

treetrum
โญ 968 TypeScript ๐Ÿ‘ค treetrum User ๐Ÿ• Updated 2025-02-27 ๐Ÿ“… Created 2023-04-07
โญ 955 Java ๐Ÿ‘ค FongMi User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2022-08-22
244
HtmlUnit/htmlunit

HtmlUnit is a "GUI-Less browser for Java programs".

HtmlUnit
โญ 951 JavaScript ๐Ÿ‘ค HtmlUnit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-09-02
245
JUk1-GH/gpt-promo-scanner

ChatGPT Team (Business) promo code scanner โ€” batch discovery, validation, price collection, supports 34 codes across 17 countries, up to 71% off | ChatGPT Business promo code scanner โ€” batch discovery, validation, price collection, 34 codes across 17 countries, up to 71% off

JUk1-GH
โญ 948 Python ๐Ÿ‘ค JUk1-GH User ๐Ÿ• Updated 2026-05-15 ๐Ÿ“… Created 2026-05-11
246
SickleSec/SnowEyes

A Chrome extension for web sensitive information detection and fingerprinting.

SickleSec
โญ 946 JavaScript ๐Ÿ‘ค SickleSec User ๐Ÿ• Updated 2026-08-10 ๐Ÿ“… Created 2024-12-27
247
BullsEye0/dorks-eye

Dorks Eye Google Hacking Dork Scraping and Searching Script. Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Author: Jolanda de Koff

BullsEye0
โญ 940 Python ๐Ÿ‘ค BullsEye0 User ๐Ÿ• Updated 2026-10-01 ๐Ÿ“… Created 2020-04-30
248
caseychu/spotify-backup

A Python script that exports all of your Spotify playlists.

caseychu
โญ 937 Python ๐Ÿ‘ค caseychu User ๐Ÿ• Updated 2024-09-22 ๐Ÿ“… Created 2014-12-24
249
facundoolano/feedi

A personal feed reader

facundoolano
โญ 933 Python ๐Ÿ‘ค facundoolano User ๐Ÿ• Updated 2026-09-19 ๐Ÿ“… Created 2023-07-15
250
browser-use/awesome-prompts

Table of awesome Browser Use prompts

browser-use
โญ 926 ๐Ÿ‘ค browser-use Organization ๐Ÿ• Updated 2025-03-26 ๐Ÿ“… Created 2025-03-26