Git Stars Git Stars

#scraping Repositories

109
Total Repositories
768,143
Total Stars
7,047
Avg Stars/Repo
51
tinyfish-io/agentql

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

tinyfish-io
โญ 1,510 Python ๐Ÿ‘ค tinyfish-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-02-20
52
rebrowser/rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

rebrowser
โญ 1,468 JavaScript ๐Ÿ‘ค rebrowser Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2024-08-06
53
ScrapingBee/chatgpt-scraper-api

Collect structured responses from a ChatGPT scraper by sending a prompt with valid ChatGPT scraping API credentials. Enable live search, inject HTML context, and automate intelligent scraper ChatGPT workflows in just a few parameters.

ScrapingBee
โญ 1,465 ๐Ÿ‘ค ScrapingBee Organization ๐Ÿ• Updated 2026-07-15 ๐Ÿ“… Created 2026-03-12
54
okfn-brasil/querido-diario

๐Ÿ“ฐ Brazilian official gazettes accessible to all | ๐Ÿ“ฐ Brazilian government gazettes, accessible to everyone.

okfn-brasil
โญ 1,424 Python ๐Ÿ‘ค okfn-brasil Organization ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2018-04-01
55
feder-cr/invisible_puppeteer

Puppeteer for Python that anti-bot systems canโ€™t detect, so there are no CAPTCHAs: the same API, anti-detection stealth headless Firefox, an undetected fingerprint, bypassing bot detection, web scraping, and browser automation.

feder-cr
โญ 1,403 Python ๐Ÿ‘ค feder-cr User ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2024-09-01
56
raznem/parsera

Lightweight library for scraping web-sites with LLMs

raznem
โญ 1,372 Python ๐Ÿ‘ค raznem User ๐Ÿ• Updated 2025-12-17 ๐Ÿ“… Created 2024-08-12
57
lorey/mlscraper

๐Ÿค– Scrape data from HTML websites automatically by just providing examples

lorey
โญ 1,370 Python ๐Ÿ‘ค lorey User ๐Ÿ• Updated 2024-03-17 ๐Ÿ“… Created 2020-07-30
58
scrapy/parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

scrapy
โญ 1,368 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2015-04-24
59
mov-cli/mov-cli

Watch everything from your terminal.

mov-cli
โญ 1,313 Python ๐Ÿ‘ค mov-cli Organization ๐Ÿ• Updated 2025-10-05 ๐Ÿ“… Created 2022-03-11
60
daijro/browserforge

๐ŸŽญ Intelligent browser header & fingerprint generator

daijro
โญ 1,298 Python ๐Ÿ‘ค daijro User ๐Ÿ• Updated 2026-10-29 ๐Ÿ“… Created 2024-03-13
61
plabayo/rama

modular service framework to move and transform network packets

plabayo
โญ 1,273 Rust ๐Ÿ‘ค plabayo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-08-07
62
Decodo/Decodo

HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.

Decodo
โญ 1,264 Java ๐Ÿ‘ค Decodo Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2019-01-07
63
neon-mmd/websurfx

:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine

neon-mmd
โญ 1,231 Rust ๐Ÿ‘ค neon-mmd User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2023-04-28
64
datawhores/OF-Scraper

A completely revamped and redesigned fork with OnlyFans and Fansly support, reimagined from scratch based on the original onlyfans-scraper.

datawhores
โญ 1,219 Python ๐Ÿ‘ค datawhores User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-03-22
65
germondai/trawl

Self-hosted scraping engine โ€” bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

germondai
โญ 1,209 TypeScript ๐Ÿ‘ค germondai User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-06-26
66
AndyTheFactory/newspaper4k

๐Ÿ“ฐ Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

AndyTheFactory
โญ 1,170 Python ๐Ÿ‘ค AndyTheFactory User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2023-10-22
67
bellingcat/auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

bellingcat
โญ 1,158 Python ๐Ÿ‘ค bellingcat Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2021-01-15
68
soxoj/socid-extractor

โ›๏ธ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites

soxoj
โญ 1,143 Python ๐Ÿ‘ค soxoj User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-11-17
69
ai-to-ai/Auto-Gmail-Creator

Open Source Bulk Auto Gmail Creator Bot with Selenium & Seleniumwire ( Python ). Feel free to contact me with Django/Flask, ML, AI, GPT, Automation, Scraping.

ai-to-ai
โญ 1,138 Python ๐Ÿ‘ค ai-to-ai User ๐Ÿ• Updated 2024-09-26 ๐Ÿ“… Created 2023-02-20
70
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
71
elixir-crawly/crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

elixir-crawly
โญ 1,129 Elixir ๐Ÿ‘ค elixir-crawly Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-09
72
online-judge-tools/oj

Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.

online-judge-tools
โญ 1,128 Python ๐Ÿ‘ค online-judge-tools Organization ๐Ÿ• Updated 2024-05-03 ๐Ÿ“… Created 2017-01-05
73
viu-media/viu

Your browser anime experience from the terminal

viu-media
โญ 1,062 Python ๐Ÿ‘ค viu-media Organization ๐Ÿ• Updated 2026-05-06 ๐Ÿ“… Created 2024-05-27
74
jfilter/clean-text

๐Ÿงน Python package for text cleaning

jfilter
โญ 1,031 Python ๐Ÿ‘ค jfilter User ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2018-12-06
75
daijro/hrequests

Web scraping for humans

daijro
โญ 976 Python ๐Ÿ‘ค daijro User ๐Ÿ• Updated 2024-12-01 ๐Ÿ“… Created 2023-07-06
76
Krasjet/pdf.tocgen

A CLI toolset to generate table of contents for PDF files automatically.

Krasjet
โญ 853 Python ๐Ÿ‘ค Krasjet User ๐Ÿ• Updated 2026-05-13 ๐Ÿ“… Created 2020-07-28
77
leoncvlt/loconotion

๐Ÿ“„ Python tool to turn Notion.so pages into lightweight, customizable static websites

leoncvlt
โญ 853 Python ๐Ÿ‘ค leoncvlt User ๐Ÿ• Updated 2024-07-22 ๐Ÿ“… Created 2020-05-16
78
tiliondev/fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

tiliondev
โญ 817 Python ๐Ÿ‘ค tiliondev Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-07-02
79
MorvanZhou/easy-scraping-tutorial

Simple but useful Python web scraping tutorial code.

MorvanZhou
โญ 814 Jupyter Notebook ๐Ÿ‘ค MorvanZhou User ๐Ÿ• Updated 2024-04-07 ๐Ÿ“… Created 2017-12-29
80
Lookyloo/lookyloo

Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.

Lookyloo
โญ 790 Python ๐Ÿ‘ค Lookyloo Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2017-07-23
81
alpkeskin/rota

A high-performance proxy rotation engine with automated IP management and real-time health monitoring

alpkeskin
โญ 775 Go ๐Ÿ‘ค alpkeskin User ๐Ÿ• Updated 2026-09-25 ๐Ÿ“… Created 2024-06-02
82
ZacharyHampton/HomeHarvest

Python package for scraping real estate property data

ZacharyHampton
โญ 764 Python ๐Ÿ‘ค ZacharyHampton User ๐Ÿ• Updated 2025-12-26 ๐Ÿ“… Created 2023-09-15
83
serpapi/google-search-results-python

Google Search Results via SERP API pip Python Package

serpapi
โญ 762 Python ๐Ÿ‘ค serpapi Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2018-01-10
84
1broseidon/ketch

Fast, stateless CLI for web search and scrape. Built for AI agents.

1broseidon
โญ 756 Go ๐Ÿ‘ค 1broseidon User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-03-25
85
josephlimtech/linkedin-profile-scraper-api

๐Ÿ•ต๏ธโ€โ™‚๏ธ LinkedIn profile scraper returning structured profile data in JSON.

josephlimtech
โญ 725 TypeScript ๐Ÿ‘ค josephlimtech User ๐Ÿ• Updated 2024-04-05 ๐Ÿ“… Created 2018-12-21
86
prescience-data/dark-knowledge

๐Ÿ˜ˆ๐Ÿ“š A curated library of research papers and presentations for counter-detection and web privacy enthusiasts.

prescience-data
โญ 715 JavaScript ๐Ÿ‘ค prescience-data User ๐Ÿ• Updated 2024-02-19 ๐Ÿ“… Created 2021-04-15
87
Yuvi9587/Kemono-Downloader

Kemono pawchive Downloader is a fast, PyQt5 app for archiving content from a wide array of sites, including Kemono, Coomer, Bunkr, Erome, Saint2.su, nhentai, and Discord. It features a Creator Browser, update checker, and supports multi-threaded, multi-part downloads. Sessions can pause, resume, or recover filters.

Yuvi9587
โญ 675 Python ๐Ÿ‘ค Yuvi9587 User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2025-05-05
88
ScriptSmith/socialreaper

Social media scraping / data collection library for Facebook, Twitter, Reddit, YouTube, Pinterest, and Tumblr APIs

ScriptSmith
โญ 668 Python ๐Ÿ‘ค ScriptSmith User ๐Ÿ• Updated 2026-09-29 ๐Ÿ“… Created 2017-01-31
89
Xonshiz/comic-dl

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.

Xonshiz
โญ 667 Python ๐Ÿ‘ค Xonshiz User ๐Ÿ• Updated 2025-07-31 ๐Ÿ“… Created 2016-11-13
90
freelawproject/juriscraper

An API to scrape American court websites for metadata.

freelawproject
โญ 664 HTML ๐Ÿ‘ค freelawproject Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-08-08
91
AnonCatalyst/Ominis-OSINT

This Python application is an OSINT (Open Source Intelligence) tool called "Ominis OSINT - Web Hunter." It performs online information gathering by querying Google for search results related to a user-inputted query. The tool extracts relevant information such as titles, URLs, and potential mentions of the query in the results.

AnonCatalyst
โญ 655 Python ๐Ÿ‘ค AnonCatalyst User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-10-09
92
gunkonchi/xscrape

Async Python client for collecting public data from X (Twitter) โ€” search, profiles, threads, replies, and media. Built-in account rotation, adaptive rate limiting, and export to JSON/CSV/SQLite.

gunkonchi
โญ 647 Python ๐Ÿ‘ค gunkonchi User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2026-09-26
93
lorey/social-media-profiles-regexs

:card_index: Extract social media profiles and more with regular expressions

lorey
โญ 642 Python ๐Ÿ‘ค lorey User ๐Ÿ• Updated 2024-01-11 ๐Ÿ“… Created 2015-06-24
94
scrapfly/Antibot-Detector

Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

scrapfly
โญ 636 JavaScript ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2025-09-03
95
l4rm4nd/LinkedInDumper

Python 3 script to dump/scrape/extract company employees from LinkedIn API

l4rm4nd
โญ 624 Python ๐Ÿ‘ค l4rm4nd User ๐Ÿ• Updated 2026-08-21 ๐Ÿ“… Created 2022-10-17
96
ksanjeev284/reddit-universal-scraper

Universal Reddit Scraper - Works on any Subreddit or User

ksanjeev284
โญ 620 Python ๐Ÿ‘ค ksanjeev284 User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2025-12-13
97
umihico/docker-selenium-lambda

The simplest demo of chrome automation by python and selenium in AWS Lambda

umihico
โญ 614 Dockerfile ๐Ÿ‘ค umihico User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2021-03-12
98
kr4ude/onlyfans-scraper

A tool that allows you to scrape media from any Onlyfans account and more

kr4ude
โญ 605 Python ๐Ÿ‘ค kr4ude User ๐Ÿ• Updated 2025-11-25 ๐Ÿ“… Created 2025-11-10
99
drudge/n8n-nodes-puppeteer

n8n node for browser automation using Puppeteer

drudge
โญ 589 TypeScript ๐Ÿ‘ค drudge User ๐Ÿ• Updated 2026-05-08 ๐Ÿ“… Created 2022-05-07
100
spekulatius/PHPScraper

A universal web-util for PHP.

spekulatius
โญ 588 PHP ๐Ÿ‘ค spekulatius User ๐Ÿ• Updated 2026-09-08 ๐Ÿ“… Created 2020-04-21