Git Stars Git Stars

#scraping Repositories

Discover 109 active open source GitHub repositories tagged with #scraping, accumulating 768,143 total stars (averaging 7,047 stars per project). Leading this topic is firecrawl/firecrawl with 205,758 stars.

109
Total Repositories
768,143
Total Stars
7,047
Avg Stars/Repo
1
firecrawl/firecrawl

Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

firecrawl
โญ 205,758 TypeScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-15
2
D4Vinci/Scrapling

๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shyโ€”join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

D4Vinci
โญ 104,760 Python ๐Ÿ‘ค D4Vinci User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2024-10-13
3
scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

scrapy
โญ 65,519 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2010-02-22
4
soxoj/maigret

๐Ÿ•ต๏ธโ€โ™‚๏ธ Collect a dossier on a person by username from 6K websites

soxoj
โญ 41,111 Python ๐Ÿ‘ค soxoj User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-06-27
5
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

ScrapeGraphAI
โญ 32,877 Python ๐Ÿ‘ค ScrapeGraphAI Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-01-27
6
apify/crawlee

Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify
โญ 26,510 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-26
7
gocolly/colly

Elegant Scraper and Crawler Framework for Golang

gocolly
โญ 25,608 Go ๐Ÿ‘ค gocolly Organization ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2017-09-29
8
psf/requests-html

Pythonic HTML Parsing for Humansโ„ข

psf
โญ 13,874 Python ๐Ÿ‘ค psf Organization ๐Ÿ• Updated 2024-04-16 ๐Ÿ“… Created 2018-02-24
9
daijro/camoufox

๐ŸฆŠ Anti-detect browser

daijro
โญ 13,533 C ๐Ÿ‘ค daijro User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-07-26
10
ultrafunkamsterdam/undetected-chromedriver

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

ultrafunkamsterdam
โญ 12,725 Python ๐Ÿ‘ค ultrafunkamsterdam User ๐Ÿ• Updated 2025-07-05 ๐Ÿ“… Created 2019-12-22
11
jo-inc/camofox-browser

Stealth headless browser for AI agents โ€” bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

jo-inc
โญ 12,437 JavaScript ๐Ÿ‘ค jo-inc Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-01-26
12
code4craft/webmagic

A scalable web crawler framework for Java.

code4craft
โญ 11,660 Java ๐Ÿ‘ค code4craft User ๐Ÿ• Updated 2025-12-20 ๐Ÿ“… Created 2013-04-23
13
apify/crawlee-python

Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify
โญ 9,679 Python ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-01-10
14
lorien/awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

lorien
โญ 8,242 JavaScript ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-08-12
15
alirezamika/autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

alirezamika
โญ 8,102 Python ๐Ÿ‘ค alirezamika User ๐Ÿ• Updated 2026-07-29 ๐Ÿ“… Created 2020-08-31
16
tabulapdf/tabula

Tabula is a tool for liberating data tables trapped inside PDF files

tabulapdf
โญ 7,414 CSS ๐Ÿ‘ค tabulapdf Organization ๐Ÿ• Updated 2025-03-14 ๐Ÿ“… Created 2013-03-11
17
adbar/trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

adbar
โญ 7,157 Python ๐Ÿ‘ค adbar User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-04-08
18
autoscrape-labs/pydoll

Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

autoscrape-labs
โญ 7,145 HTML ๐Ÿ‘ค autoscrape-labs Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2024-10-27
19
proxifly/free-proxy-list

๐Ÿš€ Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)

proxifly
โญ 7,083 ๐Ÿ‘ค proxifly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-07-13
20
jaypyles/Scraperr

Self-hosted webscraper.

jaypyles
โญ 4,876 TypeScript ๐Ÿ‘ค jaypyles User ๐Ÿ• Updated 2025-10-12 ๐Ÿ“… Created 2024-07-06
21
d60/twikit

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

d60
โญ 4,785 Python ๐Ÿ‘ค d60 User ๐Ÿ• Updated 2026-03-10 ๐Ÿ“… Created 2024-01-20
22
bjesus/pipet

Swiss-army tool for scraping and extracting data from online assets, made for hackers

bjesus
โญ 4,610 Go ๐Ÿ‘ค bjesus User ๐Ÿ• Updated 2024-10-12 ๐Ÿ“… Created 2024-08-31
23
sparklemotion/mechanize

Mechanize is a ruby library that makes automated web interaction easy.

sparklemotion
โญ 4,443 Ruby ๐Ÿ‘ค sparklemotion Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2008-04-18
24
CodeCutTech/Data-science

Collection of useful data science topics along with articles, videos, and code

CodeCutTech
โญ 4,158 Jupyter Notebook ๐Ÿ‘ค CodeCutTech Organization ๐Ÿ• Updated 2025-11-16 ๐Ÿ“… Created 2020-07-17
25
kalil0321/reverse-api-engineer

Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor, and OpenCode.

kalil0321
โญ 4,088 Python ๐Ÿ‘ค kalil0321 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-12-23
26
snooppr/snoop

Snoop โ€” An open-source intelligence (OSINT) tool

snooppr
โญ 4,072 Python ๐Ÿ‘ค snooppr User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-02-14
27
fake-useragent/fake-useragent

Up-to-date simple useragent faker with real world database

fake-useragent
โญ 4,047 Python ๐Ÿ‘ค fake-useragent Organization ๐Ÿ• Updated 2026-03-29 ๐Ÿ“… Created 2013-03-04
28
any4ai/AnyCrawl

AnyCrawl ๐Ÿš€: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

any4ai
โญ 3,580 TypeScript ๐Ÿ‘ค any4ai User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2025-03-31
29
adryfish/fingerprint-chromium

An open source fingerprint browser based on Ungoogled Chromium. Fingerprint browser Privacy browser

adryfish
โญ 3,300 ๐Ÿ‘ค adryfish User ๐Ÿ• Updated 2026-10-02 ๐Ÿ“… Created 2024-12-09
30
aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

aapatre
โญ 3,289 Python ๐Ÿ‘ค aapatre User ๐Ÿ• Updated 2026-04-28 ๐Ÿ“… Created 2020-09-12
31
symfony/panther

A browser testing and web crawling library for PHP and Symfony

symfony
โญ 3,074 PHP ๐Ÿ‘ค symfony Organization ๐Ÿ• Updated 2026-10-03 ๐Ÿ“… Created 2018-03-16
32
kevinzg/facebook-scraper

Scrape Facebook public pages without an API key

kevinzg
โญ 3,006 Python ๐Ÿ‘ค kevinzg User ๐Ÿ• Updated 2024-06-22 ๐Ÿ“… Created 2019-03-15
33
vladkens/twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

vladkens
โญ 2,980 Python ๐Ÿ‘ค vladkens User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-05-05
34
oxylabs/how-to-scrape-google-trends

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.

oxylabs
โญ 2,907 Python ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-08-06
35
spider-rs/spider

Foundational low-latency web data collection in Rust

spider-rs
โญ 2,856 Rust ๐Ÿ‘ค spider-rs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-07
36
geziyor/geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

geziyor
โญ 2,781 Go ๐Ÿ‘ค geziyor Organization ๐Ÿ• Updated 2026-07-02 ๐Ÿ“… Created 2019-06-06
37
brightdata/brightdata-mcp

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

brightdata
โญ 2,733 JavaScript ๐Ÿ‘ค brightdata Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-04-15
38
apify/fingerprint-suite

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

apify
โญ 2,678 TypeScript ๐Ÿ‘ค apify Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-05-04
39
botswin/BotBrowser

Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore

botswin
โญ 2,667 TypeScript ๐Ÿ‘ค botswin Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-09-08
40
oss-invisible/playwright

invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

oss-invisible
โญ 2,612 Python ๐Ÿ‘ค oss-invisible Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-05-13
41
simonw/shot-scraper

A CLI utility for taking screenshots of websites, recording video demos and scraping sites using JavaScript

simonw
โญ 2,609 Python ๐Ÿ‘ค simonw User ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2022-03-08
42
transitive-bullshit/awesome-puppeteer

A curated list of awesome puppeteer resources.

transitive-bullshit
โญ 2,528 ๐Ÿ‘ค transitive-bullshit User ๐Ÿ• Updated 2024-07-19 ๐Ÿ“… Created 2018-05-07
43
lorien/grab

Web Scraping Framework

lorien
โญ 2,461 Python ๐Ÿ‘ค lorien User ๐Ÿ• Updated 2025-09-19 ๐Ÿ“… Created 2013-05-01
44
Jieyab89/OSINT-Cheat-sheet

OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will grow every time will research, there is a research, science and technology, tutorial. Please use it wisely.

Jieyab89
โญ 2,333 HTML ๐Ÿ‘ค Jieyab89 User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2021-11-22
45
php-embed/Embed

Get info from any web service or page

php-embed
โญ 2,140 PHP ๐Ÿ‘ค php-embed Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2012-10-09
46
TheGP/untidetect-tools

List of anti-detect and humanizing tools and browsers, including captcha solvers and sms-activation.

TheGP
โญ 2,075 ๐Ÿ‘ค TheGP User ๐Ÿ• Updated 2026-10-07 ๐Ÿ“… Created 2024-01-21
47
iiab/iiab

Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !

iiab
โญ 2,056 Jinja ๐Ÿ‘ค iiab Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2017-05-28
48
claffin/cloudproxy

Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.

claffin
โญ 1,729 Python ๐Ÿ‘ค claffin User ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2021-04-16
49
oxylabs/scrape-google-python

In this tutorial, we showcase how to scrape public Google data with Python and Oxylabs API.

oxylabs
โญ 1,719 ๐Ÿ‘ค oxylabs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-07-30
50
emcf/thepipe

Get clean data from tricky documents, powered by vision-language models โšก

emcf
โญ 1,529 Python ๐Ÿ‘ค emcf User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2024-03-22

Frequently Asked Questions

1

What is the most popular #scraping repository on GitHub?

The top ranked open source #scraping project on GitHub is firecrawl/firecrawl with 205,758 stars. Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

2

Which programming languages are most used in #scraping repositories?

Among the top ranked #scraping projects on GitHub, the most common programming languages are Python, TypeScript, Go, JavaScript.

3

How is the #scraping repository ranking calculated?

We track 109 active open source repositories tagged with #scraping (500+ stars), totaling 768,143 stars with an average of 7,047 stars per repository. Rankings update daily from the GitHub API.