Git Stars Git Stars

#crawler Repositories

162
Total Repositories
1.1M
Total Stars
6,863
Avg Stars/Repo
101
Zeal-L/BiliBili-Manga-Downloader

A user-friendly Bilibili comic downloader with a graphical interface, supporting keyword search for comics and QR code login, black technology for downloading unlocked chapters, multi-threaded downloads, various saving formats, local comic management, and one-click update checking!

Zeal-L
โญ 1,244 Python ๐Ÿ‘ค Zeal-L User ๐Ÿ• Updated 2025-07-23 ๐Ÿ“… Created 2023-01-15
102
InstaPy/instagram-profilecrawl

๐Ÿ“ quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

InstaPy
โญ 1,244 Python ๐Ÿ‘ค InstaPy Organization ๐Ÿ• Updated 2024-03-19 ๐Ÿ“… Created 2017-02-15
103
seveniruby/AppCrawler

App auto-traversal tool based on Appium

seveniruby
โญ 1,244 Scala ๐Ÿ‘ค seveniruby User ๐Ÿ• Updated 2026-04-20 ๐Ÿ“… Created 2016-02-16
104
webrecorder/browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

webrecorder
โญ 1,199 TypeScript ๐Ÿ‘ค webrecorder Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2020-11-02
105
AndyTheFactory/newspaper4k

๐Ÿ“ฐ Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

AndyTheFactory
โญ 1,170 Python ๐Ÿ‘ค AndyTheFactory User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2023-10-22
106
dixudx/tumblr-crawler

Easily download all the photos/videos from tumblr blogs. Download pictures and videos from specified Tumblr blogs

dixudx
โญ 1,160 Python ๐Ÿ‘ค dixudx User ๐Ÿ• Updated 2025-12-26 ๐Ÿ“… Created 2016-04-15
107
codelibs/fess

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

codelibs
โญ 1,152 Java ๐Ÿ‘ค codelibs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2013-09-13
108
scrapfly/scrapfly-scrapers

Scalable Python web scraping scripts for 80+ popular domains

scrapfly
โญ 1,133 Python ๐Ÿ‘ค scrapfly Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-22
109
BaiduSpider/BaiduSpider

BaiduSpider, a crawler that crawls Baidu search results, currently supports Baidu web search, Baidu image search, Baidu Knows search, Baidu video search, Baidu news search, Baidu Wenku search, Baidu experience search, and Baidu Baike search.

BaiduSpider
โญ 1,130 Python ๐Ÿ‘ค BaiduSpider Organization ๐Ÿ• Updated 2024-06-14 ๐Ÿ“… Created 2020-10-03
110
elixir-crawly/crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

elixir-crawly
โญ 1,129 Elixir ๐Ÿ‘ค elixir-crawly Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-09
111
0x676e67/wreq

An ergonomic, privacy-aware Rust HTTP Client

0x676e67
โญ 1,118 Rust ๐Ÿ‘ค 0x676e67 User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-09-19
112
wujunwei928/parse-video

Golang Short Video Watermark Removal: Douyin, Pipixia, Huoshan, Weishi, Zuiyou, Kuaishou, Quanmin Xiaoshipin, Pipixia Gaoqiao, Xigua Video, Huya, Li Video, AcFun, Haokan Video...

wujunwei928
โญ 1,104 Go ๐Ÿ‘ค wujunwei928 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2022-07-12
113
vifreefly/kimuraframework

Write web scrapers in Ruby using a clean, AI-assisted DSL. Kimurai uses AI to figure out where the data lives, then caches the selectors and scrapes with pure Ruby. Get the intelligence of an LLM without the per-request latency or token costs.

vifreefly
โญ 1,102 Ruby ๐Ÿ‘ค vifreefly User ๐Ÿ• Updated 2026-01-27 ๐Ÿ“… Created 2018-08-23
114
antibrow/antibrow

Kernel-level antidetect browser with the Playwright API you already write. Python + Node SDKs, MCP-ready, unlimited local profiles. Linux x64 + arm64, macOS Intel + Apple Silicon, Windows x64.

antibrow
โญ 1,068 TypeScript ๐Ÿ‘ค antibrow User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2026-07-29
115
darbra/awesome-ai-reverse

AI reverse engineering, all in one go

darbra
โญ 1,026 ๐Ÿ‘ค darbra User ๐Ÿ• Updated 2026-09-29 ๐Ÿ“… Created 2026-04-14
116
janreges/siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโ€”ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

janreges
โญ 1,019 Rust ๐Ÿ‘ค janreges User ๐Ÿ• Updated 2026-09-27 ๐Ÿ“… Created 2023-10-04
117
apache/stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

apache
โญ 1,014 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2013-04-12
118
pea3nut/Pxer

A tool for pixiv.net. A P Station crawler available to everyone

pea3nut
โญ 1,010 JavaScript ๐Ÿ‘ค pea3nut User ๐Ÿ• Updated 2025-03-03 ๐Ÿ“… Created 2015-12-16
119
spidersuite/SpiderSuite

SpiderSuite (web security crawler) releases, wiki and roadmap

spidersuite
โญ 990 ๐Ÿ‘ค spidersuite Organization ๐Ÿ• Updated 2026-08-13 ๐Ÿ“… Created 2023-04-05
120
fredwu/crawler

A high performance web crawler / scraper in Elixir.

fredwu
โญ 963 Elixir ๐Ÿ‘ค fredwu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-08-08
121
ma6254/FictionDown

Novel Download|Novel Scraping|่ตท็‚น|็ฌ”่ถฃ้˜|Export Markdown|Export txt|Convert epub|Ad Filtering|Auto Proofreading

ma6254
โญ 955 Go ๐Ÿ‘ค ma6254 User ๐Ÿ• Updated 2025-01-21 ๐Ÿ“… Created 2019-03-03
122
xiaoxiunique/x-kit

Pure-Python protocol tools and research notes for X.com, with VibeLoft Twitter account datasets

xiaoxiunique
โญ 947 Python ๐Ÿ‘ค xiaoxiunique User ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2024-12-16
123
0xAllenChen/spider_reverse

Crawler reverse engineering cases, completed: TLS fingerprint | Ruishu | Zhenkunxing | NetEase Yudun | WeChat Mini Program Decompilation Reverse Engineering (Baida Xingxi) | Tonghuashun | rpc decryption | Jiasule | Jueyan slider CAPTCHA | Julang Sanshu | Boss Zhipin | Qichacha | China Minmetals | QQ Music | Industrial Policy Big Data Platform | Qizhidiao | Xueqiu.com (acw_sc__v2) | 1688 | Qimai Data | whggzy | Qiming Keji | mohurd | Ejoin Data | Ouke Yunlian

0xAllenChen
โญ 946 Python ๐Ÿ‘ค 0xAllenChen User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-06-02
124
TikHub/TikHub-API-Python-SDK

High-performance asynchronous Captcha Solver(verification code resolver) for Douyin(ๆŠ–้Ÿณ), TikTok, Xiaohongshu(ๅฐ็บขไนฆ), Kuaishou(ๅฟซๆ‰‹), Weibo(ๅพฎๅš), Instagram, YouTube(ๆฒน็ฎก), Twitter(X) with Temp Mail(temporary email) API(interface).

TikHub
โญ 943 Python ๐Ÿ‘ค TikHub Organization ๐Ÿ• Updated 2026-10-03 ๐Ÿ“… Created 2022-12-17
125
hellock/icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

hellock
โญ 936 Python ๐Ÿ‘ค hellock User ๐Ÿ• Updated 2025-12-10 ๐Ÿ“… Created 2016-03-19
126
JoMingyu/google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

JoMingyu
โญ 930 Python ๐Ÿ‘ค JoMingyu User ๐Ÿ• Updated 2024-08-05 ๐Ÿ“… Created 2019-06-04
127
SkywalkerDarren/chatWeb

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

SkywalkerDarren
โญ 916 Python ๐Ÿ‘ค SkywalkerDarren User ๐Ÿ• Updated 2026-05-25 ๐Ÿ“… Created 2023-03-09
128
scrapinghub/scrapyrt

HTTP API for Scrapy spiders

scrapinghub
โญ 884 Python ๐Ÿ‘ค scrapinghub Organization ๐Ÿ• Updated 2026-10-13 ๐Ÿ“… Created 2015-01-06
129
rndinfosecguy/Scavenger

Crawler (Bot) searching for leaks on paste sites.

rndinfosecguy
โญ 883 Python ๐Ÿ‘ค rndinfosecguy User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-03-10
130
skrapeit/skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

skrapeit
โญ 874 Kotlin ๐Ÿ‘ค skrapeit User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2019-01-14
131
StJudeWasHere/seonaut

Open source SEO audit tool.

StJudeWasHere
โญ 873 Go ๐Ÿ‘ค StJudeWasHere User ๐Ÿ• Updated 2026-10-08 ๐Ÿ“… Created 2022-03-02
132
setvisible/ArrowDL

ArrowDL (Arrow Downloader) is a download manager for Windows, MacOS and Linux

setvisible
โญ 855 C++ ๐Ÿ‘ค setvisible User ๐Ÿ• Updated 2026-05-04 ๐Ÿ“… Created 2019-07-04
133
BlessedRebuS/Krawl

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

BlessedRebuS
โญ 844 Python ๐Ÿ‘ค BlessedRebuS User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-12-10
134
postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

postmodern
โญ 835 Ruby ๐Ÿ‘ค postmodern User ๐Ÿ• Updated 2026-01-12 ๐Ÿ“… Created 2009-03-08
135
tiliondev/fortress

Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

tiliondev
โญ 817 Python ๐Ÿ‘ค tiliondev Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-07-02
136
MorvanZhou/easy-scraping-tutorial

Simple but useful Python web scraping tutorial code.

MorvanZhou
โญ 814 Jupyter Notebook ๐Ÿ‘ค MorvanZhou User ๐Ÿ• Updated 2024-04-07 ๐Ÿ“… Created 2017-12-29
137
fffonion/xeHentai

Doujinshi downloader Erotic manga download

fffonion
โญ 808 Python ๐Ÿ‘ค fffonion User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2013-03-23
138
fanyong920/jvppeteer

Java API For Chrome and Firefox

fanyong920
โญ 805 Java ๐Ÿ‘ค fanyong920 User ๐Ÿ• Updated 2026-10-04 ๐Ÿ“… Created 2020-03-11
139
chadi0x/TheBigBrother

The Big Brother is a weaponized OSINT platform featuring username enumeration (473+ platforms), quad-vector visual intelligence, Sky Radar tracking, crypto wallet analysis, SSL intelligence, digital footprint reconstruction, EXIF extraction, advanced dorking, and network reconnaissance.

chadi0x
โญ 787 Python ๐Ÿ‘ค chadi0x User ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2025-12-08
140
JSREI/js-cookie-monitor-debugger-hook

JS Cookie Reverse Engineering Power Tool: JS Cookie Change Monitoring Visualization Tool & JS Cookie Hook for Conditional Breakpoints

JSREI
โญ 779 TypeScript ๐Ÿ‘ค JSREI Organization ๐Ÿ• Updated 2026-01-28 ๐Ÿ“… Created 2022-07-22
141
lixi5338619/lxBook

Code repository for the book "Advanced Practice of Web Scraping Reverse Engineering"

lixi5338619
โญ 770 JavaScript ๐Ÿ‘ค lixi5338619 User ๐Ÿ• Updated 2026-01-26 ๐Ÿ“… Created 2021-11-16
142
xuxueli/xxl-crawler

A lightweight web crawler framework. (Java crawler framework)

xuxueli
โญ 756 Java ๐Ÿ‘ค xuxueli User ๐Ÿ• Updated 2026-08-08 ๐Ÿ“… Created 2017-10-17
143
polyrabbit/hacker-news-digest

๐Ÿ“ฐ Let LLM Summarize Hacker News for You

polyrabbit
โญ 754 Python ๐Ÿ‘ค polyrabbit User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-09-17
144
TumblThreeApp/TumblThree

A Tumblr and Twitter Blog Backup Application

TumblThreeApp
โญ 744 C# ๐Ÿ‘ค TumblThreeApp Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2018-12-29
145
PyPtt/PyPtt

The best PTT library

PyPtt
โญ 733 Python ๐Ÿ‘ค PyPtt Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2017-05-28
146
josephlimtech/linkedin-profile-scraper-api

๐Ÿ•ต๏ธโ€โ™‚๏ธ LinkedIn profile scraper returning structured profile data in JSON.

josephlimtech
โญ 725 TypeScript ๐Ÿ‘ค josephlimtech User ๐Ÿ• Updated 2024-04-05 ๐Ÿ“… Created 2018-12-21
147
chushuai/wscan

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

chushuai
โญ 717 Go ๐Ÿ‘ค chushuai User ๐Ÿ• Updated 2026-09-21 ๐Ÿ“… Created 2022-05-06
148
tasos-py/Search-Engines-Scraper

Search google, bing, yahoo, and other search engines with python

tasos-py
โญ 672 Python ๐Ÿ‘ค tasos-py User ๐Ÿ• Updated 2025-04-02 ๐Ÿ“… Created 2018-01-08
149
C0D3D3V/Moodle-DL

Moodle-DL downloads course content fast from Moodle (eg. lecture pdfs)

C0D3D3V
โญ 668 Python ๐Ÿ‘ค C0D3D3V User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2019-12-19
150
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17