Git Stars Git Stars

#data-collection Repositories

Discover 52 active open source GitHub repositories tagged with #data-collection, accumulating 268,114 total stars (averaging 5,156 stars per project). Leading this topic is NanmiCoder/MediaCrawler with 71,528 stars.

52
Total Repositories
268,114
Total Stars
5,156
Avg Stars/Repo
1
NanmiCoder/MediaCrawler

Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝœComment Crawler

NanmiCoder
โญ 71,528 Python ๐Ÿ‘ค NanmiCoder User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-06-09
2
NaiboWang/EasySpider

A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.

NaiboWang
โญ 44,801 JavaScript ๐Ÿ‘ค NaiboWang User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2020-07-18
3
airbytehq/airbyte

Open-source data movement for ELT pipelines and AI agents โ€” from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

airbytehq
โญ 22,426 Python ๐Ÿ‘ค airbytehq Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-07-27
4
Evil0ctal/Douyin_TikTok_Download_API

๐Ÿš€ Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader โ€” asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.

Evil0ctal
โญ 21,507 Python ๐Ÿ‘ค Evil0ctal User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-11-07
5
cporter202/API-mega-list

This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHubโ€”period. ๐Ÿ’ช

cporter202
โญ 8,406 JavaScript ๐Ÿ‘ค cporter202 User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-12-09
6
firecrawl/firecrawl-mcp-server

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

firecrawl
โญ 7,871 JavaScript ๐Ÿ‘ค firecrawl Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-12-06
7
snowplow/snowplow

The leader in Customer Data Infrastructure

snowplow
โญ 7,052 Scala ๐Ÿ‘ค snowplow Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2012-03-01
8
cloudquery/cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

cloudquery
โญ 6,561 Go ๐Ÿ‘ค cloudquery Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-11-18
9
ResearchKit/ResearchKit

ResearchKit is an open source software framework that makes it easy to create apps for medical research or for other research projects.

ResearchKit
โญ 5,758 Objective-C ๐Ÿ‘ค ResearchKit Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2015-03-10
10
jitsucom/jitsu

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

jitsucom
โญ 5,138 TypeScript ๐Ÿ‘ค jitsucom Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-08-04
11
CollegesChat/university-information

Requirements and details that universities nationwide do not specify during admissions, but which will practically affect the quality of university life

CollegesChat
โญ 5,105 ๐Ÿ‘ค CollegesChat Organization ๐Ÿ• Updated 2026-10-18 ๐Ÿ“… Created 2021-06-25
12
REMitchell/python-scraping

Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do

REMitchell
โญ 4,669 Jupyter Notebook ๐Ÿ‘ค REMitchell User ๐Ÿ• Updated 2024-06-01 ๐Ÿ“… Created 2015-06-14
13
php-debugbar/php-debugbar

Debug bar for PHP

php-debugbar
โญ 4,380 PHP ๐Ÿ‘ค php-debugbar Organization ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2013-06-07
14
cv-cat/DouYin_Spider

TikTok reverse engineering, TikTok crawler, all TikTok APIs, private messages, live room monitoring

cv-cat
โญ 3,711 Python ๐Ÿ‘ค cv-cat User ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2023-10-17
15
Gioman101/FlipperAmiibo

Made to be used with Flipper just drag the folder into NFC

Gioman101
โญ 3,422 Python ๐Ÿ‘ค Gioman101 User ๐Ÿ• Updated 2024-08-15 ๐Ÿ“… Created 2022-04-22
16
scrapy/scrapyd

A service daemon to run Scrapy spiders

scrapy
โญ 3,100 Python ๐Ÿ‘ค scrapy Organization ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2013-02-01
17
vladkens/twscrape

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

vladkens
โญ 2,980 Python ๐Ÿ‘ค vladkens User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-05-05
18
M-cheng-web/web-tracing

Provide monitoring methods for frontend projects: ใ€tracking, behavior, performance, exceptions, requests, resources, routes, impressions, screen recordingใ€‘

M-cheng-web
โญ 2,950 TypeScript ๐Ÿ‘ค M-cheng-web User ๐Ÿ• Updated 2026-09-21 ๐Ÿ“… Created 2021-12-24
19
brightdata/brightdata-mcp

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

brightdata
โญ 2,733 JavaScript ๐Ÿ‘ค brightdata Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-04-15
20
SPX372928/MyIPTV

Standard protocol mobile live source

SPX372928
โญ 2,704 ๐Ÿ‘ค SPX372928 User ๐Ÿ• Updated 2026-02-19 ๐Ÿ“… Created 2020-04-07
โญ 2,096 Python ๐Ÿ‘ค tonyzhaozh User ๐Ÿ• Updated 2024-04-19 ๐Ÿ“… Created 2023-02-22
22
X-zhangyang/Real-World-Masked-Face-Dataset

Real-World Masked Face Dataset, Masked Face Dataset

X-zhangyang
โญ 2,030 Python ๐Ÿ‘ค X-zhangyang User ๐Ÿ• Updated 2024-11-20 ๐Ÿ“… Created 2020-02-13
23
KEV0143/Parser-Chitai-Gorod

Intelligent parser for the "Chitai-Gorod" catalog. High-speed gathering and structuring of data based on any specified parameters: from ISBN to discount dynamics.

KEV0143
โญ 1,979 Python ๐Ÿ‘ค KEV0143 User ๐Ÿ• Updated 2026-09-17 ๐Ÿ“… Created 2025-05-18
24
pyper-dev/pyper

Concurrent Python made simple

pyper-dev
โญ 1,521 Python ๐Ÿ‘ค pyper-dev User ๐Ÿ• Updated 2025-02-04 ๐Ÿ“… Created 2024-05-22
25
sensorsdata/sa-sdk-android

Sensors Data Official Android Tracking SDK is a lightweight SDK for data collection and tracking on the Android side, featuring code tracking, full tracking, click maps, and visual full tracking. Full tracking is implemented by integrating with the official Sensors Data Android Tracking Plugin. The official Sensors Data Android Tracking Plugin uses bytecode instrumentation (ASM) technology to achieve full tracking on the Android side (no-tracking, no-code tracking, traceless tracking, auto-tracking).

sensorsdata
โญ 1,374 Java ๐Ÿ‘ค sensorsdata Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2015-11-06
26
pnoker/iot-dc3

IoT DC3 โ€” Connect the Physical World to AI. An open-source Industrial IoT Runtime for Physical AI: 36 protocol drivers, edge-to-cloud data delivery, and an MCP tool gateway so AI agents act on devices in a safe, controllable, traceable loop.

pnoker
โญ 1,274 Java ๐Ÿ‘ค pnoker User ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2018-08-28
27
Decodo/Decodo

HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.

Decodo
โญ 1,264 Java ๐Ÿ‘ค Decodo Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2019-01-07
28
rohunvora/x-research-skill

X/Twitter research skill for Claude Code and OpenClaw. Agentic search, thread following, deep dives, sourced briefings.

rohunvora
โญ 1,248 TypeScript ๐Ÿ‘ค rohunvora User ๐Ÿ• Updated 2026-02-19 ๐Ÿ“… Created 2026-02-08
29
washingtonpost/data-police-shootings

The Washington Post is compiling a database of every fatal shooting in the United States by a police officer in the line of duty since 2015.

washingtonpost
โญ 1,147 ๐Ÿ‘ค washingtonpost Organization ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2015-06-30
31
plan-player-analytics/Plan

Player Analytics plugin for Minecraft Server platforms - View player activity of your server with ease. :calendar:

plan-player-analytics
โญ 1,041 Java ๐Ÿ‘ค plan-player-analytics Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-12-08
32
estuary/flow

๐ŸŒŠ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐ŸŒŠ

estuary
โญ 995 Rust ๐Ÿ‘ค estuary Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-02-13
33
sensorsdata/sa-sdk-javascript

Shence Data official Web JavaScript tracking SDK is a lightweight data collection tracking SDK for Web and H5 platforms. It uses native JavaScript technology to implement code tracking, full tracking, visual full tracking, web heatmaps, and reach maps.

sensorsdata
โญ 956 JavaScript ๐Ÿ‘ค sensorsdata Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2015-11-06
34
levinunnink/html-form-to-google-sheet

How to submit HTML forms to Google Sheets. (Updated for 2026 Script Editor)

levinunnink
โญ 948 JavaScript ๐Ÿ‘ค levinunnink User ๐Ÿ• Updated 2026-01-14 ๐Ÿ“… Created 2021-09-20
35
TheBurnsy/Vehicle-OSINT-Collection

A comprehensive list of tools that can be used for finding information related to a specific vehicle.

TheBurnsy
โญ 912 ๐Ÿ‘ค TheBurnsy User ๐Ÿ• Updated 2024-11-20 ๐Ÿ“… Created 2022-11-16
36
Cl0udG0d/Fofa-hack

Non-paying member, fofa data collection tool

Cl0udG0d
โญ 882 Python ๐Ÿ‘ค Cl0udG0d User ๐Ÿ• Updated 2025-10-28 ๐Ÿ“… Created 2022-09-15
37
FalconForceTeam/SOAPHound

SOAPHound is a custom-developed .NET data collector tool which can be used to enumerate Active Directory environments via the Active Directory Web Services (ADWS) protocol.

FalconForceTeam
โญ 840 C# ๐Ÿ‘ค FalconForceTeam Organization ๐Ÿ• Updated 2024-02-03 ๐Ÿ“… Created 2024-01-25
38
zqzess/MyWebStorage

Integrates existing Xiangse Guguan sources on the web, the crawler automatically updates book sources weekly

zqzess
โญ 813 HTML ๐Ÿ‘ค zqzess User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2021-05-18
39
getodk/collect

ODK Collect is an Android app for filling out forms. It's been used to collect billions of data points in challenging environments around the world. Contribute and make the world a better place! โœจ๐Ÿ“‹โœจ

getodk
โญ 807 Kotlin ๐Ÿ‘ค getodk Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-08-04
40
quantumproxies/quantumproxies.io

Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries โ€” plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#, Ruby and curl.

quantumproxies
โญ 780 Java ๐Ÿ‘ค quantumproxies User ๐Ÿ• Updated 2026-09-06 ๐Ÿ“… Created 2025-11-17
41
LAION-AI/audio-dataset

Audio Dataset for training CLAP and other models

LAION-AI
โญ 750 Python ๐Ÿ‘ค LAION-AI Organization ๐Ÿ• Updated 2026-01-08 ๐Ÿ“… Created 2022-03-07
42
thinkoaa/Deadpool

The Deadpool proxy pool tool can fetch high-quality socks5 proxies from cyberspace mapping platforms such as Hunter, Quake, Fofa, or import socks5 proxies locally, and use the proxies in a round-robin fashion for traffic forwarding.

thinkoaa
โญ 741 Go ๐Ÿ‘ค thinkoaa User ๐Ÿ• Updated 2026-05-06 ๐Ÿ“… Created 2024-08-26
43
augurlabs/augur

When Augur reached a wall we made Aveloxis: Reliably collecting OSS Metrics Data. 40,000 repositories fully collected in 3 days is good, right?

augurlabs
โญ 695 Go ๐Ÿ‘ค augurlabs Organization ๐Ÿ• Updated 2026-08-04 ๐Ÿ“… Created 2017-01-05
44
bangumi-data/bangumi-data

Raw data for Japanese Anime

bangumi-data
โญ 665 JavaScript ๐Ÿ‘ค bangumi-data Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2016-09-04
45
ohyicong/Google-Image-Scraper

A library to scrap google images

ohyicong
โญ 636 Python ๐Ÿ‘ค ohyicong User ๐Ÿ• Updated 2025-02-16 ๐Ÿ“… Created 2020-07-18
46
nghuyong/weibo-cov

Weibo-COV: A Large-Scale COVID-19 Social Media Dataset from Weibo

nghuyong
โญ 626 ๐Ÿ‘ค nghuyong User ๐Ÿ• Updated 2025-08-23 ๐Ÿ“… Created 2020-02-20
47
Robotics-Ark/ark_framework

The Ark framework.

Robotics-Ark
โญ 625 Python ๐Ÿ‘ค Robotics-Ark Organization ๐Ÿ• Updated 2026-06-20 ๐Ÿ“… Created 2025-06-25
48
john-friedman/datamule-python

A package to work with SEC data. Incorporates datamule endpoints.

john-friedman
โญ 560 Python ๐Ÿ‘ค john-friedman User ๐Ÿ• Updated 2026-08-14 ๐Ÿ“… Created 2024-09-06
49
sindresorhus/insight

Node.js module to help you understand how your tool is being used by anonymously reporting usage metrics to Google Analytics

sindresorhus
โญ 553 JavaScript ๐Ÿ‘ค sindresorhus User ๐Ÿ• Updated 2026-09-18 ๐Ÿ“… Created 2012-12-12
50
zhaoyachao/zdh_web

Big Data Collection, Extraction Platform, zdh_web is the visual management platform for zdh series services, including modules such as data collection, scheduling, permissions, approval workflow, private domain marketing, etc.

zhaoyachao
โญ 536 HTML ๐Ÿ‘ค zhaoyachao User ๐Ÿ• Updated 2026-06-07 ๐Ÿ“… Created 2020-02-03

Frequently Asked Questions

1

What is the most popular #data-collection repository on GitHub?

The top ranked open source #data-collection project on GitHub is NanmiCoder/MediaCrawler with 71,528 stars. Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝœComment Crawler

2

Which programming languages are most used in #data-collection repositories?

Among the top ranked #data-collection projects on GitHub, the most common programming languages are Python, JavaScript, Java, Go.

3

How is the #data-collection repository ranking calculated?

We track 52 active open source repositories tagged with #data-collection (500+ stars), totaling 268,114 stars with an average of 5,156 stars per repository. Rankings update daily from the GitHub API.