#data-collection Repositories
Discover 52 active open source GitHub repositories tagged with #data-collection, accumulating 268,114 total stars (averaging 5,156 stars per project). Leading this topic is NanmiCoder/MediaCrawler with 71,528 stars.
52 repositories total
Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝComment Crawler
A visual no-code web crawler/spider: A browser automation, data collection, and web scraping software that allows for the design and execution of crawling tasks graphically without writing code. Alias: ServiceWrapper, an intelligent service encapsulation system for web applications.
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
๐ Self-hosted data collection and watermark-free video download API for Douyin, TikTok, and Bilibili. Supports AI agent integration via MCP and offline speech-to-text. One-click Docker deployment. | Self-hosted TikTok, Douyin & Bilibili scraper and watermark-free video downloader โ asynchronous REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one `docker compose up`.
This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHubโperiod. ๐ช
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
ResearchKit is an open source software framework that makes it easy to create apps for medical research or for other research projects.
Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
Requirements and details that universities nationwide do not specify during admissions, but which will practically affect the quality of university life
Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do
TikTok reverse engineering, TikTok crawler, all TikTok APIs, private messages, live room monitoring
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
Provide monitoring methods for frontend projects: ใtracking, behavior, performance, exceptions, requests, resources, routes, impressions, screen recordingใ
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Intelligent parser for the "Chitai-Gorod" catalog. High-speed gathering and structuring of data based on any specified parameters: from ISBN to discount dynamics.
Sensors Data Official Android Tracking SDK is a lightweight SDK for data collection and tracking on the Android side, featuring code tracking, full tracking, click maps, and visual full tracking. Full tracking is implemented by integrating with the official Sensors Data Android Tracking Plugin. The official Sensors Data Android Tracking Plugin uses bytecode instrumentation (ASM) technology to achieve full tracking on the Android side (no-tracking, no-code tracking, traceless tracking, auto-tracking).
IoT DC3 โ Connect the Physical World to AI. An open-source Industrial IoT Runtime for Physical AI: 36 protocol drivers, edge-to-cloud data delivery, and an MCP tool gateway so AI agents act on devices in a safe, controllable, traceable loop.
X/Twitter research skill for Claude Code and OpenClaw. Agentic search, thread following, deep dives, sourced briefings.
The Washington Post is compiling a database of every fatal shooting in the United States by a police officer in the line of duty since 2015.
Player Analytics plugin for Minecraft Server platforms - View player activity of your server with ease. :calendar:
๐ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐
Shence Data official Web JavaScript tracking SDK is a lightweight data collection tracking SDK for Web and H5 platforms. It uses native JavaScript technology to implement code tracking, full tracking, visual full tracking, web heatmaps, and reach maps.
How to submit HTML forms to Google Sheets. (Updated for 2026 Script Editor)
A comprehensive list of tools that can be used for finding information related to a specific vehicle.
SOAPHound is a custom-developed .NET data collector tool which can be used to enumerate Active Directory environments via the Active Directory Web Services (ADWS) protocol.
Integrates existing Xiangse Guguan sources on the web, the crawler automatically updates book sources weekly
ODK Collect is an Android app for filling out forms. It's been used to collect billions of data points in challenging environments around the world. Contribute and make the world a better place! โจ๐โจ
Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries โ plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#, Ruby and curl.
The Deadpool proxy pool tool can fetch high-quality socks5 proxies from cyberspace mapping platforms such as Hunter, Quake, Fofa, or import socks5 proxies locally, and use the proxies in a round-robin fashion for traffic forwarding.
When Augur reached a wall we made Aveloxis: Reliably collecting OSS Metrics Data. 40,000 repositories fully collected in 3 days is good, right?
Node.js module to help you understand how your tool is being used by anonymously reporting usage metrics to Google Analytics
Big Data Collection, Extraction Platform, zdh_web is the visual management platform for zdh series services, including modules such as data collection, scheduling, permissions, approval workflow, private domain marketing, etc.
Frequently Asked Questions
What is the most popular #data-collection repository on GitHub?
The top ranked open source #data-collection project on GitHub is NanmiCoder/MediaCrawler with 71,528 stars. Xiaohongshu Note | Comment Crawler, Douyin Video | Comment Crawler, Kuaishou Video | Comment Crawler, Bilibili Video | Comment Crawler, Weibo Post | Comment Crawler, Baidu Tieba Post | Baidu Tieba Comment Reply Crawler | Zhihu Q&A Article๏ฝComment Crawler
Which programming languages are most used in #data-collection repositories?
Among the top ranked #data-collection projects on GitHub, the most common programming languages are Python, JavaScript, Java, Go.
How is the #data-collection repository ranking calculated?
We track 52 active open source repositories tagged with #data-collection (500+ stars), totaling 268,114 stars with an average of 5,156 stars per repository. Rankings update daily from the GitHub API.