#big-data Repositories
Discover 133 active open source GitHub repositories tagged with #big-data, accumulating 683,600 total stars (averaging 5,139 stars per project). Leading this topic is binhnguyennus/awesome-scalability with 75,924 stars.
133 repositories total
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
All-in-one IoT Platform - Device management, data collection, processing and visualization.
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
A distributed, fast open-source graph database featuring horizontal scalability and high availability
The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Arkime is an open source, large scale, full packet capturing, indexing, and database system.
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
โก๏ธA vue component support big amount data list with high render performance and efficient.
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.
๐จ ๐ ๐ป ๐ GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | One-Stop Graph Computing System
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)
Frequently Asked Questions
What is the most popular #big-data repository on GitHub?
The top ranked open source #big-data project on GitHub is binhnguyennus/awesome-scalability with 75,924 stars. The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
Which programming languages are most used in #big-data repositories?
Among the top ranked #big-data projects on GitHub, the most common programming languages are Java, Python, Rust, C++.
How is the #big-data repository ranking calculated?
We track 133 active open source repositories tagged with #big-data (500+ stars), totaling 683,600 stars with an average of 5,139 stars per repository. Rankings update daily from the GitHub API.