Git Stars Git Stars

#big-data Repositories

Discover 133 active open source GitHub repositories tagged with #big-data, accumulating 683,600 total stars (averaging 5,139 stars per project). Leading this topic is binhnguyennus/awesome-scalability with 75,924 stars.

133
Total Repositories
683,600
Total Stars
5,139
Avg Stars/Repo
1
binhnguyennus/awesome-scalability

The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

binhnguyennus
โญ 75,924 ๐Ÿ‘ค binhnguyennus User ๐Ÿ• Updated 2026-01-04 ๐Ÿ“… Created 2017-12-27
2
ClickHouse/ClickHouse

ClickHouseยฎ is a real-time analytics database management system

ClickHouse
โญ 51,051 C++ ๐Ÿ‘ค ClickHouse Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-06-02
3
apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

apache
โญ 44,593 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-02-25
4
donnemartin/data-science-ipython-notebooks

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

donnemartin
โญ 28,776 Python ๐Ÿ‘ค donnemartin User ๐Ÿ• Updated 2024-03-20 ๐Ÿ“… Created 2015-01-23
5
apache/flink

Apache Flink

apache
โญ 26,487 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-06-07
6
thingsboard/thingsboard

All-in-one IoT Platform - Device management, data collection, processing and visualization.

thingsboard
โญ 22,701 Java ๐Ÿ‘ค thingsboard Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-12-01
7
amark/gun

An open source cybersecurity protocol for syncing decentralized graph data.

amark
โญ 19,166 JavaScript ๐Ÿ‘ค amark User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2014-07-31
8
heibaiying/BigData-Notes

Big Data Introduction Guide :star:

heibaiying
โญ 16,818 Java ๐Ÿ‘ค heibaiying User ๐Ÿ• Updated 2024-01-05 ๐Ÿ“… Created 2019-03-10
9
prestodb/presto

The official home of the Presto distributed SQL query engine for big data

prestodb
โญ 16,777 Java ๐Ÿ‘ค prestodb Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2012-08-09
10
andkret/Cookbook

The Data Engineering Cookbook

andkret
โญ 15,594 Python ๐Ÿ‘ค andkret User ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2019-03-10
11
trinodb/trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

trinodb
โญ 13,532 Java ๐Ÿ‘ค trinodb Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-01-19
12
vesoft-inc/nebula

A distributed, fast open-source graph database featuring horizontal scalability and high availability

vesoft-inc
โญ 12,474 C++ ๐Ÿ‘ค vesoft-inc Organization ๐Ÿ• Updated 2026-09-08 ๐Ÿ“… Created 2018-08-28
13
StarRocks/starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

StarRocks
โญ 12,278 Java ๐Ÿ‘ค StarRocks Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-09-04
14
quickwit-oss/quickwit

Cloud-native OSS search engine for observability

quickwit-oss
โญ 11,810 Rust ๐Ÿ‘ค quickwit-oss Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-04-13
15
provectus/kafka-ui

Open-Source Web UI for Apache Kafka Management

provectus
โญ 11,754 Java ๐Ÿ‘ค provectus Organization ๐Ÿ• Updated 2024-07-26 ๐Ÿ“… Created 2019-11-26
16
cython/cython

The most widely used Python to C compiler

cython
โญ 10,898 Cython ๐Ÿ‘ค cython Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2010-11-21
17
apache/datafusion

Apache DataFusion SQL Query Engine

apache
โญ 9,631 Rust ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-04-17
18
catboost/catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

catboost
โญ 9,210 C++ ๐Ÿ‘ค catboost Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2017-07-18
19
delta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

delta-io
โญ 9,083 Scala ๐Ÿ‘ค delta-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-04-22
20
apache/beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

apache
โญ 8,723 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-02-02
21
arkime/arkime

Arkime is an open source, large scale, full packet capturing, indexing, and database system.

arkime
โญ 7,583 C ๐Ÿ‘ค arkime Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2012-07-06
22
h2oai/h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

h2oai
โญ 7,527 Jupyter Notebook ๐Ÿ‘ค h2oai Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2014-03-03
23
feast-dev/feast

The Open Source Feature Store for AI/ML

feast-dev
โญ 7,362 Python ๐Ÿ‘ค feast-dev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-12-10
24
vespa-engine/vespa

The AI search platform

vespa-engine
โญ 7,190 Java ๐Ÿ‘ค vespa-engine Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-06-03
25
apache/couchdb

Seamless multi-primary syncing database with an intuitive HTTP/JSON API, designed for reliability

apache
โญ 7,013 Erlang ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2009-05-21
26
apache/zeppelin

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

apache
โญ 6,670 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-03-25
27
hazelcast/hazelcast

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

hazelcast
โญ 6,636 Java ๐Ÿ‘ค hazelcast Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2012-03-21
28
apache/iotdb

Apache IoTDB

apache
โญ 6,437 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-11-24
29
pachyderm/pachyderm

Data-Centric Pipelines and Data Versioning

pachyderm
โญ 6,295 Go ๐Ÿ‘ค pachyderm Organization ๐Ÿ• Updated 2025-02-03 ๐Ÿ“… Created 2014-09-04
30
vcaesar/riot

Go Open Source, Distributed, Simple and fastest efficient text/vector Search Engine

vcaesar
โญ 6,068 Go ๐Ÿ‘ค vcaesar User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2017-06-21
31
apache/hive

Apache Hive

apache
โญ 6,038 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2009-05-21
32
Eventual-Inc/Daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

Eventual-Inc
โญ 5,858 Rust ๐Ÿ‘ค Eventual-Inc Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2022-04-25
33
microsoft/SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

microsoft
โญ 5,257 Scala ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2017-06-05
34
apache/calcite

Apache Calcite

apache
โญ 5,216 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2014-06-25
35
apache/ignite

Apache Ignite

apache
โญ 5,090 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-02-19
36
tschellenbach/Stream-Framework

Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:

tschellenbach
โญ 4,739 Python ๐Ÿ‘ค tschellenbach User ๐Ÿ• Updated 2025-12-04 ๐Ÿ“… Created 2012-08-08
37
tangbc/vue-virtual-scroll-list

โšก๏ธA vue component support big amount data list with high render performance and efficient.

tangbc
โญ 4,544 JavaScript ๐Ÿ‘ค tangbc User ๐Ÿ• Updated 2024-05-08 ๐Ÿ“… Created 2017-03-17
38
rom1504/img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

rom1504
โญ 4,458 Python ๐Ÿ‘ค rom1504 User ๐Ÿ• Updated 2025-10-19 ๐Ÿ“… Created 2021-08-11
39
crate/crate

CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.

crate
โญ 4,456 Java ๐Ÿ‘ค crate Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2013-04-10
40
alibaba/fastjson2

๐Ÿš„ FASTJSON2 is a Java JSON library with excellent performance.

alibaba
โญ 4,433 Java ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2022-04-17
41
Moataz-Elmesmary/Data-Science-Roadmap

Data Science Roadmap from A to Z

Moataz-Elmesmary
โญ 4,429 ๐Ÿ‘ค Moataz-Elmesmary User ๐Ÿ• Updated 2025-12-06 ๐Ÿ“… Created 2022-04-17
42
Rdatatable/data.table

R's data.table package extends data.frame:

Rdatatable
โญ 3,930 R ๐Ÿ‘ค Rdatatable Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2014-06-07
43
alibaba/GraphScope

๐Ÿ”จ ๐Ÿ‡ ๐Ÿ’ป ๐Ÿš€ GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | One-Stop Graph Computing System

alibaba
โญ 3,564 C++ ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2020-10-28
44
lakehq/sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

lakehq
โญ 3,504 Rust ๐Ÿ‘ค lakehq Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-21
45
apache/datafusion-sqlparser-rs

Extensible SQL Lexer and Parser for Rust

apache
โญ 3,483 Rust ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-02-06
โญ 3,476 HTML ๐Ÿ‘ค NLPIR-team Organization ๐Ÿ• Updated 2026-07-30 ๐Ÿ“… Created 2015-05-04
47
apache/paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

apache
โญ 3,454 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-01-12
48
databricks/koalas

Koalas: pandas API on Apache Spark

databricks
โญ 3,368 Python ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2024-03-20 ๐Ÿ“… Created 2019-01-03
49
apache/hugegraph

A graph database that supports more than 100+ billion data, high performance and scalability (Include OLTP Engine & REST-API & Backends)

apache
โญ 3,211 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-07-18
50
TuiQiao/CBoard

An easy to use, self-service open BI reporting and BI dashboard platform.

TuiQiao
โญ 3,099 JavaScript ๐Ÿ‘ค TuiQiao Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2016-09-21

Frequently Asked Questions

1

What is the most popular #big-data repository on GitHub?

The top ranked open source #big-data project on GitHub is binhnguyennus/awesome-scalability with 75,924 stars. The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

2

Which programming languages are most used in #big-data repositories?

Among the top ranked #big-data projects on GitHub, the most common programming languages are Java, Python, Rust, C++.

3

How is the #big-data repository ranking calculated?

We track 133 active open source repositories tagged with #big-data (500+ stars), totaling 683,600 stars with an average of 5,139 stars per repository. Rankings update daily from the GitHub API.