Git Stars Git Stars

#bigdata Repositories

Discover 51 active open source GitHub repositories tagged with #bigdata, accumulating 261,861 total stars (averaging 5,134 stars per project). Leading this topic is DataExpert-io/data-engineer-handbook with 45,237 stars.

51
Total Repositories
261,861
Total Stars
5,134
Avg Stars/Repo
1
DataExpert-io/data-engineer-handbook

This is a repo with links to everything you'd ever want to learn about data engineering

DataExpert-io
โญ 45,237 Jupyter Notebook ๐Ÿ‘ค DataExpert-io Organization ๐Ÿ• Updated 2026-08-03 ๐Ÿ“… Created 2023-11-19
2
rustfs/rustfs

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

rustfs
โญ 36,341 Rust ๐Ÿ‘ค rustfs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-11-23
3
taosdata/TDengine

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

taosdata
โญ 25,203 C ๐Ÿ‘ค taosdata Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2019-07-11
4
apache/shardingsphere

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

apache
โญ 20,832 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-01-18
5
heibaiying/BigData-Notes

Big Data Introduction Guide :star:

heibaiying
โญ 16,818 Java ๐Ÿ‘ค heibaiying User ๐Ÿ• Updated 2024-01-05 ๐Ÿ“… Created 2019-03-10
6
juicedata/juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

juicedata
โญ 14,766 Go ๐Ÿ‘ค juicedata Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-01-08
7
oxnr/awesome-bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.

oxnr
โญ 14,744 ๐Ÿ‘ค oxnr User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2014-07-04
8
databendlabs/databend

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. โ€” rebuilt from scratch. Unified architecture on your S3.

databendlabs
โญ 9,502 Rust ๐Ÿ‘ค databendlabs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-10-10
9
vaexio/vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second ๐Ÿš€

vaexio
โญ 8,518 Python ๐Ÿ‘ค vaexio Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2014-09-27
10
apache/hudi

Upserts, Deletes And Incremental Processing on Big Data.

apache
โญ 6,387 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-12-14
11
volcano-sh/volcano

A Cloud Native Batch System (Project under CNCF)

volcano-sh
โญ 6,106 Go ๐Ÿ‘ค volcano-sh Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2019-03-14
12
iGaoWei/BigDataView

100+ sets of Big Data visualization cool large screen Html5 templates; Included industries: community, property management, government affairs, transportation, financial banking, etc., the latest, most numerous, most complete, coolest, most dazzling Big Data visualization templates on the whole network. Gradually updating.

iGaoWei
โญ 5,352 JavaScript ๐Ÿ‘ค iGaoWei User ๐Ÿ• Updated 2025-10-24 ๐Ÿ“… Created 2021-07-29
13
DTStack/chunjun

A data integration framework

DTStack
โญ 4,107 Java ๐Ÿ‘ค DTStack Organization ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2018-04-03
14
liyupi/sql-generator

๐Ÿ”จ Generate structured SQL statements using JSON, implemented with Vue3 + TypeScript + Vite + Ant Design + MonacoEditor, the project is simple (heavy logic, light pages), suitable for practice~

liyupi
โญ 3,455 Vue ๐Ÿ‘ค liyupi User ๐Ÿ• Updated 2024-01-17 ๐Ÿ“… Created 2022-05-12
15
apache/avro

Apache Avro is a data serialization system.

apache
โญ 3,314 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2009-05-21
16
MoRan1607/BigDataGuide

Big Data Learning, Learn Big Data from Scratch, Includes Learning Videos and Interview Materials for Each Stage of Big Data Learning

MoRan1607
โญ 3,259 ๐Ÿ‘ค MoRan1607 User ๐Ÿ• Updated 2026-01-20 ๐Ÿ“… Created 2019-11-30
17
griddb/griddb

GridDB is a next-generation open source database that makes time series IoT and big data fast and easy.

griddb
โญ 2,470 C++ ๐Ÿ‘ค griddb Organization ๐Ÿ• Updated 2026-03-19 ๐Ÿ“… Created 2016-02-24
18
apconw/Aix-DB

Aix-DB, based on the LangChain/LangGraph framework and combined with the MCP Skills multi-agent collaboration architecture, accomplishes end-to-end conversion from natural language to data insights.

apconw
โญ 2,268 JavaScript ๐Ÿ‘ค apconw User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2024-11-15
19
dotnet/spark

.NET for Apacheยฎ Sparkโ„ข makes Apache Sparkโ„ข easily accessible to .NET developers.

dotnet
โญ 2,096 C# ๐Ÿ‘ค dotnet Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2019-04-22
20
DTStack/flinkStreamSQL

Based on the open-source Flink, it extends its real-time SQL; mainly implements the join between streams and dimension tables, supporting all syntax of native Flink SQL

DTStack
โญ 2,058 Java ๐Ÿ‘ค DTStack Organization ๐Ÿ• Updated 2024-02-21 ๐Ÿ“… Created 2018-09-12
21
byzer-org/byzer-lang

Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.

byzer-org
โญ 1,849 Scala ๐Ÿ‘ค byzer-org Organization ๐Ÿ• Updated 2024-05-29 ๐Ÿ“… Created 2016-04-28
22
collabH/bigdata-growth

The Big Data knowledge repository involves data warehouse modeling, real-time computing, Big Data, data middle platform, system design, Java, algorithms, etc.

collabH
โญ 1,838 Python ๐Ÿ‘ค collabH User ๐Ÿ• Updated 2026-04-18 ๐Ÿ“… Created 2020-06-10
23
Netflix/genie

Distributed Big Data Orchestration Service

Netflix
โญ 1,766 Java ๐Ÿ‘ค Netflix Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2013-06-20
24
YoongiKim/AutoCrawler

Google, Naver multiprocess image web crawler (Selenium)

YoongiKim
โญ 1,700 Python ๐Ÿ‘ค YoongiKim User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2018-11-21
25
jadianes/spark-py-notebooks

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

jadianes
โญ 1,667 Jupyter Notebook ๐Ÿ‘ค jadianes User ๐Ÿ• Updated 2024-03-16 ๐Ÿ“… Created 2015-05-06
26
hi-primus/optimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

hi-primus
โญ 1,539 Python ๐Ÿ‘ค hi-primus Organization ๐Ÿ• Updated 2024-12-02 ๐Ÿ“… Created 2017-07-13
27
opendatadiscovery/odd-platform

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

opendatadiscovery
โญ 1,441 Java ๐Ÿ‘ค opendatadiscovery Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2021-07-07
28
apache/celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

apache
โญ 1,081 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-12-10
29
zeromicro/cds

Data syncing in golang for ClickHouse.

zeromicro
โญ 982 Go ๐Ÿ‘ค zeromicro Organization ๐Ÿ• Updated 2025-06-07 ๐Ÿ“… Created 2020-11-25
30
apache/livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

apache
โญ 962 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2017-06-25
31
microsoft/Mobius

C# and F# language binding and extensions to Apache Spark

microsoft
โญ 948 C# ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-12-11 ๐Ÿ“… Created 2015-10-27
32
visualpython/visualpython

GUI-based Python code generator for data science, extension to Jupyter Lab, Jupyter Notebook and Google Colab.

visualpython
โญ 917 JavaScript ๐Ÿ‘ค visualpython User ๐Ÿ• Updated 2024-07-03 ๐Ÿ“… Created 2021-05-02
33
pingcap/tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV

pingcap
โญ 889 Scala ๐Ÿ‘ค pingcap Organization ๐Ÿ• Updated 2026-08-20 ๐Ÿ“… Created 2017-05-02
34
intsmaze/flink-boot

The Lazy Squirrel Flink-Boot scaffold allows Flink to fully embrace the Spring ecosystem, enabling developers to develop distributed running stream processing programs with the Java WEB development mode. Lazy Squirrel makes crossover simpler. Lazy Squirrel aims to enable developers to quickly write and implement business code with lower entry costs (without needing to understand the theory of distributed computing and the details of the Flink framework). To further enhance the agility of developers using the Lazy Squirrel scaffold to develop large projects, the scaffold defaults to integrating the Spring framework for Bean management, while also integrating frameworks commonly used in the microservices and WEB development fields, further improving development speed. For example, integrating the Mybatis ORM framework, Hibernate Validator validation framework, Spring Retry retry framework, etc. See the scaffold features below for details.

intsmaze
โญ 827 Java ๐Ÿ‘ค intsmaze User ๐Ÿ• Updated 2024-04-27 ๐Ÿ“… Created 2020-11-10
35
NationalSecurityAgency/datawave

DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.

NationalSecurityAgency
โญ 779 Java ๐Ÿ‘ค NationalSecurityAgency Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-10
36
gearpump/gearpump

Lightweight real-time big data streaming engine over Apache Pekko

gearpump
โญ 757 Scala ๐Ÿ‘ค gearpump Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2014-07-23
37
MrXujiang/v6.dooring.public

Visual dashboard solution, providing a visual editing engine to help individuals or businesses easily customize their own visual dashboard applications.

MrXujiang
โญ 715 TypeScript ๐Ÿ‘ค MrXujiang User ๐Ÿ• Updated 2025-07-06 ๐Ÿ“… Created 2021-03-08
38
WeBankFinTech/WeDataSphere

WeDataSphere is a financial grade, one-stop big data platform suite.

WeBankFinTech
โญ 676 ๐Ÿ‘ค WeBankFinTech Organization ๐Ÿ• Updated 2024-03-14 ๐Ÿ“… Created 2019-07-23
39
bigartm/bigartm

Fast topic modeling platform

bigartm
โญ 673 C++ ๐Ÿ‘ค bigartm Organization ๐Ÿ• Updated 2026-02-05 ๐Ÿ“… Created 2014-09-15
40
AbsaOSS/spline

Data Lineage Tracking And Visualization Solution

AbsaOSS
โญ 669 Scala ๐Ÿ‘ค AbsaOSS Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2017-05-30
41
unum-cloud/UStore

Embedded transactional database for blobs, documents & graphs, with kernel-bypass NVMe through io_uring & SPDK, GPU compaction, and one C ABI over its own, RocksDB & LevelDB engines ๐Ÿ—„๏ธ

unum-cloud
โญ 649 C++ ๐Ÿ‘ค unum-cloud Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-06-12
42
gangly/datafaker

Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to varied data sources. Test Data Generation Tool

gangly
โญ 643 Python ๐Ÿ‘ค gangly User ๐Ÿ• Updated 2026-04-04 ๐Ÿ“… Created 2019-03-26
43
marmotdata/marmot

The open source knowledgehouse for your company. Marmot knows what your data means, governs who may use it and runs every query your people and agents make, checked and recorded. Catalog, lineage, governed SQL and an MCP server in one Go binary.

marmotdata
โญ 639 Go ๐Ÿ‘ค marmotdata Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-03-12
44
minio/sidekick

High Performance HTTP Sidecar Load Balancer

minio
โญ 585 Go ๐Ÿ‘ค minio Organization ๐Ÿ• Updated 2025-07-14 ๐Ÿ“… Created 2020-03-05
45
achtungsoftware/alarik

High performance distributed S3 compatible object storage focused on speed and designed to be an open alternative to MinIO and RustFS.

achtungsoftware
โญ 584 Swift ๐Ÿ‘ค achtungsoftware Organization ๐Ÿ• Updated 2026-09-19 ๐Ÿ“… Created 2025-12-06
46
ganweisoft/TOMs

TOMs is a fully open-source, high-performance, systematic, plugin-oriented, and scenario-agnostic general-purpose development framework.

ganweisoft
โญ 581 CSS ๐Ÿ‘ค ganweisoft Organization ๐Ÿ• Updated 2025-12-04 ๐Ÿ“… Created 2025-06-11
47
mvillarrealb/docker-spark-cluster

A simple spark standalone cluster for your testing environment purposes

mvillarrealb
โญ 570 Dockerfile ๐Ÿ‘ค mvillarrealb User ๐Ÿ• Updated 2024-03-06 ๐Ÿ“… Created 2018-09-23
48
leesf/hudi-resources

Summary of Apache Hudi related materials

leesf
โญ 554 ๐Ÿ‘ค leesf User ๐Ÿ• Updated 2026-03-31 ๐Ÿ“… Created 2019-12-11
49
zhaoyachao/zdh_web

Big Data Collection, Extraction Platform, zdh_web is the visual management platform for zdh series services, including modules such as data collection, scheduling, permissions, approval workflow, private domain marketing, etc.

zhaoyachao
โญ 536 HTML ๐Ÿ‘ค zhaoyachao User ๐Ÿ• Updated 2026-06-07 ๐Ÿ“… Created 2020-02-03
50
nicgirault/circosJS

d3 library to build circular graphs

nicgirault
โญ 534 JavaScript ๐Ÿ‘ค nicgirault User ๐Ÿ• Updated 2026-01-23 ๐Ÿ“… Created 2014-11-28

Frequently Asked Questions

1

What is the most popular #bigdata repository on GitHub?

The top ranked open source #bigdata project on GitHub is DataExpert-io/data-engineer-handbook with 45,237 stars. This is a repo with links to everything you'd ever want to learn about data engineering

2

Which programming languages are most used in #bigdata repositories?

Among the top ranked #bigdata projects on GitHub, the most common programming languages are Java, Go, Python, Scala.

3

How is the #bigdata repository ranking calculated?

We track 51 active open source repositories tagged with #bigdata (500+ stars), totaling 261,861 stars with an average of 5,134 stars per repository. Rankings update daily from the GitHub API.