Git Stars Git Stars

#spark Repositories

Discover 126 active open source GitHub repositories tagged with #spark, accumulating 513,269 total stars (averaging 4,073 stars per project). Leading this topic is DataTalksClub/data-engineering-zoomcamp with 46,997 stars.

126
Total Repositories
513,269
Total Stars
4,073
Avg Stars/Repo
1
DataTalksClub/data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐Ÿ‘‡๐Ÿผ

DataTalksClub
โญ 46,997 Jupyter Notebook ๐Ÿ‘ค DataTalksClub Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2021-10-21
2
apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

apache
โญ 44,593 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-02-25
3
getredash/redash

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

getredash
โญ 28,895 TypeScript ๐Ÿ‘ค getredash Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2013-10-28
4
donnemartin/data-science-ipython-notebooks

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

donnemartin
โญ 28,776 Python ๐Ÿ‘ค donnemartin User ๐Ÿ• Updated 2024-03-20 ๐Ÿ“… Created 2015-01-23
5
yeasy/docker_practice

Latest Docker container technology, learn best practices from real cases! | Learn and understand Docker & Container technologies, with real DevOps practice!

yeasy
โญ 26,382 Go ๐Ÿ‘ค yeasy User ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2014-09-05
6
heibaiying/BigData-Notes

Big Data Introduction Guide :star:

heibaiying
โญ 16,818 Java ๐Ÿ‘ค heibaiying User ๐Ÿ• Updated 2024-01-05 ๐Ÿ“… Created 2019-03-10
7
FavioVazquez/ds-cheatsheets

List of Data Science Cheatsheets to rule the world

FavioVazquez
โญ 15,921 ๐Ÿ‘ค FavioVazquez User ๐Ÿ• Updated 2024-07-18 ๐Ÿ“… Created 2018-12-20
8
GaiZhenbiao/ChuanhuChatGPT

GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.

GaiZhenbiao
โญ 15,237 Python ๐Ÿ‘ค GaiZhenbiao User ๐Ÿ• Updated 2026-09-16 ๐Ÿ“… Created 2023-03-02
9
zhisheng17/flink-learning

Flink learning blog. http://www.54tianzhisheng.cn/ Contains content on Flink introduction, concepts, principles, practical implementation, performance tuning, source code analysis, etc. Involves learning cases on Flink Connector, Metrics, Library, DataStream API, Table API & SQL, etc., and sharing of large project cases for Flink production applications (PVUV, log storage, real-time deduplication of billions of data, monitoring alerts). Welcome everyone to support my column "Big Data Real-time Computing Engine Flink Practical and Performance Optimization".

zhisheng17
โญ 15,109 Java ๐Ÿ‘ค zhisheng17 User ๐Ÿ• Updated 2026-05-06 ๐Ÿ“… Created 2019-01-01
10
aalansehaiyang/technology-talk

ใ€Big Tech Interview Columnใ€‘ A technical guide for Java programmers, featuring interview questions, system architecture, career tips, mainstream middleware, and more, to help you become a better version of yourself!

aalansehaiyang
โญ 14,717 ๐Ÿ‘ค aalansehaiyang User ๐Ÿ• Updated 2025-07-21 ๐Ÿ“… Created 2016-06-10
11
horovod/horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

horovod
โญ 14,695 Python ๐Ÿ‘ค horovod Organization ๐Ÿ• Updated 2026-06-20 ๐Ÿ“… Created 2017-08-09
12
deeplearning4j/deeplearning4j

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

deeplearning4j
โญ 14,283 Java ๐Ÿ‘ค deeplearning4j Organization ๐Ÿ• Updated 2026-10-05 ๐Ÿ“… Created 2013-11-27
13
tobymao/sqlglot

Python SQL Parser and Transpiler

tobymao
โญ 9,809 Python ๐Ÿ‘ค tobymao User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-03-13
14
delta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

delta-io
โญ 9,083 Scala ๐Ÿ‘ค delta-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-04-22
15
mage-ai/mage-ai

๐Ÿง™ Build, run, and manage data pipelines for integrating and transforming data.

mage-ai
โญ 8,850 Python ๐Ÿ‘ค mage-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-05-16
16
h2oai/h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

h2oai
โญ 7,527 Jupyter Notebook ๐Ÿ‘ค h2oai Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2014-03-03
17
Alluxio/alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud

Alluxio
โญ 7,264 Java ๐Ÿ‘ค Alluxio Organization ๐Ÿ• Updated 2026-09-01 ๐Ÿ“… Created 2012-12-21
18
Angel-ML/angel

A Flexible and Powerful Parameter Server for large-scale machine learning

Angel-ML
โญ 6,797 Java ๐Ÿ‘ค Angel-ML Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2017-04-25
19
apache/zeppelin

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

apache
โญ 6,670 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-03-25
20
microsoft/SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

microsoft
โญ 5,257 Scala ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2017-06-05
21
tencentmusic/cube-studio

cube studio Open-Source Cloud-Native All-in-One Machine Learning/Deep Learning/Large Model AI Platform, MLOps algorithm link full process, computing power rental platform, notebook online development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference service VGPU virtualization, edge computing, annotation platform automated annotation, deepseek and other large models sft fine-tuning/reward model/reinforcement learning training, vllm/ollama/mindie large model multi-machine inference, private knowledge base, AI model market, supports domestic CPU/GPU/NPU Ascend ecosystem, supports RDMA, supports distributed training such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano

tencentmusic
โญ 5,085 ๐Ÿ‘ค tencentmusic Organization ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2021-08-17
22
apache/streampark

The Open Control Plane for Real-Time Data

apache
โญ 4,352 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-10-31 ๐Ÿ“… Created 2019-05-27
23
JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

JohnSnowLabs
โญ 4,162 Scala ๐Ÿ‘ค JohnSnowLabs Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2017-09-24
24
RoaringBitmap/RoaringBitmap

A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

RoaringBitmap
โญ 3,957 Java ๐Ÿ‘ค RoaringBitmap Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2013-06-17
25
Cyb3rWard0g/HELK

The Hunting ELK

Cyb3rWard0g
โญ 3,901 Jupyter Notebook ๐Ÿ‘ค Cyb3rWard0g User ๐Ÿ• Updated 2024-06-01 ๐Ÿ“… Created 2017-03-14
26
awslabs/deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

awslabs
โญ 3,660 Scala ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2018-08-07
27
lakehq/sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

lakehq
โญ 3,504 Rust ๐Ÿ‘ค lakehq Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-21
28
liyupi/sql-generator

๐Ÿ”จ Generate structured SQL statements using JSON, implemented with Vue3 + TypeScript + Vite + Ant Design + MonacoEditor, the project is simple (heavy logic, light pages), suitable for practice~

liyupi
โญ 3,455 Vue ๐Ÿ‘ค liyupi User ๐Ÿ• Updated 2024-01-17 ๐Ÿ“… Created 2022-05-12
29
apache/paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

apache
โญ 3,454 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-01-12
30
apache/linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

apache
โญ 3,417 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2019-07-23
31
databricks/koalas

Koalas: pandas API on Apache Spark

databricks
โญ 3,368 Python ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2024-03-20 ๐Ÿ“… Created 2019-01-03
32
WeBankFinTech/DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

WeBankFinTech
โญ 3,275 Java ๐Ÿ‘ค WeBankFinTech Organization ๐Ÿ• Updated 2025-11-04 ๐Ÿ“… Created 2019-11-24
33
lakesoul-io/LakeSoul

An open lakehouse engine for agents โ€” Postgres-native, vector-native, and Python-native. Speaks PostgreSQL, runs on your object storage, written in Rust.

lakesoul-io
โญ 3,261 Rust ๐Ÿ‘ค lakesoul-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-12-28
34
MoRan1607/BigDataGuide

Big Data Learning, Learn Big Data from Scratch, Includes Learning Videos and Interview Materials for Each Stage of Big Data Learning

MoRan1607
โญ 3,259 ๐Ÿ‘ค MoRan1607 User ๐Ÿ• Updated 2026-01-20 ๐Ÿ“… Created 2019-11-30
35
kubeflow/spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

kubeflow
โญ 3,162 Python ๐Ÿ‘ค kubeflow Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-01-03
36
Luce-Org/lucebox

Lucebox inference engine for heterogeneous hardware & consumer GPUs

Luce-Org
โญ 3,024 C++ ๐Ÿ‘ค Luce-Org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-03
37
RayXBuilds/Movie_Recommend

Historical learning project: Spark movie recommendations, web scraping, web app, and backend. Ongoing development of the new version: cinematch-recommender.

RayXBuilds
โญ 3,007 Java ๐Ÿ‘ค RayXBuilds User ๐Ÿ• Updated 2026-10-21 ๐Ÿ“… Created 2018-04-18
38
spark-jobserver/spark-jobserver

REST job server for Apache Spark

spark-jobserver
โญ 2,833 Scala ๐Ÿ‘ค spark-jobserver Organization ๐Ÿ• Updated 2026-03-03 ๐Ÿ“… Created 2014-08-21
39
vector4wang/spring-boot-quick

:herb: Quick learning examples based on Spring Boot, integrating open-source frameworks I've encountered, such as: RabbitMQ (delay queue), Kafka, JPA, Redis, OAuth2, Swagger, JSP, Docker, K3s, K3d, K8s, MyBatis encryption/decryption plugin, exception handling, log output, multi-module development, multi-environment packaging, cache, web crawler, JWT, GraphQL, Dubbo, Zookeeper, and Async, etc. :pushpin:

vector4wang
โญ 2,789 Java ๐Ÿ‘ค vector4wang User ๐Ÿ• Updated 2026-05-16 ๐Ÿ“… Created 2017-03-13
40
moj-analytical-services/splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

moj-analytical-services
โญ 2,592 Python ๐Ÿ‘ค moj-analytical-services Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2019-11-22
41
geekyouth/SZT-bigdata

Shenzhen Metro Big Data Passenger Flow Analysis System๐Ÿš‡๐Ÿš„๐ŸŒŸ

geekyouth
โญ 2,469 Scala ๐Ÿ‘ค geekyouth User ๐Ÿ• Updated 2026-05-12 ๐Ÿ“… Created 2020-04-13
42
XZB-1248/Spark

โœจSpark is a web-based, cross-platform and full-featured Remote Administration Tool (RAT) written in Go that allows you control all your devices anywhere. Spark is a Go-written, web-based, cross-platform and feature-rich remote control and monitoring tool that allows you to monitor and control all devices anytime and anywhere.

XZB-1248
โญ 2,412 Go ๐Ÿ‘ค XZB-1248 User ๐Ÿ• Updated 2026-03-16 ๐Ÿ“… Created 2022-03-16
43
apache/kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

apache
โญ 2,379 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2017-12-18
44
ytsaurus/ytsaurus

YTsaurus is a scalable and fault-tolerant open-source big data platform.

ytsaurus
โญ 2,325 C++ ๐Ÿ‘ค ytsaurus Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-12-05
45
o-gs/dji-firmware-tools

Tools for handling firmwares of DJI products, with focus on quadcopters.

o-gs
โญ 2,309 C ๐Ÿ‘ค o-gs Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2016-08-03
46
salesforce/TransmogrifAI

TransmogrifAI (pronounced trฤƒns-mลgหˆrษ™-fฤซ) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning

salesforce
โญ 2,280 Scala ๐Ÿ‘ค salesforce Organization ๐Ÿ• Updated 2026-06-02 ๐Ÿ“… Created 2017-11-02
47
fugue-project/fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

fugue-project
โญ 2,168 Python ๐Ÿ‘ค fugue-project Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2020-03-24
48
zio/zio-quill

Compile-time Language Integrated Queries for Scala

zio
โญ 2,165 Scala ๐Ÿ‘ค zio Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-07-20
49
dotnet/spark

.NET for Apacheยฎ Sparkโ„ข makes Apache Sparkโ„ข easily accessible to .NET developers.

dotnet
โญ 2,096 C# ๐Ÿ‘ค dotnet Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2019-04-22
50
broadinstitute/gatk

Official code repository for GATK versions 4 and up

broadinstitute
โญ 2,010 Java ๐Ÿ‘ค broadinstitute Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-12-02

Frequently Asked Questions

1

What is the most popular #spark repository on GitHub?

The top ranked open source #spark project on GitHub is DataTalksClub/data-engineering-zoomcamp with 46,997 stars. Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐Ÿ‘‡๐Ÿผ

2

Which programming languages are most used in #spark repositories?

Among the top ranked #spark projects on GitHub, the most common programming languages are Java, Scala, Python, Jupyter Notebook.

3

How is the #spark repository ranking calculated?

We track 126 active open source repositories tagged with #spark (500+ stars), totaling 513,269 stars with an average of 4,073 stars per repository. Rankings update daily from the GitHub API.