Git Stars Git Stars

#spark Repositories

126
Total Repositories
513,269
Total Stars
4,073
Avg Stars/Repo
51
elastic/elasticsearch-hadoop

:elephant: Elasticsearch real-time search and analytics natively integrated with Hadoop

elastic
โญ 1,966 Java ๐Ÿ‘ค elastic Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2013-03-11
52
apache/cassandra-spark-connector

Apache Spark to Apache Cassandra connector

apache
โญ 1,948 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2025-04-29 ๐Ÿ“… Created 2014-06-27
53
apache/auron

Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.

apache
โญ 1,849 Rust ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-06-28
54
collabH/bigdata-growth

The Big Data knowledge repository involves data warehouse modeling, real-time computing, Big Data, data middle platform, system design, Java, algorithms, etc.

collabH
โญ 1,838 Python ๐Ÿ‘ค collabH User ๐Ÿ• Updated 2026-04-18 ๐Ÿ“… Created 2020-06-10
55
OBenner/data-engineering-interview-questions

More than 2000+ Data engineer interview questions.

OBenner
โญ 1,815 Python ๐Ÿ‘ค OBenner User ๐Ÿ• Updated 2026-01-13 ๐Ÿ“… Created 2021-08-08
56
zhonghuasheng/Tutorial

Backend (Java Golang) Full-Stack Knowledge Architecture System Summary

zhonghuasheng
โญ 1,756 Shell ๐Ÿ‘ค zhonghuasheng User ๐Ÿ• Updated 2025-07-20 ๐Ÿ“… Created 2016-01-23
57
apachecn/.github

ApacheCN Open Source Organization: Announcements, Introduction, Members, Activities, Communication Methods

apachecn
โญ 1,750 CSS ๐Ÿ‘ค apachecn Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2018-07-19
58
strapdata/elassandra

Elassandra = Elasticsearch + Apache Cassandra

strapdata
โญ 1,718 Java ๐Ÿ‘ค strapdata Organization ๐Ÿ• Updated 2026-05-17 ๐Ÿ“… Created 2015-08-22
59
jadianes/spark-py-notebooks

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

jadianes
โญ 1,667 Jupyter Notebook ๐Ÿ‘ค jadianes User ๐Ÿ• Updated 2024-03-16 ๐Ÿ“… Created 2015-05-06
60
almond-sh/almond

A Scala kernel for Jupyter

almond-sh
โญ 1,631 Scala ๐Ÿ‘ค almond-sh Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2015-03-10
61
japila-books/apache-spark-internals

The Internals of Apache Spark

japila-books
โญ 1,553 ๐Ÿ‘ค japila-books Organization ๐Ÿ• Updated 2026-10-10 ๐Ÿ“… Created 2015-08-31
62
combust/mleap

MLeap: Deploy ML Pipelines to Production

combust
โญ 1,544 Scala ๐Ÿ‘ค combust Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2016-08-23
63
hi-primus/optimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

hi-primus
โญ 1,539 Python ๐Ÿ‘ค hi-primus Organization ๐Ÿ• Updated 2024-12-02 ๐Ÿ“… Created 2017-07-13
64
projectnessie/nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

projectnessie
โญ 1,533 Java ๐Ÿ‘ค projectnessie Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-04-09
65
apache/carbondata

High performance data store solution

apache
โญ 1,452 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-10-02 ๐Ÿ“… Created 2016-06-28
66
databricks/LearningSparkV2

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

databricks
โญ 1,396 Scala ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2025-01-28 ๐Ÿ“… Created 2019-02-10
67
HariSekhon/Dockerfiles

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

HariSekhon
โญ 1,382 Shell ๐Ÿ‘ค HariSekhon User ๐Ÿ• Updated 2026-02-03 ๐Ÿ“… Created 2016-01-17
68
jupyter-incubator/sparkmagic

Jupyter magics and kernels for working with remote Spark clusters

jupyter-incubator
โญ 1,368 Python ๐Ÿ‘ค jupyter-incubator Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-09-21
69
apache/datafusion-comet

Apache DataFusion Comet Spark Accelerator

apache
โญ 1,326 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-01-15
70
iflytek/datasophon

The next generation of cloud-native big data management expert , Aims to help users rapidly build stable, efficient, and scalable cloud-native platforms for big data.

iflytek
โญ 1,325 Java ๐Ÿ‘ค iflytek Organization ๐Ÿ• Updated 2025-07-22 ๐Ÿ“… Created 2022-10-11
71
DTStack/Taier

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display

DTStack
โญ 1,285 Java ๐Ÿ‘ค DTStack Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2021-03-02
72
mahmoudparsian/pyspark-tutorial

PySpark-Tutorial provides basic algorithms using PySpark

mahmoudparsian
โญ 1,276 Jupyter Notebook ๐Ÿ‘ค mahmoudparsian User ๐Ÿ• Updated 2025-05-26 ๐Ÿ“… Created 2015-03-12
73
zinggAI/zingg

AI-based entity resolution, identity resolution, master data management, and deduplication at scale for humans and agents

zinggAI
โญ 1,275 Java ๐Ÿ‘ค zinggAI Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2021-08-25
74
graphframes/graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes
โญ 1,213 Scala ๐Ÿ‘ค graphframes Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2016-01-20
75
apache/amoro

Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

apache
โญ 1,207 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-07-14
76
abhishek-ch/around-dataengineering

A Data Engineering & Machine Learning Knowledge Hub

abhishek-ch
โญ 1,140 Python ๐Ÿ‘ค abhishek-ch User ๐Ÿ• Updated 2024-02-02 ๐Ÿ“… Created 2020-09-18
77
apache/celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

apache
โญ 1,081 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-12-10
78
metabrainz/listenbrainz-server

Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.

metabrainz
โญ 1,074 Python ๐Ÿ‘ค metabrainz Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-09-05
79
bigdatagenomics/adam

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

bigdatagenomics
โญ 1,060 Scala ๐Ÿ‘ค bigdatagenomics Organization ๐Ÿ• Updated 2026-03-17 ๐Ÿ“… Created 2013-11-19
80
NVIDIA/cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

NVIDIA
โญ 1,021 Scala ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-05-14
81
deanwampler/spark-scala-tutorial

A free tutorial for Apache Spark.

deanwampler
โญ 988 Jupyter Notebook ๐Ÿ‘ค deanwampler User ๐Ÿ• Updated 2026-01-05 ๐Ÿ“… Created 2014-05-01
82
h2oai/sparkling-water

Sparkling Water provides H2O functionality inside Spark cluster

h2oai
โญ 982 Scala ๐Ÿ‘ค h2oai Organization ๐Ÿ• Updated 2025-11-05 ๐Ÿ“… Created 2014-10-13
83
sparklyr/sparklyr

R interface for Apache Spark

sparklyr
โญ 970 R ๐Ÿ‘ค sparklyr Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2016-05-20
84
data-prep-kit/data-prep-kit

Open source project for data preparation for GenAI applications

data-prep-kit
โญ 968 HTML ๐Ÿ‘ค data-prep-kit Organization ๐Ÿ• Updated 2026-09-29 ๐Ÿ“… Created 2024-04-08
85
delta-io/delta-sharing

An open protocol for secure data sharing

delta-io
โญ 965 Scala ๐Ÿ‘ค delta-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-04-08
86
apache/livy

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

apache
โญ 962 Scala ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2017-06-25
87
microsoft/Mobius

C# and F# language binding and extensions to Apache Spark

microsoft
โญ 948 C# ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-12-11 ๐Ÿ“… Created 2015-10-27
88
RedisLabs/spark-redis

A connector for Spark that allows reading and writing to/from Redis cluster

RedisLabs
โญ 944 Scala ๐Ÿ‘ค RedisLabs Organization ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2015-08-16
89
XHToken/Spark-X2.6

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

XHToken
โญ 927 ๐Ÿ‘ค XHToken Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2026-08-24
90
typelevel/frameless

Expressive types for Spark.

typelevel
โญ 898 Scala ๐Ÿ‘ค typelevel Organization ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2015-03-28
91
pingcap/tispark

TiSpark is built for running Apache Spark on top of TiDB/TiKV

pingcap
โญ 889 Scala ๐Ÿ‘ค pingcap Organization ๐Ÿ• Updated 2026-08-20 ๐Ÿ“… Created 2017-05-02
92
ohenley/awesome-ada

A curated list of awesome resources related to the Ada and SPARK programming language

ohenley
โญ 881 ๐Ÿ‘ค ohenley User ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2018-10-24
93
awslabs/data-on-eks

DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS

awslabs
โญ 863 Shell ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-09-25
94
LucaCanali/sparkMeasure

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

LucaCanali
โญ 831 Scala ๐Ÿ‘ค LucaCanali User ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2017-03-16
95
HariSekhon/DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

HariSekhon
โญ 824 Python ๐Ÿ‘ค HariSekhon User ๐Ÿ• Updated 2026-02-03 ๐Ÿ“… Created 2015-10-27
96
WeBankFinTech/Scriptis

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

WeBankFinTech
โญ 812 Vue ๐Ÿ‘ค WeBankFinTech Organization ๐Ÿ• Updated 2024-12-11 ๐Ÿ“… Created 2019-07-23
98
cdapio/cdap

An open source framework for building data analytic applications.

cdapio
โญ 791 Java ๐Ÿ‘ค cdapio Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-08-02
99
a616567126/GPT-WEB-JAVA

JDK8 based AI Chatbot! WeChat Official Account Midjourney Graphics, Key Redemption, Web Support ChatGPT, Midjourney Graphics, SD Graphics, Key Redemption, YiPay, Official Account Traffic Acquisition, Email Registration๐Ÿ”ฅ

a616567126
โญ 774 Java ๐Ÿ‘ค a616567126 User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-03-28
100
MigoXLab/dingo

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

MigoXLab
โญ 771 Python ๐Ÿ‘ค MigoXLab Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-12-24