Git Stars Git Stars

#pyspark Repositories

Discover 35 active open source GitHub repositories tagged with #pyspark, accumulating 60,070 total stars (averaging 1,716 stars per project). Leading this topic is ibis-project/ibis with 6,705 stars.

35
Total Repositories
60,070
Total Stars
1,716
Avg Stars/Repo
1
ibis-project/ibis

the portable Python dataframe library

ibis-project
โญ 6,705 Python ๐Ÿ‘ค ibis-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-04-17
2
microsoft/SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

microsoft
โญ 5,257 Scala ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2017-06-05
3
JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

JohnSnowLabs
โญ 4,162 Scala ๐Ÿ‘ค JohnSnowLabs Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2017-09-24
4
databricks/learning-spark

Example code from Learning Spark book

databricks
โญ 3,895 Java ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2026-06-30 ๐Ÿ“… Created 2014-06-16
5
lakehq/sail

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

lakehq
โญ 3,504 Rust ๐Ÿ‘ค lakehq Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-21
6
apache/linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

apache
โญ 3,417 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2019-07-23
7
danielbeach/data-engineering-practice

Data Engineering Practice Problems

danielbeach
โญ 2,706 Python ๐Ÿ‘ค danielbeach User ๐Ÿ• Updated 2025-01-08 ๐Ÿ“… Created 2022-02-17
8
awesome-spark/awesome-spark

A curated list of awesome Apache Spark packages and resources.

awesome-spark
โญ 1,903 Shell ๐Ÿ‘ค awesome-spark Organization ๐Ÿ• Updated 2026-02-27 ๐Ÿ“… Created 2016-02-01
9
uber/petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

uber
โญ 1,894 Python ๐Ÿ‘ค uber Organization ๐Ÿ• Updated 2026-01-02 ๐Ÿ“… Created 2018-06-15
10
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

narwhals-dev
โญ 1,754 Python ๐Ÿ‘ค narwhals-dev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-02-19
11
jadianes/spark-py-notebooks

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

jadianes
โญ 1,667 Jupyter Notebook ๐Ÿ‘ค jadianes User ๐Ÿ• Updated 2024-03-16 ๐Ÿ“… Created 2015-05-06
12
hi-primus/optimus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

hi-primus
โญ 1,539 Python ๐Ÿ‘ค hi-primus Organization ๐Ÿ• Updated 2024-12-02 ๐Ÿ“… Created 2017-07-13
โญ 1,435 ๐Ÿ‘ค anshlambagit User ๐Ÿ• Updated 2026-01-31 ๐Ÿ“… Created 2025-10-25
14
spark-examples/pyspark-examples

Pyspark RDD, DataFrame and Dataset Examples in Python language

spark-examples
โญ 1,371 Python ๐Ÿ‘ค spark-examples Organization ๐Ÿ• Updated 2025-12-07 ๐Ÿ“… Created 2019-11-26
15
jupyter-incubator/sparkmagic

Jupyter magics and kernels for working with remote Spark clusters

jupyter-incubator
โญ 1,368 Python ๐Ÿ‘ค jupyter-incubator Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-09-21
16
logicalclocks/hopsworks

Hopsworks - Data-Intensive AI platform with a Feature Store

logicalclocks
โญ 1,299 Java ๐Ÿ‘ค logicalclocks Organization ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2018-07-26
17
mahmoudparsian/pyspark-tutorial

PySpark-Tutorial provides basic algorithms using PySpark

mahmoudparsian
โญ 1,276 Jupyter Notebook ๐Ÿ‘ค mahmoudparsian User ๐Ÿ• Updated 2025-05-26 ๐Ÿ“… Created 2015-03-12
18
graphframes/graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes
โญ 1,213 Scala ๐Ÿ‘ค graphframes Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2016-01-20
19
JohnSnowLabs/spark-nlp-workshop

Public runnable examples of using John Snow Labs' NLP for Apache Spark.

JohnSnowLabs
โญ 1,089 Jupyter Notebook ๐Ÿ‘ค JohnSnowLabs Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2018-08-20
20
mahmoudparsian/data-algorithms-book

MapReduce, Spark, Java, and Scala for Data Algorithms Book

mahmoudparsian
โญ 1,083 Java ๐Ÿ‘ค mahmoudparsian User ๐Ÿ• Updated 2024-10-14 ๐Ÿ“… Created 2014-08-06
21
h2oai/sparkling-water

Sparkling Water provides H2O functionality inside Spark cluster

h2oai
โญ 982 Scala ๐Ÿ‘ค h2oai Organization ๐Ÿ• Updated 2025-11-05 ๐Ÿ“… Created 2014-10-13
22
criteo/autofaiss

Automatically create Faiss knn indices with the most optimal similarity search parameters.

criteo
โญ 906 Python ๐Ÿ‘ค criteo Organization ๐Ÿ• Updated 2025-11-04 ๐Ÿ“… Created 2021-04-28
23
PatrikHlobil/Pandas-Bokeh

Bokeh Plotting Backend for Pandas and GeoPandas

PatrikHlobil
โญ 889 Python ๐Ÿ‘ค PatrikHlobil User ๐Ÿ• Updated 2024-04-10 ๐Ÿ“… Created 2018-11-23
24
pyspark-ai/pyspark-ai

English SDK for Apache Spark

pyspark-ai
โญ 879 Python ๐Ÿ‘ค pyspark-ai Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2023-04-27
25
awslabs/python-deequ

Python API for Deequ

awslabs
โญ 826 Jupyter Notebook ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2026-10-24 ๐Ÿ“… Created 2020-11-09
26
HariSekhon/DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

HariSekhon
โญ 824 Python ๐Ÿ‘ค HariSekhon User ๐Ÿ• Updated 2026-02-03 ๐Ÿ“… Created 2015-10-27
27
Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Nike-Inc
โญ 818 Python ๐Ÿ‘ค Nike-Inc Organization ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2024-05-06
28
WeBankFinTech/Scriptis

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

WeBankFinTech
โญ 812 Vue ๐Ÿ‘ค WeBankFinTech Organization ๐Ÿ• Updated 2024-12-11 ๐Ÿ“… Created 2019-07-23
29
MrPowers/chispa

PySpark test helper methods with beautiful error messages

MrPowers
โญ 778 Python ๐Ÿ‘ค MrPowers User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-19
30
databricks/tensorframes

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

databricks
โญ 745 Scala ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2024-07-30 ๐Ÿ“… Created 2016-03-04
31
awslabs/aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.

awslabs
โญ 701 Python ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2017-08-11
32
mrpowers-io/quinn

pyspark methods to enhance developer productivity ๐Ÿ“ฃ ๐Ÿ‘ฏ ๐ŸŽ‰

mrpowers-io
โญ 691 Python ๐Ÿ‘ค mrpowers-io Organization ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2017-09-15
33
capitalone/datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

capitalone
โญ 669 Python ๐Ÿ‘ค capitalone Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2018-03-23
34
cluster-apps-on-docker/spark-standalone-cluster-on-docker

Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. :zap:

cluster-apps-on-docker
โญ 510 Jupyter Notebook ๐Ÿ‘ค cluster-apps-on-docker Organization ๐Ÿ• Updated 2025-11-07 ๐Ÿ“… Created 2020-07-03
35
databrickslabs/dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) can be used to generate large simulated/synthetic datasets for testing, POCs, and other uses in Databricks environments, including Delta Live Tables pipelines.

databrickslabs
โญ 506 Python ๐Ÿ‘ค databrickslabs Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2019-07-23

Frequently Asked Questions

1

What is the most popular #pyspark repository on GitHub?

The top ranked open source #pyspark project on GitHub is ibis-project/ibis with 6,705 stars. the portable Python dataframe library

2

Which programming languages are most used in #pyspark repositories?

Among the top ranked #pyspark projects on GitHub, the most common programming languages are Python, Scala, Jupyter Notebook, Java.

3

How is the #pyspark repository ranking calculated?

We track 35 active open source repositories tagged with #pyspark (500+ stars), totaling 60,070 stars with an average of 1,716 stars per repository. Rankings update daily from the GitHub API.