#pyspark Repositories
Discover 35 active open source GitHub repositories tagged with #pyspark, accumulating 60,070 total stars (averaging 1,716 stars per project). Leading this topic is ibis-project/ibis with 6,705 stars.
35 repositories total
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Lightweight and extensible compatibility layer between dataframe libraries!
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
Public runnable examples of using John Snow Labs' NLP for Apache Spark.
Automatically create Faiss knn indices with the most optimal similarity search parameters.
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. :zap:
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) can be used to generate large simulated/synthetic datasets for testing, POCs, and other uses in Databricks environments, including Delta Live Tables pipelines.
Frequently Asked Questions
What is the most popular #pyspark repository on GitHub?
The top ranked open source #pyspark project on GitHub is ibis-project/ibis with 6,705 stars. the portable Python dataframe library
Which programming languages are most used in #pyspark repositories?
Among the top ranked #pyspark projects on GitHub, the most common programming languages are Python, Scala, Jupyter Notebook, Java.
How is the #pyspark repository ranking calculated?
We track 35 active open source repositories tagged with #pyspark (500+ stars), totaling 60,070 stars with an average of 1,716 stars per repository. Rankings update daily from the GitHub API.