#spark Repositories
Discover 126 active open source GitHub repositories tagged with #spark, accumulating 513,269 total stars (averaging 4,073 stars per project). Leading this topic is DataTalksClub/data-engineering-zoomcamp with 46,997 stars.
126 repositories total
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐๐ผ
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Latest Docker container technology, learn best practices from real cases! | Learn and understand Docker & Container technologies, with real DevOps practice!
GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.
Flink learning blog. http://www.54tianzhisheng.cn/ Contains content on Flink introduction, concepts, principles, practical implementation, performance tuning, source code analysis, etc. Involves learning cases on Flink Connector, Metrics, Library, DataStream API, Table API & SQL, etc., and sharing of large project cases for Flink production applications (PVUV, log storage, real-time deduplication of billions of data, monitoring alerts). Welcome everyone to support my column "Big Data Real-time Computing Engine Flink Practical and Performance Optimization".
ใBig Tech Interview Columnใ A technical guide for Java programmers, featuring interview questions, system architecture, career tips, mainstream middleware, and more, to help you become a better version of yourself!
Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
๐ง Build, run, and manage data pipelines for integrating and transforming data.
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
cube studio Open-Source Cloud-Native All-in-One Machine Learning/Deep Learning/Large Model AI Platform, MLOps algorithm link full process, computing power rental platform, notebook online development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference service VGPU virtualization, edge computing, annotation platform automated annotation, deepseek and other large models sft fine-tuning/reward model/reinforcement learning training, vllm/ollama/mindie large model multi-machine inference, private knowledge base, AI model market, supports domestic CPU/GPU/NPU Ascend ecosystem, supports RDMA, supports distributed training such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano
A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
๐จ Generate structured SQL statements using JSON, implemented with Vue3 + TypeScript + Vite + Ant Design + MonacoEditor, the project is simple (heavy logic, light pages), suitable for practice~
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
An open lakehouse engine for agents โ Postgres-native, vector-native, and Python-native. Speaks PostgreSQL, runs on your object storage, written in Rust.
Big Data Learning, Learn Big Data from Scratch, Includes Learning Videos and Interview Materials for Each Stage of Big Data Learning
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Historical learning project: Spark movie recommendations, web scraping, web app, and backend. Ongoing development of the new version: cinematch-recommender.
:herb: Quick learning examples based on Spring Boot, integrating open-source frameworks I've encountered, such as: RabbitMQ (delay queue), Kafka, JPA, Redis, OAuth2, Swagger, JSP, Docker, K3s, K3d, K8s, MyBatis encryption/decryption plugin, exception handling, log output, multi-module development, multi-environment packaging, cache, web crawler, JWT, GraphQL, Dubbo, Zookeeper, and Async, etc. :pushpin:
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
โจSpark is a web-based, cross-platform and full-featured Remote Administration Tool (RAT) written in Go that allows you control all your devices anywhere. Spark is a Go-written, web-based, cross-platform and feature-rich remote control and monitoring tool that allows you to monitor and control all devices anytime and anywhere.
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
TransmogrifAI (pronounced trฤns-mลgหrษ-fฤซ) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
.NET for Apacheยฎ Sparkโข makes Apache Sparkโข easily accessible to .NET developers.
Frequently Asked Questions
What is the most popular #spark repository on GitHub?
The top ranked open source #spark project on GitHub is DataTalksClub/data-engineering-zoomcamp with 46,997 stars. Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐๐ผ
Which programming languages are most used in #spark repositories?
Among the top ranked #spark projects on GitHub, the most common programming languages are Java, Scala, Python, Jupyter Notebook.
How is the #spark repository ranking calculated?
We track 126 active open source repositories tagged with #spark (500+ stars), totaling 513,269 stars with an average of 4,073 stars per repository. Rankings update daily from the GitHub API.