#bigdata Repositories
Discover 51 active open source GitHub repositories tagged with #bigdata, accumulating 261,861 total stars (averaging 5,134 stars per project). Leading this topic is DataExpert-io/data-engineer-handbook with 45,237 stars.
51 repositories total
This is a repo with links to everything you'd ever want to learn about data engineering
RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
A curated list of awesome big data frameworks, ressources and other awesomeness.
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. โ rebuilt from scratch. Unified architecture on your S3.
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second ๐
100+ sets of Big Data visualization cool large screen Html5 templates; Included industries: community, property management, government affairs, transportation, financial banking, etc., the latest, most numerous, most complete, coolest, most dazzling Big Data visualization templates on the whole network. Gradually updating.
๐จ Generate structured SQL statements using JSON, implemented with Vue3 + TypeScript + Vite + Ant Design + MonacoEditor, the project is simple (heavy logic, light pages), suitable for practice~
Big Data Learning, Learn Big Data from Scratch, Includes Learning Videos and Interview Materials for Each Stage of Big Data Learning
GridDB is a next-generation open source database that makes time series IoT and big data fast and easy.
Aix-DB, based on the LangChain/LangGraph framework and combined with the MCP Skills multi-agent collaboration architecture, accomplishes end-to-end conversion from natural language to data insights.
.NET for Apacheยฎ Sparkโข makes Apache Sparkโข easily accessible to .NET developers.
Based on the open-source Flink, it extends its real-time SQL; mainly implements the join between streams and dimension tables, supporting all syntax of native Flink SQL
Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.
The Big Data knowledge repository involves data warehouse modeling, real-time computing, Big Data, data middle platform, system design, Java, algorithms, etc.
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
GUI-based Python code generator for data science, extension to Jupyter Lab, Jupyter Notebook and Google Colab.
The Lazy Squirrel Flink-Boot scaffold allows Flink to fully embrace the Spring ecosystem, enabling developers to develop distributed running stream processing programs with the Java WEB development mode. Lazy Squirrel makes crossover simpler. Lazy Squirrel aims to enable developers to quickly write and implement business code with lower entry costs (without needing to understand the theory of distributed computing and the details of the Flink framework). To further enhance the agility of developers using the Lazy Squirrel scaffold to develop large projects, the scaffold defaults to integrating the Spring framework for Bean management, while also integrating frameworks commonly used in the microservices and WEB development fields, further improving development speed. For example, integrating the Mybatis ORM framework, Hibernate Validator validation framework, Spring Retry retry framework, etc. See the scaffold features below for details.
DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.
Visual dashboard solution, providing a visual editing engine to help individuals or businesses easily customize their own visual dashboard applications.
Embedded transactional database for blobs, documents & graphs, with kernel-bypass NVMe through io_uring & SPDK, GPU compaction, and one C ABI over its own, RocksDB & LevelDB engines ๐๏ธ
Datafaker is a large-scale test data and flow test data generation tool. Datafaker fakes data and inserts to varied data sources. Test Data Generation Tool
The open source knowledgehouse for your company. Marmot knows what your data means, governs who may use it and runs every query your people and agents make, checked and recorded. Catalog, lineage, governed SQL and an MCP server in one Go binary.
High performance distributed S3 compatible object storage focused on speed and designed to be an open alternative to MinIO and RustFS.
TOMs is a fully open-source, high-performance, systematic, plugin-oriented, and scenario-agnostic general-purpose development framework.
A simple spark standalone cluster for your testing environment purposes
Big Data Collection, Extraction Platform, zdh_web is the visual management platform for zdh series services, including modules such as data collection, scheduling, permissions, approval workflow, private domain marketing, etc.
Frequently Asked Questions
What is the most popular #bigdata repository on GitHub?
The top ranked open source #bigdata project on GitHub is DataExpert-io/data-engineer-handbook with 45,237 stars. This is a repo with links to everything you'd ever want to learn about data engineering
Which programming languages are most used in #bigdata repositories?
Among the top ranked #bigdata projects on GitHub, the most common programming languages are Java, Go, Python, Scala.
How is the #bigdata repository ranking calculated?
We track 51 active open source repositories tagged with #bigdata (500+ stars), totaling 261,861 stars with an average of 5,134 stars per repository. Rankings update daily from the GitHub API.