#data-engineering Repositories
Discover 152 active open source GitHub repositories tagged with #data-engineering, accumulating 727,632 total stars (averaging 4,787 stars per project). Leading this topic is apache/superset with 75,603 stars.
152 repositories total
Learn how to develop, deploy and iterate on production-grade ML applications.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐๐ผ
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.
๐ Papers & tech blogs by companies sharing their work on data science & machine learning in production.
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
An orchestration platform for the development, production, and observation of data assets.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
๐ง Build, run, and manage data pipelines for integrating and transforming data.
LLM-Driven Extraction of Unstructured Data โ Built for API Deployments & ETL Pipeline Workflows
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
data load tool (dlt) is an open source Python library that makes data loading easy ๐ ๏ธ
DataX integrated visualization page: Select a data source to generate data synchronization tasks with one click. Supports data sources such as RDBMS, Hive, HBase, ClickHouse, MongoDB, etc. Batch creation of RDBMS data synchronization tasks, integration with open-source scheduling systems, support for distributed processing, incremental data synchronization, real-time viewing of running logs, monitoring executor resources, KILL running processes, data source information encryption, and more.
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Summaries and resources for Designing Machine Learning Systems book (Chip Huyen, O'Reilly 2022)
DeepAnalyze is the first agentic LLM for autonomous data science. ๐ Your AI data analyst automatically analyzes massive amounts of data and generates professional analysis reports with a single click!
SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
A list of useful resources to learn Data Engineering from scratch
A list of many interview questions & real-world challenges in Tech! (https://tech-vault-web.vercel.app/)
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Learn how to design, develop, deploy and iterate on production-grade ML applications.
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.
A new SOTA for RAG โ an original retrieval architecture and an open-source knowledge base for humans and agents.
An Awesome List of Open-Source Data Engineering Projects
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Frequently Asked Questions
What is the most popular #data-engineering repository on GitHub?
The top ranked open source #data-engineering project on GitHub is apache/superset with 75,603 stars. Apache Superset is a Data Visualization and Data Exploration Platform
Which programming languages are most used in #data-engineering repositories?
Among the top ranked #data-engineering projects on GitHub, the most common programming languages are Python, Go, Rust, Jupyter Notebook.
How is the #data-engineering repository ranking calculated?
We track 152 active open source repositories tagged with #data-engineering (500+ stars), totaling 727,632 stars with an average of 4,787 stars per repository. Rankings update daily from the GitHub API.