Git Stars Git Stars

#data-engineering Repositories

152
Total Repositories
727,632
Total Stars
4,787
Avg Stars/Repo
101
latitude-dev/latitude

Developer-first embedded analytics

latitude-dev
โญ 949 TypeScript ๐Ÿ‘ค latitude-dev Organization ๐Ÿ• Updated 2024-09-30 ๐Ÿ“… Created 2024-01-31
102
youssefHosni/Awesome-AI-Data-GitHub-Repos

A collection of the most important Github repos for ML, AI & Data science practitioners

youssefHosni
โญ 946 ๐Ÿ‘ค youssefHosni User ๐Ÿ• Updated 2024-12-24 ๐Ÿ“… Created 2023-05-09
103
odpi/egeria

Project Egeria's core platform providing Open Metadata and Governance support. It has an open type system, distributed knowledge graph, REST APIs, intelligent governance, and pre-built content.

odpi
โญ 925 Java ๐Ÿ‘ค odpi Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2018-05-31
104
natayadev/dataengineering-roadmap

Another repository with basic concepts, technical challenges, and resources on data engineering in Spanish ๐Ÿง™โœจ

natayadev
โญ 919 ๐Ÿ‘ค natayadev User ๐Ÿ• Updated 2026-02-28 ๐Ÿ“… Created 2023-11-24
105
luisguiserrano/manning

Repository for the book Grokking Machine Learning, by Manning Editors

luisguiserrano
โญ 917 Jupyter Notebook ๐Ÿ‘ค luisguiserrano User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2019-03-25
106
pyspark-ai/pyspark-ai

English SDK for Apache Spark

pyspark-ai
โญ 879 Python ๐Ÿ‘ค pyspark-ai Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2023-04-27
107
bacalhau-project/bacalhau

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

bacalhau-project
โญ 878 Go ๐Ÿ‘ค bacalhau-project Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2021-11-05
108
CloudWise-OpenSource/FlyFish

FlyFish is a data visualization coding platform. We can create a data model quickly in a simple way, and quickly generate a set of data visualization solutions by dragging.

CloudWise-OpenSource
โญ 877 JavaScript ๐Ÿ‘ค CloudWise-OpenSource Organization ๐Ÿ• Updated 2024-06-11 ๐Ÿ“… Created 2021-10-08
109
flow-php/flow

The most advanced data processing framework allowing to build scalable data processing pipelines and move data between various data sources and destinations.

flow-php
โญ 875 PHP ๐Ÿ‘ค flow-php Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-05-23
110
NeumTry/NeumAI

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

NeumTry
โญ 864 Python ๐Ÿ‘ค NeumTry Organization ๐Ÿ• Updated 2024-01-15 ๐Ÿ“… Created 2023-09-14
111
frictionlessdata/frictionless-py

Data management framework for Python that provides functionality to describe, extract, validate, and transform tabular data

frictionlessdata
โญ 859 Python ๐Ÿ‘ค frictionlessdata Organization ๐Ÿ• Updated 2026-10-15 ๐Ÿ“… Created 2014-12-23
112
zoltanctoth/complete-dbt-bootcamp-zero-to-hero

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course

zoltanctoth
โญ 842 Python ๐Ÿ‘ค zoltanctoth User ๐Ÿ• Updated 2026-10-05 ๐Ÿ“… Created 2022-02-24
113
ssp-data/practical-data-engineering

Practical Data Engineering: A Hands-On Real-Estate Project Guide

ssp-data
โญ 833 Jupyter Notebook ๐Ÿ‘ค ssp-data Organization ๐Ÿ• Updated 2026-06-25 ๐Ÿ“… Created 2020-11-18
114
AltimateAI/altimate-code

Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered.

AltimateAI
โญ 822 TypeScript ๐Ÿ‘ค AltimateAI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-27
115
Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Nike-Inc
โญ 818 Python ๐Ÿ‘ค Nike-Inc Organization ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2024-05-06
116
astronomer/airflow-guides

Guides and docs to help you get up and running with Apache Airflow.

astronomer
โญ 816 JavaScript ๐Ÿ‘ค astronomer Organization ๐Ÿ• Updated 2025-11-15 ๐Ÿ“… Created 2018-05-17
117
BasPH/data-pipelines-with-apache-airflow

Code for Data Pipelines with Apache Airflow

BasPH
โญ 810 Python ๐Ÿ‘ค BasPH User ๐Ÿ• Updated 2024-08-15 ๐Ÿ“… Created 2019-06-10
118
liuhuanyong/ChainKnowledgeGraph

ChainKnowledgeGraph, the industry chain knowledge graph includes A-share listed companies, industries, and products, a total of 3 types of entities, including industry belonging relationships of listed companies, superior industry relationships, upstream raw material relationships of products, downstream product relationships, main products of companies, and product subcategories, a total of 6 major categories. Listed companies: 4,654, industries: 511, products: 95,559, upstream raw materials: 56,824, superior industries: 480, downstream products: 390, product subcategories: 52,937, belonging industries: 3,946.

liuhuanyong
โญ 779 Python ๐Ÿ‘ค liuhuanyong User ๐Ÿ• Updated 2025-12-22 ๐Ÿ“… Created 2021-10-26
119
microsoft/sqlworkshops

SQL Server Workshops

microsoft
โญ 777 Jupyter Notebook ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2024-05-14 ๐Ÿ“… Created 2019-02-05
120
datavane/datavines

Know your data better๏ผDatavines is Next-gen Data Observability Platform, support metadata manage and data quality.

datavane
โญ 769 Java ๐Ÿ‘ค datavane Organization ๐Ÿ• Updated 2026-07-27 ๐Ÿ“… Created 2022-04-02
121
derar-alhussein/Databricks-Certified-Data-Engineer-Associate

The resources of the preparation course for Databricks Data Engineer Associate certification exam

derar-alhussein
โญ 762 Python ๐Ÿ‘ค derar-alhussein User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2022-11-06
122
unytics/bigfunctions

Supercharge BigQuery with BigFunctions

unytics
โญ 761 Python ๐Ÿ‘ค unytics User ๐Ÿ• Updated 2026-08-23 ๐Ÿ“… Created 2022-08-24
123
defog-ai/sql-eval

Evaluate the accuracy of LLM generated outputs

defog-ai
โญ 744 Jupyter Notebook ๐Ÿ‘ค defog-ai Organization ๐Ÿ• Updated 2025-08-14 ๐Ÿ“… Created 2023-08-10
124
kennethleungty/Failed-ML

Compilation of high-profile real-world examples of failed machine learning projects

kennethleungty
โญ 743 ๐Ÿ‘ค kennethleungty User ๐Ÿ• Updated 2024-06-14 ๐Ÿ“… Created 2022-08-15
125
DataExpert-io/cumulative-table-design

This repository helps teach people how to correctly define and create cumulative tables!

DataExpert-io
โญ 741 Python ๐Ÿ‘ค DataExpert-io Organization ๐Ÿ• Updated 2024-10-29 ๐Ÿ“… Created 2021-12-30
126
databricks/devrel

This repository contains the notebooks and presentations we use for our Databricks Tech Talks "

databricks
โญ 734 HTML ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2025-01-06 ๐Ÿ“… Created 2020-04-03
127
awslabs/aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.

awslabs
โญ 701 Python ๐Ÿ‘ค awslabs Organization ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2017-08-11
128
dgarnitz/vectorflow

VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.

dgarnitz
โญ 698 Python ๐Ÿ‘ค dgarnitz User ๐Ÿ• Updated 2024-05-16 ๐Ÿ“… Created 2023-07-25
129
risesoft-y9/DataFlow-Engine

The data stream engine is a data-driven underlying engine for data integration, data synchronization, data exchange, data sharing, task configuration, and task scheduling. The data stream engine adopts architectures such as control-execution separation, multi-stream layers, and a plugin library to address the actual data problems of large-scale data tasks, high-frequency data reporting, high-frequency data acquisition, and heterogeneous data compatibility.

risesoft-y9
โญ 697 Java ๐Ÿ‘ค risesoft-y9 Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2024-08-26
130
petedannemann/gis-programming-roadmap

One stop shop for all your GIS Programming needs

petedannemann
โญ 686 ๐Ÿ‘ค petedannemann User ๐Ÿ• Updated 2024-01-02 ๐Ÿ“… Created 2019-01-11
131
kaushiksrini/parqeye

Peek inside Parquet files right from your terminal

kaushiksrini
โญ 682 Rust ๐Ÿ‘ค kaushiksrini User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2025-07-02
132
databricks/reference-apps

Spark reference applications

databricks
โญ 652 Scala ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2024-10-03 ๐Ÿ“… Created 2014-08-19
133
JakobGM/patito

A data modelling layer built on top of polars and pydantic

JakobGM
โญ 643 Python ๐Ÿ‘ค JakobGM User ๐Ÿ• Updated 2026-05-08 ๐Ÿ“… Created 2023-06-21
134
rpbouman/huey

Browser-based data analytics & BI with DuckDB-WASM. Privacy-first: your data stays local; no server required.

rpbouman
โญ 639 JavaScript ๐Ÿ‘ค rpbouman User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-12-25
135
google/patents-public-data

Patent analysis using the Google Patents Public Datasets on BigQuery

google
โญ 637 Jupyter Notebook ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2024-06-17 ๐Ÿ“… Created 2017-06-05
136
dbt-labs/dbt-mcp

A MCP (Model Context Protocol) server for interacting with dbt.

dbt-labs
โญ 624 Python ๐Ÿ‘ค dbt-labs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-03-12
137
ConduitIO/conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

ConduitIO
โญ 617 Go ๐Ÿ‘ค ConduitIO Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-01-11
138
synmetrix/synmetrix

Synmetrix โ€“ production-ready open source semantic layer on Cube

synmetrix
โญ 616 JavaScript ๐Ÿ‘ค synmetrix Organization ๐Ÿ• Updated 2025-02-07 ๐Ÿ“… Created 2021-06-18
139
insitro/redun

Yet another redundant workflow engine

insitro
โญ 610 Python ๐Ÿ‘ค insitro Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2021-11-04
140
matsonj/nba-monte-carlo

Monte Carlo simulation of the NBA season, leveraging dbt, duckdb and evidence.dev

matsonj
โญ 609 Python ๐Ÿ‘ค matsonj User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2022-09-29
141
apecloud/myduckserver

Unified MySQL, Postgres & FlightSQL Server, Powered by DuckDB.

apecloud
โญ 595 Go ๐Ÿ‘ค apecloud Organization ๐Ÿ• Updated 2026-09-19 ๐Ÿ“… Created 2024-08-22
142
argilla-io/synthetic-data-generator

Build datasets using natural language

argilla-io
โญ 590 Python ๐Ÿ‘ค argilla-io Organization ๐Ÿ• Updated 2025-09-19 ๐Ÿ“… Created 2024-11-21
143
singer-io/singer-python

Writes the Singer format from Python

singer-io
โญ 585 Python ๐Ÿ‘ค singer-io Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2016-10-28
144
taz4et/bitcoin-etl

A Python toolkit for extracting and transforming Bitcoin blockchain data into structured formats.

taz4et
โญ 580 Python ๐Ÿ‘ค taz4et User ๐Ÿ• Updated 2025-11-24 ๐Ÿ“… Created 2025-11-14
145
spark-examples/spark-scala-examples

This project provides Apache Spark SQL, RDD, DataFrame and Dataset examples in Scala language

spark-examples
โญ 567 Scala ๐Ÿ‘ค spark-examples Organization ๐Ÿ• Updated 2024-03-20 ๐Ÿ“… Created 2019-11-26
146
pracdata/awesome-open-source-data-engineering

A curated list of open source tools used in analytics platforms and data engineering ecosystem

pracdata
โญ 564 ๐Ÿ‘ค pracdata Organization ๐Ÿ• Updated 2025-03-12 ๐Ÿ“… Created 2024-02-22
147
GoogleCloudPlatform/bigquery-oreilly-book

Source code accompanying: BigQuery: The Definitive Guide by Lakshmanan & Tigani to be published by O'Reilly Media

GoogleCloudPlatform
โญ 547 Jupyter Notebook ๐Ÿ‘ค GoogleCloudPlatform Organization ๐Ÿ• Updated 2024-05-02 ๐Ÿ“… Created 2018-11-26
148
kimtth/software-architect-mindmap

๐Ÿง  Mind map of ๐Ÿ—บ๏ธ Software Architecture, Software Engineering

kimtth
โญ 543 Python ๐Ÿ‘ค kimtth User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-02-02
149
kvakvs/beam-wisdoms

Wiki containing pieces of information and knowledge about BEAM Virtual machine (which runs Erlang/OTP and other BEAM-based languages).

kvakvs
โญ 539 ๐Ÿ‘ค kvakvs User ๐Ÿ• Updated 2024-06-10 ๐Ÿ“… Created 2016-08-27
150
silver-dev-org/interview-ready

A repo for interview practice.

silver-dev-org
โญ 529 TypeScript ๐Ÿ‘ค silver-dev-org Organization ๐Ÿ• Updated 2026-02-05 ๐Ÿ“… Created 2023-08-22