Git Stars Git Stars

#data-engineering Repositories

152
Total Repositories
727,632
Total Stars
4,787
Avg Stars/Repo
51
apache/hamilton

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

apache
โญ 2,631 Jupyter Notebook ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-02-23
52
sodadata/soda-core

Data Contracts engine for the modern data stack. https://www.soda.io

sodadata
โญ 2,463 Python ๐Ÿ‘ค sodadata Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-12-14
53
metarank/metarank

A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine

metarank
โญ 2,461 Scala ๐Ÿ‘ค metarank Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2020-08-19
54
running-elephant/datart

Datart is a next generation Data Visualization Open Platform

running-elephant
โญ 2,288 TypeScript ๐Ÿ‘ค running-elephant Organization ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2021-10-15
55
bytewax/bytewax

Python Stream Processing

bytewax
โญ 2,074 Python ๐Ÿ‘ค bytewax Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2022-02-04
56
data-engineering-community/data-engineering-wiki

The best place to learn data engineering. Built and maintained by the data engineering community.

data-engineering-community
โญ 2,067 CSS ๐Ÿ‘ค data-engineering-community Organization ๐Ÿ• Updated 2026-09-15 ๐Ÿ“… Created 2021-05-04
57
getnao/nao

๐Ÿ‘พ nao is the #1 open-source agentic BI. (1) Create context with the nao-core CLI, (2) deploy the nao chat interface for everyone.

getnao
โญ 2,034 TypeScript ๐Ÿ‘ค getnao Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-12-30
58
Datus-ai/Datus-agent

The Future of Data Engineering โ€” A CLI SQL client for the modern data stack, enabling AI-native context engineering for data.

Datus-ai
โญ 1,932 Python ๐Ÿ‘ค Datus-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-07-04
59
feathr-ai/feathr

Feathr โ€“ A scalable, unified data and AI engineering platform for enterprise

feathr-ai
โญ 1,916 Scala ๐Ÿ‘ค feathr-ai Organization ๐Ÿ• Updated 2024-04-04 ๐Ÿ“… Created 2022-02-18
60
dbt-labs/dbt-utils

Utility functions for dbt projects.

dbt-labs
โญ 1,837 Makefile ๐Ÿ‘ค dbt-labs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2017-07-16
61
kiwenlau/hadoop-cluster-docker

Run Hadoop Cluster within Docker Containers

kiwenlau
โญ 1,827 Shell ๐Ÿ‘ค kiwenlau User ๐Ÿ• Updated 2024-07-01 ๐Ÿ“… Created 2015-05-21
62
OBenner/data-engineering-interview-questions

More than 2000+ Data engineer interview questions.

OBenner
โญ 1,815 Python ๐Ÿ‘ค OBenner User ๐Ÿ• Updated 2026-01-13 ๐Ÿ“… Created 2021-08-08
63
Hiflylabs/awesome-dbt

A curated list of awesome dbt resources

Hiflylabs
โญ 1,754 ๐Ÿ‘ค Hiflylabs Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2021-10-09
64
benthecoder/yt-channels-DS-AI-ML-CS

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

benthecoder
โญ 1,732 ๐Ÿ‘ค benthecoder User ๐Ÿ• Updated 2026-07-19 ๐Ÿ“… Created 2021-06-14
65
DataWithBaraa/sql-ultimate-course

The materials for the free 30-hour SQL course: the databases for SQL Server, MySQL, and PostgreSQL; every query from the 41 chapters; and the slides.

DataWithBaraa
โญ 1,711 TSQL ๐Ÿ‘ค DataWithBaraa User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2025-03-18
66
mlrun/mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun
โญ 1,703 Python ๐Ÿ‘ค mlrun Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-09-01
67
Multiwoven/multiwoven

๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ Open source Reverse ETL - alternative to hightouch and census.

Multiwoven
โญ 1,685 Ruby ๐Ÿ‘ค Multiwoven Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-10-20
68
Kaelio/ktx

ktx is an executable context layer for data and analytics agents ๐Ÿ™ Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company

Kaelio
โญ 1,633 TypeScript ๐Ÿ‘ค Kaelio Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-05-10
69
kantord/just-dashboard

:bar_chart: :clipboard: Dashboards using YAML or JSON files

kantord
โญ 1,587 TypeScript ๐Ÿ‘ค kantord User ๐Ÿ• Updated 2026-02-21 ๐Ÿ“… Created 2018-01-18
70
quixio/quix-streams

Python Streaming DataFrames for Kafka

quixio
โญ 1,582 Python ๐Ÿ‘ค quixio Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-11-17
71
JeffSackmann/tennis_atp

ATP Tennis Rankings, Results, and Stats

JeffSackmann
โญ 1,577 ๐Ÿ‘ค JeffSackmann User ๐Ÿ• Updated 2026-06-08 ๐Ÿ“… Created 2015-03-13
72
chenditc/investment_data

Scripts and doc for https://www.dolthub.com/repositories/chenditc/investment_data

chenditc
โญ 1,571 Python ๐Ÿ‘ค chenditc User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-07-19
73
slothflowlabs/duckle

Open-source ETL/ELT that you deploy on your own servers or in the cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, and MCP for AI agents. No vendor cloud, no per-row billing.

slothflowlabs
โญ 1,558 Rust ๐Ÿ‘ค slothflowlabs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-05-21
74
duneanalytics/spellbook

SQL views for Dune

duneanalytics
โญ 1,529 Python ๐Ÿ‘ค duneanalytics Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-01-31
75
pyper-dev/pyper

Concurrent Python made simple

pyper-dev
โญ 1,521 Python ๐Ÿ‘ค pyper-dev User ๐Ÿ• Updated 2025-02-04 ๐Ÿ“… Created 2024-05-22
76
pyjanitor-devs/pyjanitor

Clean APIs for data cleaning. Python implementation of R package Janitor

pyjanitor-devs
โญ 1,500 Python ๐Ÿ‘ค pyjanitor-devs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-03-04
77
ashishps1/awesome-engineering-articles

A curated collection of 300+ engineering blog articles from top tech companies. Learn how the best engineering teams solve real-world problems at scale.

ashishps1
โญ 1,491 ๐Ÿ‘ค ashishps1 User ๐Ÿ• Updated 2026-03-01 ๐Ÿ“… Created 2026-02-15
78
lvgalvao/data-engineering-roadmap

Professional Training in Data Engineering and AI

lvgalvao
โญ 1,451 Python ๐Ÿ‘ค lvgalvao User ๐Ÿ• Updated 2026-05-12 ๐Ÿ“… Created 2024-01-16
79
opendatadiscovery/odd-platform

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

opendatadiscovery
โญ 1,441 Java ๐Ÿ‘ค opendatadiscovery Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2021-07-07
80
GoogleCloudPlatform/data-science-on-gcp

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

GoogleCloudPlatform
โญ 1,432 Jupyter Notebook ๐Ÿ‘ค GoogleCloudPlatform Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2017-02-10
81
zapret-info/z-i

Register of Internet Addresses filtered in Russian Federation

zapret-info
โญ 1,388 ๐Ÿ‘ค zapret-info User ๐Ÿ• Updated 2025-10-01 ๐Ÿ“… Created 2012-11-19
82
quiltdata/quilt

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

quiltdata
โญ 1,373 TypeScript ๐Ÿ‘ค quiltdata Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2017-02-10
83
spark-examples/pyspark-examples

Pyspark RDD, DataFrame and Dataset Examples in Python language

spark-examples
โญ 1,371 Python ๐Ÿ‘ค spark-examples Organization ๐Ÿ• Updated 2025-12-07 ๐Ÿ“… Created 2019-11-26
84
gtoonstra/etl-with-airflow

ETL best practices with airflow, with examples

gtoonstra
โญ 1,353 Shell ๐Ÿ‘ค gtoonstra User ๐Ÿ• Updated 2024-09-25 ๐Ÿ“… Created 2016-10-16
85
Sniper970119/dianping_spider

Dianping Crawler (crawlable on the whole site, solves dynamic font encryption, not OCR). Continuous update

Sniper970119
โญ 1,329 Python ๐Ÿ‘ค Sniper970119 User ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2021-01-22
86
Data-Learn/data-engineering

Getting Started with Data Engineering

Data-Learn
โญ 1,323 Python ๐Ÿ‘ค Data-Learn Organization ๐Ÿ• Updated 2025-04-20 ๐Ÿ“… Created 2020-05-14
87
bitol-io/open-data-contract-standard

Home of the Open Data Contract Standard (ODCS).

bitol-io
โญ 1,238 Shell ๐Ÿ‘ค bitol-io Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-06-27
88
abhishek-ch/around-dataengineering

A Data Engineering & Machine Learning Knowledge Hub

abhishek-ch
โญ 1,140 Python ๐Ÿ‘ค abhishek-ch User ๐Ÿ• Updated 2024-02-02 ๐Ÿ“… Created 2020-09-18
89
daochenzha/data-centric-AI

A curated, but incomplete, list of data-centric AI resources.

daochenzha
โญ 1,138 ๐Ÿ‘ค daochenzha User ๐Ÿ• Updated 2024-06-26 ๐Ÿ“… Created 2023-03-07
90
datacontract/datacontract-cli

Open-source data contract testing for Snowflake, Databricks, BigQuery, Kafka & 15 more. Built on the Open Data Contract Standard (ODCS). MIT.

datacontract
โญ 1,114 JavaScript ๐Ÿ‘ค datacontract Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-07-24
91
youngwookim/awesome-hadoop

A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources

youngwookim
โญ 1,113 ๐Ÿ‘ค youngwookim User ๐Ÿ• Updated 2024-05-07 ๐Ÿ“… Created 2014-07-08
92
DataWithBaraa/sql-data-warehouse-project

Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.

DataWithBaraa
โญ 1,089 TSQL ๐Ÿ‘ค DataWithBaraa User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2024-12-30
93
JohnSnowLabs/spark-nlp-workshop

Public runnable examples of using John Snow Labs' NLP for Apache Spark.

JohnSnowLabs
โญ 1,089 Jupyter Notebook ๐Ÿ‘ค JohnSnowLabs Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2018-08-20
94
mattm/sql-style-guide

An opinionated guide for writing clean, maintainable SQL.

mattm
โญ 1,088 ๐Ÿ‘ค mattm User ๐Ÿ• Updated 2025-01-14 ๐Ÿ“… Created 2019-03-27
95
nijikokun/generate-schema

๐Ÿงž Convert JSON Objects to MySQL, JSON Schema, Mongoose, Google BigQuery, Swagger, and more.

nijikokun
โญ 1,067 JavaScript ๐Ÿ‘ค nijikokun User ๐Ÿ• Updated 2024-01-30 ๐Ÿ“… Created 2015-02-16
96
opendatadiscovery/awesome-data-catalogs

๐Ÿ“™ Awesome Data Catalogs and Observability Platforms.

opendatadiscovery
โญ 1,059 ๐Ÿ‘ค opendatadiscovery Organization ๐Ÿ• Updated 2025-08-14 ๐Ÿ“… Created 2021-07-14
97
DataTalksClub/stock-markets-analytics-zoomcamp

Course Materials for Analytics in Stock Markets Zoomcamp

DataTalksClub
โญ 1,052 Jupyter Notebook ๐Ÿ‘ค DataTalksClub Organization ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2024-02-14
โญ 1,043 HTML ๐Ÿ‘ค ajitpal User ๐Ÿ• Updated 2026-04-09 ๐Ÿ“… Created 2015-10-23
99
dataform-co/dataform

Dataform is a framework for managing SQL based data operations in BigQuery

dataform-co
โญ 998 TypeScript ๐Ÿ‘ค dataform-co Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-09-03
100
estuary/flow

๐ŸŒŠ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐ŸŒŠ

estuary
โญ 995 Rust ๐Ÿ‘ค estuary Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-02-13