Git Stars Git Stars

#data-pipelines Repositories

Discover 50 active open source GitHub repositories tagged with #data-pipelines, accumulating 323,108 total stars (averaging 6,462 stars per project). Leading this topic is twitter/the-algorithm with 73,884 stars.

50
Total Repositories
323,108
Total Stars
6,462
Avg Stars/Repo
1
twitter/the-algorithm

Source code for the X Recommendation Algorithm

twitter
โญ 73,884 Scala ๐Ÿ‘ค twitter Organization ๐Ÿ• Updated 2025-09-08 ๐Ÿ“… Created 2023-03-27
2
pathwaycom/pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

pathwaycom
โญ 61,976 Python ๐Ÿ‘ค pathwaycom Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-11-27
3
apache/airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

apache
โญ 47,583 Python ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-04-13
4
dagster-io/dagster

An orchestration platform for the development, production, and observation of data assets.

dagster-io
โญ 16,453 Python ๐Ÿ‘ค dagster-io Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-04-30
5
Unstructured-IO/unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

Unstructured-IO
โญ 15,769 HTML ๐Ÿ‘ค Unstructured-IO Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2022-09-26
6
apache/dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

apache
โญ 14,559 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-03-01
7
kedro-org/kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

kedro-org
โญ 11,059 Python ๐Ÿ‘ค kedro-org Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2019-04-18
8
mage-ai/mage-ai

๐Ÿง™ Build, run, and manage data pipelines for integrating and transforming data.

mage-ai
โญ 8,850 Python ๐Ÿ‘ค mage-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-05-16
9
OpenDCAI/DataFlow

[SIGMOD'27] Easy Data Preparation with the Latest LLM-Based Operators and Pipelines.

OpenDCAI
โญ 8,650 Python ๐Ÿ‘ค OpenDCAI Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-10-13
10
fluvio-community/fluvio

๐Ÿฆ€ event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

fluvio-community
โญ 5,270 Rust ๐Ÿ‘ค fluvio-community Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2019-08-31
11
gayanvoice/top-github-users

Check your ranking in GitHub! Don't forget to star โญ this repository.

gayanvoice
โญ 4,988 ๐Ÿ‘ค gayanvoice User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-06-03
12
StructuredLabs/preswald

Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

StructuredLabs
โญ 4,266 Python ๐Ÿ‘ค StructuredLabs Organization ๐Ÿ• Updated 2026-06-11 ๐Ÿ“… Created 2024-12-14
13
ucbepic/docetl

A system for agentic LLM-powered data processing and ETL

ucbepic
โญ 4,266 Python ๐Ÿ‘ค ucbepic Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2024-07-09
14
Netflix/maestro

Maestro: Netflixโ€™s Workflow Orchestrator

Netflix
โญ 3,852 Java ๐Ÿ‘ค Netflix Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2024-04-17
15
meltano/meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

meltano
โญ 2,683 Python ๐Ÿ‘ค meltano Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-06-21
16
elementary-data/elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

elementary-data
โญ 2,435 HTML ๐Ÿ‘ค elementary-data Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2021-08-30
17
feldera/feldera

The Feldera Incremental Computation Engine

feldera
โญ 2,181 Rust ๐Ÿ‘ค feldera Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-11
18
data-engineering-community/data-engineering-wiki

The best place to learn data engineering. Built and maintained by the data engineering community.

data-engineering-community
โญ 2,067 CSS ๐Ÿ‘ค data-engineering-community Organization ๐Ÿ• Updated 2026-09-15 ๐Ÿ“… Created 2021-05-04
19
bruin-data/bruin

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

bruin-data
โญ 1,816 Go ๐Ÿ‘ค bruin-data Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-08-03
20
yobix-ai/extractous

Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.

yobix-ai
โญ 1,660 Rust ๐Ÿ‘ค yobix-ai Organization ๐Ÿ• Updated 2024-12-21 ๐Ÿ“… Created 2024-06-04
21
dashbitco/flow

Computational parallel flows on top of GenStage

dashbitco
โญ 1,625 Elixir ๐Ÿ‘ค dashbitco Organization ๐Ÿ• Updated 2026-09-25 ๐Ÿ“… Created 2017-01-18
22
combust/mleap

MLeap: Deploy ML Pipelines to Production

combust
โญ 1,544 Scala ๐Ÿ‘ค combust Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2016-08-23
23
pyper-dev/pyper

Concurrent Python made simple

pyper-dev
โญ 1,521 Python ๐Ÿ‘ค pyper-dev User ๐Ÿ• Updated 2025-02-04 ๐Ÿ“… Created 2024-05-22
24
Netflix/mantis

A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications

Netflix
โญ 1,472 Java ๐Ÿ‘ค Netflix Organization ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2019-06-06
25
opendatadiscovery/odd-platform

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

opendatadiscovery
โญ 1,441 Java ๐Ÿ‘ค opendatadiscovery Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2021-07-07
27
amphi-ai/amphi-etl

visual data prep powered by python

amphi-ai
โญ 1,410 TypeScript ๐Ÿ‘ค amphi-ai Organization ๐Ÿ• Updated 2026-08-24 ๐Ÿ“… Created 2024-03-20
28
gtoonstra/etl-with-airflow

ETL best practices with airflow, with examples

gtoonstra
โญ 1,353 Shell ๐Ÿ‘ค gtoonstra User ๐Ÿ• Updated 2024-09-25 ๐Ÿ“… Created 2016-10-16
29
meta-pytorch/data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

meta-pytorch
โญ 1,266 Python ๐Ÿ‘ค meta-pytorch Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2021-05-12
30
dataform-co/dataform

Dataform is a framework for managing SQL based data operations in BigQuery

dataform-co
โญ 998 TypeScript ๐Ÿ‘ค dataform-co Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-09-03
31
fal-ai/fal

โšก Fastest way to serve open source ML models to millions

fal-ai
โญ 983 Python ๐Ÿ‘ค fal-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-11
32
ocsf/ocsf-schema

OCSF Schema

ocsf
โญ 917 ๐Ÿ‘ค ocsf Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-04-16
33
mafintosh/pump

pipe streams together and close all of them if one of them closes

mafintosh
โญ 917 JavaScript ๐Ÿ‘ค mafintosh User ๐Ÿ• Updated 2026-02-28 ๐Ÿ“… Created 2013-08-14
โญ 895 Ruby ๐Ÿ‘ค uken Organization ๐Ÿ• Updated 2026-09-16 ๐Ÿ“… Created 2013-02-12
35
artie-labs/transfer

Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.

artie-labs
โญ 845 Go ๐Ÿ‘ค artie-labs Organization ๐Ÿ• Updated 2026-05-09 ๐Ÿ“… Created 2022-11-06
36
joboccara/pipes

Pipelines for expressive code on collections in C++

joboccara
โญ 835 C++ ๐Ÿ‘ค joboccara User ๐Ÿ• Updated 2024-10-30 ๐Ÿ“… Created 2017-02-19
37
astronomer/airflow-guides

Guides and docs to help you get up and running with Apache Airflow.

astronomer
โญ 816 JavaScript ๐Ÿ‘ค astronomer Organization ๐Ÿ• Updated 2025-11-15 ๐Ÿ“… Created 2018-05-17
38
tuplex/tuplex

Tuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code. Tuplex has similar Python APIs to Apache Spark or Dask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set.

tuplex
โญ 812 C++ ๐Ÿ‘ค tuplex Organization ๐Ÿ• Updated 2025-08-10 ๐Ÿ“… Created 2021-06-30
39
BasPH/data-pipelines-with-apache-airflow

Code for Data Pipelines with Apache Airflow

BasPH
โญ 810 Python ๐Ÿ‘ค BasPH User ๐Ÿ• Updated 2024-08-15 ๐Ÿ“… Created 2019-06-10
40
turbolytics/sql-flow

DuckDB for streaming data

turbolytics
โญ 808 Go ๐Ÿ‘ค turbolytics User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-11-21
41
raystack/optimus

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

raystack
โญ 757 Go ๐Ÿ‘ค raystack Organization ๐Ÿ• Updated 2024-06-08 ๐Ÿ“… Created 2021-03-22
42
datalab-to/pdftext

Extract structured text from pdfs quickly

datalab-to
โญ 744 Python ๐Ÿ‘ค datalab-to Organization ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2024-04-24
43
digitalocean/firebolt

Golang framework for streaming ETL, observability data pipeline, and event processing apps

digitalocean
โญ 717 Go ๐Ÿ‘ค digitalocean Organization ๐Ÿ• Updated 2024-06-14 ๐Ÿ“… Created 2019-10-31
44
noneback/go-taskflow

A pure go General-purpose Task-parallel Programming Framework

noneback
โญ 642 Go ๐Ÿ‘ค noneback User ๐Ÿ• Updated 2026-08-16 ๐Ÿ“… Created 2024-03-06
45
cgrand/xforms

Extra transducers and reducing fns for Clojure(script)

cgrand
โญ 591 Clojure ๐Ÿ‘ค cgrand User ๐Ÿ• Updated 2024-01-22 ๐Ÿ“… Created 2015-09-03
46
singer-io/singer-python

Writes the Singer format from Python

singer-io
โญ 585 Python ๐Ÿ‘ค singer-io Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2016-10-28
47
apache/flink-playgrounds

Apache Flink Playgrounds

apache
โญ 555 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2019-07-12
48
cas-bigdatalab/piflow

ฯ€flow is a big data flow engine with spark support

cas-bigdatalab
โญ 541 Python ๐Ÿ‘ค cas-bigdatalab Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-05-03
49
elementary-data/dbt-data-reliability

This dbt package captures metadata, artifacts, and test results so you can detect anomalies, monitor data quality, and build metadata tables. It powers Elementary OSS and feeds the wider context layer used by Elementary Cloudโ€™s full Data & AI Control Plane.

elementary-data
โญ 531 Python ๐Ÿ‘ค elementary-data Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-01-09
50
LearningJournal/Spark-Programming-In-Python

Apache Spark 3 - Spark Programming in Python for Beginners

LearningJournal
โญ 510 Python ๐Ÿ‘ค LearningJournal User ๐Ÿ• Updated 2024-07-25 ๐Ÿ“… Created 2020-05-31

Frequently Asked Questions

1

What is the most popular #data-pipelines repository on GitHub?

The top ranked open source #data-pipelines project on GitHub is twitter/the-algorithm with 73,884 stars. Source code for the X Recommendation Algorithm

2

Which programming languages are most used in #data-pipelines repositories?

Among the top ranked #data-pipelines projects on GitHub, the most common programming languages are Python, Go, Java, Rust.

3

How is the #data-pipelines repository ranking calculated?

We track 50 active open source repositories tagged with #data-pipelines (500+ stars), totaling 323,108 stars with an average of 6,462 stars per repository. Rankings update daily from the GitHub API.