Git Stars Git Stars

#data-pipeline Repositories

Discover 67 active open source GitHub repositories tagged with #data-pipeline, accumulating 219,024 total stars (averaging 3,269 stars per project). Leading this topic is xai-org/x-algorithm with 34,361 stars.

67
Total Repositories
219,024
Total Stars
3,269
Avg Stars/Repo
1
xai-org/x-algorithm

Algorithm powering the For You feed on X

xai-org
โญ 34,361 Rust ๐Ÿ‘ค xai-org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-01-19
2
airbytehq/airbyte

Open-source data movement for ELT pipelines and AI agents โ€” from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

airbytehq
โญ 22,426 Python ๐Ÿ‘ค airbytehq Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-07-27
3
apache/shardingsphere

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

apache
โญ 20,832 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-01-18
4
debezium/debezium

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

debezium
โญ 13,366 Java ๐Ÿ‘ค debezium Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2016-01-22
5
pentaho/pentaho-kettle

Pentaho Data Integration (ETL) a.k.a Kettle

pentaho
โญ 8,419 Java ๐Ÿ‘ค pentaho Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2013-10-08
6
meirwah/awesome-workflow-engines

A curated list of awesome open source workflow engines

meirwah
โญ 7,967 ๐Ÿ‘ค meirwah User ๐Ÿ• Updated 2026-10-11 ๐Ÿ“… Created 2015-05-20
7
datajuicer/data-juicer

Data processing for and with foundation models! ๐ŸŽ ๐Ÿ‹ ๐ŸŒฝ โžก๏ธ โžก๏ธ๐Ÿธ ๐Ÿน ๐Ÿท

datajuicer
โญ 7,266 Python ๐Ÿ‘ค datajuicer Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-08-01
8
snowplow/snowplow

The leader in Customer Data Infrastructure

snowplow
โญ 7,052 Scala ๐Ÿ‘ค snowplow Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2012-03-01
9
apache/flink-cdc

Flink CDC is a streaming data integration tool

apache
โญ 6,495 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-07-27
10
clidey/whodb

Where data access meets operational intelligence

clidey
โญ 5,065 Go ๐Ÿ‘ค clidey Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-06-11
11
togethercomputer/RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

togethercomputer
โญ 4,997 Python ๐Ÿ‘ค togethercomputer Organization ๐Ÿ• Updated 2026-06-03 ๐Ÿ“… Created 2023-04-14
12
dagucloud/dagu

Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc.; keep workflows separate from business logic. One binary, no database, runs on limited hardware resources. Alternative to Airflow / Cron / Job Scheduler.

dagucloud
โญ 4,642 Go ๐Ÿ‘ค dagucloud Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-04-22
13
rudderlabs/rudder-server

Privacy and Security focused Segment-alternative, in Golang and React

rudderlabs
โญ 4,520 Go ๐Ÿ‘ค rudderlabs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-07-19
14
bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

bruin-data
โญ 4,049 Go ๐Ÿ‘ค bruin-data Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-02-12
15
adilkhash/Data-Engineering-HowTo

A list of useful resources to learn Data Engineering from scratch

adilkhash
โญ 4,049 ๐Ÿ‘ค adilkhash User ๐Ÿ• Updated 2026-10-30 ๐Ÿ“… Created 2019-03-28
16
superlinked/sie

Open-source inference server and production cluster for all the models your agent needs.

superlinked
โญ 3,534 Python ๐Ÿ‘ค superlinked Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-07
17
superstreamlabs/memphis

Memphis.dev is a highly scalable and effortless data streaming platform

superstreamlabs
โญ 3,442 Go ๐Ÿ‘ค superstreamlabs Organization ๐Ÿ• Updated 2026-03-02 ๐Ÿ“… Created 2022-02-01
18
whylabs/whylogs

An open-source data logging library for machine learning models and data pipelines. ๐Ÿ“š Provides visibility into data quality & model performance over time. ๐Ÿ›ก๏ธ Supports privacy-preserving data collection, ensuring safety & robustness. ๐Ÿ“ˆ

whylabs
โญ 2,819 Jupyter Notebook ๐Ÿ‘ค whylabs Organization ๐Ÿ• Updated 2025-01-10 ๐Ÿ“… Created 2020-08-14
19
unitedstates/congress-legislators

Members of the United States Congress, 1789-Present, in YAML/JSON/CSV, as well as committees, presidents, and vice presidents.

unitedstates
โญ 2,473 Python ๐Ÿ‘ค unitedstates Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2012-09-27
20
elementary-data/elementary

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

elementary-data
โญ 2,435 HTML ๐Ÿ‘ค elementary-data Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2021-08-30
21
iliane5/meridian

Meridian cuts through news noise by scraping hundreds of sources, analyzing stories with AI, and delivering concise, personalized daily briefs.

iliane5
โญ 2,415 TypeScript ๐Ÿ‘ค iliane5 User ๐Ÿ• Updated 2025-05-30 ๐Ÿ“… Created 2025-03-06
22
warproxxx/poly_data

Polymarket Data Retriever that fetches, processes, and structures Polymarket data including markets, order events and trades.

warproxxx
โญ 2,353 Python ๐Ÿ‘ค warproxxx User ๐Ÿ• Updated 2026-09-08 ๐Ÿ“… Created 2025-10-03
23
reugn/go-streams

A lightweight stream processing library for Go

reugn
โญ 2,178 Go ๐Ÿ‘ค reugn User ๐Ÿ• Updated 2026-01-14 ๐Ÿ“… Created 2019-04-30
24
datazip-inc/olake-go

OLake Go: open-source replication from databases, Kafka, and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

datazip-inc
โญ 2,166 Go ๐Ÿ‘ค datazip-inc Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-10-15
25
oficcejo/aiagents-stock

Composite Multi-AI Intelligent Agent Stock Team Analysis Watchlist System, based on multiple AI agents, simulates the securities analyst team analysis process, providing comprehensive stock investment analysis and decision suggestions. Newly added: hot money dragon-tiger list tracking analysis, sector alert rotation analysis, supports batch multi-threaded analysis, real-time monitoring of key price points, sending alert messages, reserved miniqmt interface, supports quantitative trading, truly achieves 'AI trades stocks for you', suitable for Mainland China A-shares. For the test website below, please fill in the deepseek temporary key, delete after testing to avoid leakage risk. Note: Excessive test cloud service access may cause data source rejection. For use, please deploy locally.

oficcejo
โญ 2,101 Python ๐Ÿ‘ค oficcejo User ๐Ÿ• Updated 2026-10-10 ๐Ÿ“… Created 2025-10-03
26
pydoit/doit

CLI task management & automation tool

pydoit
โญ 2,098 Python ๐Ÿ‘ค pydoit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2014-02-14
27
Multiwoven/multiwoven

๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ Open source Reverse ETL - alternative to hightouch and census.

Multiwoven
โญ 1,685 Ruby ๐Ÿ‘ค Multiwoven Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-10-20
28
bazaarvoice/jolt

JSON to JSON transformation library written in Java.

bazaarvoice
โญ 1,674 Java ๐Ÿ‘ค bazaarvoice Organization ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2013-02-15
29
tanjiti/sec_profile

Crawl secwiki and xuanwu.github.io/sec.today, analyze security information sites, security trends, extract security workers' accounts (twitter, weixin, github, etc.)

tanjiti
โญ 1,640 HTML ๐Ÿ‘ค tanjiti User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-02-19
30
slothflowlabs/duckle

Open-source ETL/ELT that you deploy on your own servers or in the cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, and MCP for AI agents. No vendor cloud, no per-row billing.

slothflowlabs
โญ 1,558 Rust ๐Ÿ‘ค slothflowlabs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-05-21
31
xitongsys/parquet-go

pure golang library for reading/writing parquet file

xitongsys
โญ 1,432 Go ๐Ÿ‘ค xitongsys User ๐Ÿ• Updated 2026-05-08 ๐Ÿ“… Created 2017-09-02
32
GoogleCloudPlatform/data-science-on-gcp

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

GoogleCloudPlatform
โญ 1,432 Jupyter Notebook ๐Ÿ‘ค GoogleCloudPlatform Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2017-02-10
33
weifuwan/seatunnel-web

Making Apache SeaTunnel easier to use.

weifuwan
โญ 1,406 TypeScript ๐Ÿ‘ค weifuwan User ๐Ÿ• Updated 2026-07-16 ๐Ÿ“… Created 2026-02-10
34
kkangert/kspider

Kspider is a crawler platform that allows you to define crawler workflows graphically, enabling the implementation of a crawler workflow without code. Kspider is not limited to web crawling, but can also be used for WEB automation testing. More features await your exploration.

kkangert
โญ 1,285 Java ๐Ÿ‘ค kkangert User ๐Ÿ• Updated 2024-08-19 ๐Ÿ“… Created 2024-07-07
35
DataWithBaraa/sql-data-warehouse-project

Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.

DataWithBaraa
โญ 1,089 TSQL ๐Ÿ‘ค DataWithBaraa User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2024-12-30
36
authorjapps/zerocode

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka (Data Streams), Databases, and Load scenariosโ€”all defined in simple JSON or YAML, with zero coding.

authorjapps
โญ 1,017 Java ๐Ÿ‘ค authorjapps User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2016-07-05
37
estuary/flow

๐ŸŒŠ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐ŸŒŠ

estuary
โญ 995 Rust ๐Ÿ‘ค estuary Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-02-13
38
qiantongtech/qData

qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.

qiantongtech
โญ 950 Java ๐Ÿ‘ค qiantongtech Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-17
39
apache/seatunnel-web

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

apache
โญ 888 Java ๐Ÿ‘ค apache Organization ๐Ÿ• Updated 2026-01-22 ๐Ÿ“… Created 2022-09-26
40
AgnostiqHQ/covalent

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

AgnostiqHQ
โญ 869 Python ๐Ÿ‘ค AgnostiqHQ Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2021-11-03
41
spotify/klio

Smarter data pipelines for audio.

spotify
โญ 866 Python ๐Ÿ‘ค spotify Organization ๐Ÿ• Updated 2024-01-10 ๐Ÿ“… Created 2020-08-07
42
censusreporter/censusreporter

Census Reporter is a Knight News Challenge-funded project to make it easier for journalists to write stories using information from the U.S. Census bureau.

censusreporter
โญ 846 HTML ๐Ÿ‘ค censusreporter Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2013-05-20
43
ssp-data/practical-data-engineering

Practical Data Engineering: A Hands-On Real-Estate Project Guide

ssp-data
โญ 833 Jupyter Notebook ๐Ÿ‘ค ssp-data Organization ๐Ÿ• Updated 2026-06-25 ๐Ÿ“… Created 2020-11-18
44
justjavac/weibo-trending-hot-search

The Weibo Hot Search list records popular searches on Weibo starting from November 24, 2020. Data is fetched hourly and archived daily.

justjavac
โญ 804 TypeScript ๐Ÿ‘ค justjavac User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-11-24
45
tidymodels/infer

An R package for tidyverse-friendly statistical inference

tidymodels
โญ 796 R ๐Ÿ‘ค tidymodels Organization ๐Ÿ• Updated 2026-04-13 ๐Ÿ“… Created 2017-06-05
46
geolonia/japanese-addresses

Open data of addresses at the town and district level nationwide (277,191 items)

geolonia
โญ 774 JavaScript ๐Ÿ‘ค geolonia Organization ๐Ÿ• Updated 2026-08-05 ๐Ÿ“… Created 2020-07-12
47
hudl/statsbombpy

Easily stream StatsBomb data into Python

hudl
โญ 758 Python ๐Ÿ‘ค hudl Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2020-01-08
48
CZ-NIC/pz

Easily handle day to day CLI operation via Python instead of regular Bash programs. ๐Ÿ‡บ๐Ÿ‡ฆ #supporting

CZ-NIC
โญ 732 Python ๐Ÿ‘ค CZ-NIC Organization ๐Ÿ• Updated 2025-11-24 ๐Ÿ“… Created 2020-11-25
49
karpathy/hn-time-capsule

Analyzing Hacker News discussions from a decade ago in hindsight with LLMs

karpathy
โญ 693 Python ๐Ÿ‘ค karpathy User ๐Ÿ• Updated 2025-12-10 ๐Ÿ“… Created 2025-12-10
50
azmatt/chasing_your_tail

Tool for using wireless signals to see if you're being followed.

azmatt
โญ 691 Python ๐Ÿ‘ค azmatt User ๐Ÿ• Updated 2025-02-18 ๐Ÿ“… Created 2022-01-01

Frequently Asked Questions

1

What is the most popular #data-pipeline repository on GitHub?

The top ranked open source #data-pipeline project on GitHub is xai-org/x-algorithm with 34,361 stars. Algorithm powering the For You feed on X

2

Which programming languages are most used in #data-pipeline repositories?

Among the top ranked #data-pipeline projects on GitHub, the most common programming languages are Python, Java, Go, Rust.

3

How is the #data-pipeline repository ranking calculated?

We track 67 active open source repositories tagged with #data-pipeline (500+ stars), totaling 219,024 stars with an average of 3,269 stars per repository. Rankings update daily from the GitHub API.