#data-pipeline Repositories
Discover 67 active open source GitHub repositories tagged with #data-pipeline, accumulating 219,024 total stars (averaging 3,269 stars per project). Leading this topic is xai-org/x-algorithm with 34,361 stars.
67 repositories total
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
The RedPajama-Data repository contains code for preparing large datasets for training large language models.
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc.; keep workflows separate from business logic. One binary, no database, runs on limited hardware resources. Alternative to Airflow / Cron / Job Scheduler.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
A list of useful resources to learn Data Engineering from scratch
Open-source inference server and production cluster for all the models your agent needs.
An open-source data logging library for machine learning models and data pipelines. ๐ Provides visibility into data quality & model performance over time. ๐ก๏ธ Supports privacy-preserving data collection, ensuring safety & robustness. ๐
Members of the United States Congress, 1789-Present, in YAML/JSON/CSV, as well as committees, presidents, and vice presidents.
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Meridian cuts through news noise by scraping hundreds of sources, analyzing stories with AI, and delivering concise, personalized daily briefs.
Polymarket Data Retriever that fetches, processes, and structures Polymarket data including markets, order events and trades.
OLake Go: open-source replication from databases, Kafka, and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
Composite Multi-AI Intelligent Agent Stock Team Analysis Watchlist System, based on multiple AI agents, simulates the securities analyst team analysis process, providing comprehensive stock investment analysis and decision suggestions. Newly added: hot money dragon-tiger list tracking analysis, sector alert rotation analysis, supports batch multi-threaded analysis, real-time monitoring of key price points, sending alert messages, reserved miniqmt interface, supports quantitative trading, truly achieves 'AI trades stocks for you', suitable for Mainland China A-shares. For the test website below, please fill in the deepseek temporary key, delete after testing to avoid leakage risk. Note: Excessive test cloud service access may cause data source rejection. For use, please deploy locally.
๐ฅ๐ฅ๐ฅ Open source Reverse ETL - alternative to hightouch and census.
Crawl secwiki and xuanwu.github.io/sec.today, analyze security information sites, security trends, extract security workers' accounts (twitter, weixin, github, etc.)
Open-source ETL/ELT that you deploy on your own servers or in the cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, and MCP for AI agents. No vendor cloud, no per-row billing.
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Kspider is a crawler platform that allows you to define crawler workflows graphically, enabling the implementation of a crawler workflow without code. Kspider is not limited to web crawling, but can also be used for WEB automation testing. More features await your exploration.
Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.
zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka (Data Streams), Databases, and Load scenariosโall defined in simple JSON or YAML, with zero coding.
๐ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐
qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.
SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Census Reporter is a Knight News Challenge-funded project to make it easier for journalists to write stories using information from the U.S. Census bureau.
Practical Data Engineering: A Hands-On Real-Estate Project Guide
The Weibo Hot Search list records popular searches on Weibo starting from November 24, 2020. Data is fetched hourly and archived daily.
Open data of addresses at the town and district level nationwide (277,191 items)
Easily handle day to day CLI operation via Python instead of regular Bash programs. ๐บ๐ฆ #supporting
Analyzing Hacker News discussions from a decade ago in hindsight with LLMs
Frequently Asked Questions
What is the most popular #data-pipeline repository on GitHub?
The top ranked open source #data-pipeline project on GitHub is xai-org/x-algorithm with 34,361 stars. Algorithm powering the For You feed on X
Which programming languages are most used in #data-pipeline repositories?
Among the top ranked #data-pipeline projects on GitHub, the most common programming languages are Python, Java, Go, Rust.
How is the #data-pipeline repository ranking calculated?
We track 67 active open source repositories tagged with #data-pipeline (500+ stars), totaling 219,024 stars with an average of 3,269 stars per repository. Rankings update daily from the GitHub API.