#data-engineering Repositories
152 repositories total
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
The best place to learn data engineering. Built and maintained by the data engineering community.
๐พ nao is the #1 open-source agentic BI. (1) Create context with the nao-core CLI, (2) deploy the nao chat interface for everyone.
The Future of Data Engineering โ A CLI SQL client for the modern data stack, enabling AI-native context engineering for data.
A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.
The materials for the free 30-hour SQL course: the databases for SQL Server, MySQL, and PostgreSQL; every query from the 41 chapters; and the slides.
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
๐ฅ๐ฅ๐ฅ Open source Reverse ETL - alternative to hightouch and census.
ktx is an executable context layer for data and analytics agents ๐ Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company
Scripts and doc for https://www.dolthub.com/repositories/chenditc/investment_data
Open-source ETL/ELT that you deploy on your own servers or in the cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, and MCP for AI agents. No vendor cloud, no per-row billing.
Clean APIs for data cleaning. Python implementation of R package Janitor
A curated collection of 300+ engineering blog articles from top tech companies. Learn how the best engineering teams solve real-world problems at scale.
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.
Dianping Crawler (crawlable on the whole site, solves dynamic font encryption, not OCR). Continuous update
Open-source data contract testing for Snowflake, Databricks, BigQuery, Kafka & 15 more. Built on the Open Data Contract Standard (ODCS). MIT.
A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources
Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.
Public runnable examples of using John Snow Labs' NLP for Apache Spark.
๐ง Convert JSON Objects to MySQL, JSON Schema, Mongoose, Google BigQuery, Swagger, and more.
Course Materials for Analytics in Stock Markets Zoomcamp
๐ Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. ๐