#data-engineering Repositories
152 repositories total
A collection of the most important Github repos for ML, AI & Data science practitioners
Project Egeria's core platform providing Open Metadata and Governance support. It has an open type system, distributed knowledge graph, REST APIs, intelligent governance, and pre-built content.
Another repository with basic concepts, technical challenges, and resources on data engineering in Spanish ๐งโจ
Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.
FlyFish is a data visualization coding platform. We can create a data model quickly in a simple way, and quickly generate a set of data visualization solutions by dragging.
The most advanced data processing framework allowing to build scalable data processing pipelines and move data between various data sources and destinations.
Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.
Data management framework for Python that provides functionality to describe, extract, validate, and transform tabular data
Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course
Practical Data Engineering: A Hands-On Real-Estate Project Guide
Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered.
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
ChainKnowledgeGraph, the industry chain knowledge graph includes A-share listed companies, industries, and products, a total of 3 types of entities, including industry belonging relationships of listed companies, superior industry relationships, upstream raw material relationships of products, downstream product relationships, main products of companies, and product subcategories, a total of 6 major categories. Listed companies: 4,654, industries: 511, products: 95,559, upstream raw materials: 56,824, superior industries: 480, downstream products: 390, product subcategories: 52,937, belonging industries: 3,946.
Know your data better๏ผDatavines is Next-gen Data Observability Platform, support metadata manage and data quality.
The resources of the preparation course for Databricks Data Engineer Associate certification exam
Compilation of high-profile real-world examples of failed machine learning projects
This repository helps teach people how to correctly define and create cumulative tables!
This repository contains the notebooks and presentations we use for our Databricks Tech Talks "
AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.
The data stream engine is a data-driven underlying engine for data integration, data synchronization, data exchange, data sharing, task configuration, and task scheduling. The data stream engine adopts architectures such as control-execution separation, multi-stream layers, and a plugin library to address the actual data problems of large-scale data tasks, high-frequency data reporting, high-frequency data acquisition, and heterogeneous data compatibility.
Browser-based data analytics & BI with DuckDB-WASM. Privacy-first: your data stays local; no server required.
Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
Monte Carlo simulation of the NBA season, leveraging dbt, duckdb and evidence.dev
A Python toolkit for extracting and transforming Bitcoin blockchain data into structured formats.
This project provides Apache Spark SQL, RDD, DataFrame and Dataset examples in Scala language
A curated list of open source tools used in analytics platforms and data engineering ecosystem
Source code accompanying: BigQuery: The Definitive Guide by Lakshmanan & Tigani to be published by O'Reilly Media
๐ง Mind map of ๐บ๏ธ Software Architecture, Software Engineering
Wiki containing pieces of information and knowledge about BEAM Virtual machine (which runs Erlang/OTP and other BEAM-based languages).