#data-processing Repositories
Discover 72 active open source GitHub repositories tagged with #data-processing, accumulating 211,976 total stars (averaging 2,944 stars per project). Leading this topic is pathwaycom/pathway with 61,976 stars.
72 repositories total
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
[SIGMOD'27] Easy Data Preparation with the Latest LLM-Based Operators and Pipelines.
Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
An implementation of differential dataflow using timely dataflow on Rust.
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Concurrent and multi-stage data ingestion and data processing with Elixir
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
A curated list of awesome Synthetic Aperture Radar (SAR) software, libraries, and resources.
A Model Context Protocol (MCP) server that reads and writes MS Excel data
This is the official development repository for BCFtools. See installation instructions and other documentation here http://samtools.github.io/bcftools/howtos/install.html
Extracts emails and attachments saved in Microsoft Outlook's .msg files
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Flux is a lightweight scripting language for querying databases (like InfluxDB) and working with data. It's part of InfluxDB 1.7 and 2.0, but can be run independently of those.
An open source library for interacting with and processing radar data, specialized for MIMO mmWave radars
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Frequently Asked Questions
What is the most popular #data-processing repository on GitHub?
The top ranked open source #data-processing project on GitHub is pathwaycom/pathway with 61,976 stars. Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Which programming languages are most used in #data-processing repositories?
Among the top ranked #data-processing projects on GitHub, the most common programming languages are Python, C++, Jupyter Notebook, Rust.
How is the #data-processing repository ranking calculated?
We track 72 active open source repositories tagged with #data-processing (500+ stars), totaling 211,976 stars with an average of 2,944 stars per repository. Rankings update daily from the GitHub API.