Git Stars Git Stars

#data-processing Repositories

Discover 72 active open source GitHub repositories tagged with #data-processing, accumulating 211,976 total stars (averaging 2,944 stars per project). Leading this topic is pathwaycom/pathway with 61,976 stars.

72
Total Repositories
211,976
Total Stars
2,944
Avg Stars/Repo
1
pathwaycom/pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

pathwaycom
โญ 61,976 Python ๐Ÿ‘ค pathwaycom Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-11-27
2
cocoindex-io/cocoindex

Incremental engine for long horizon agents ๐ŸŒŸ Star if you like it!

cocoindex-io
โญ 11,923 Rust ๐Ÿ‘ค cocoindex-io Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-03-03
3
onceupon/Bash-Oneliner

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

onceupon
โญ 10,805 ๐Ÿ‘ค onceupon User ๐Ÿ• Updated 2026-09-22 ๐Ÿ“… Created 2016-06-14
4
johnkerl/miller

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

johnkerl
โญ 10,069 Go ๐Ÿ‘ค johnkerl User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-05-03
5
OpenDCAI/DataFlow

[SIGMOD'27] Easy Data Preparation with the Latest LLM-Based Operators and Pipelines.

OpenDCAI
โญ 8,650 Python ๐Ÿ‘ค OpenDCAI Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-10-13
6
TomWright/dasel

Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.

TomWright
โญ 8,093 Go ๐Ÿ‘ค TomWright User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2020-09-22
7
datajuicer/data-juicer

Data processing for and with foundation models! ๐ŸŽ ๐Ÿ‹ ๐ŸŒฝ โžก๏ธ โžก๏ธ๐Ÿธ ๐Ÿน ๐Ÿท

datajuicer
โญ 7,266 Python ๐Ÿ‘ค datajuicer Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-08-01
8
google/snappy

A fast compressor/decompressor

google
โญ 6,633 C++ ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2014-03-03
9
NVIDIA/DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

NVIDIA
โญ 5,797 C++ ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-06-01
10
deepseek-ai/smallpond

A lightweight data processing framework built on DuckDB and 3FS.

deepseek-ai
โญ 4,960 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-03-05 ๐Ÿ“… Created 2025-02-24
11
unionai-oss/pandera

A light-weight, flexible, and expressive statistical data testing library

unionai-oss
โญ 4,532 Python ๐Ÿ‘ค unionai-oss Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2018-11-01
12
knownsec/aipyapp

AI-Powered Python & Python-Powered AI (Python-Use)

knownsec
โญ 4,026 HTML ๐Ÿ‘ค knownsec Organization ๐Ÿ• Updated 2026-02-15 ๐Ÿ“… Created 2025-04-06
13
Rdatatable/data.table

R's data.table package extends data.frame:

Rdatatable
โญ 3,930 R ๐Ÿ‘ค Rdatatable Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2014-06-07
14
TimelyDataflow/differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.

TimelyDataflow
โญ 3,040 Rust ๐Ÿ‘ค TimelyDataflow Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2015-05-05
15
numaproj/numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs

numaproj
โญ 2,837 Rust ๐Ÿ‘ค numaproj Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2022-05-20
16
datachain-ai/datachain

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

datachain-ai
โญ 2,831 Python ๐Ÿ‘ค datachain-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-06-25
17
elixir-broadway/broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir-broadway
โญ 2,694 Elixir ๐Ÿ‘ค elixir-broadway Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2018-11-05
18
google/googlesql

GoogleSQL (formerly ZetaSQL) - Analyzer Framework for SQL

google
โญ 2,644 C++ ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2019-04-16
19
asyml/texar

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

asyml
โญ 2,391 Python ๐Ÿ‘ค asyml Organization ๐Ÿ• Updated 2026-07-21 ๐Ÿ“… Created 2017-07-22
20
bytewax/bytewax

Python Stream Processing

bytewax
โญ 2,074 Python ๐Ÿ‘ค bytewax Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2022-02-04
21
NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

NVIDIA-NeMo
โญ 1,866 Python ๐Ÿ‘ค NVIDIA-NeMo Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-03-14
22
felt/tippecanoe

Build vector tilesets from large collections of GeoJSON features.

felt
โญ 1,638 C++ ๐Ÿ‘ค felt Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2022-08-15
23
allenai/dolma

Data and tools for generating and inspecting OLMo pre-training data.

allenai
โญ 1,564 Python ๐Ÿ‘ค allenai Organization ๐Ÿ• Updated 2026-08-24 ๐Ÿ“… Created 2023-06-20
24
pyper-dev/pyper

Concurrent Python made simple

pyper-dev
โญ 1,521 Python ๐Ÿ‘ค pyper-dev User ๐Ÿ• Updated 2025-02-04 ๐Ÿ“… Created 2024-05-22
25
GoogleCloudPlatform/data-science-on-gcp

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

GoogleCloudPlatform
โญ 1,432 Jupyter Notebook ๐Ÿ‘ค GoogleCloudPlatform Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2017-02-10
26
zapret-info/z-i

Register of Internet Addresses filtered in Russian Federation

zapret-info
โญ 1,388 ๐Ÿ‘ค zapret-info User ๐Ÿ• Updated 2025-10-01 ๐Ÿ“… Created 2012-11-19
27
ncbi
โญ 1,376 C ๐Ÿ‘ค ncbi Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2014-08-21
28
run-house/kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

run-house
โญ 1,228 Python ๐Ÿ‘ค run-house Organization ๐Ÿ• Updated 2026-05-29 ๐Ÿ“… Created 2022-05-10
29
RadarCODE/awesome-sar

A curated list of awesome Synthetic Aperture Radar (SAR) software, libraries, and resources.

RadarCODE
โญ 1,193 ๐Ÿ‘ค RadarCODE Organization ๐Ÿ• Updated 2025-07-22 ๐Ÿ“… Created 2016-06-03
30
GeospatialPython/pyshp

This library reads and writes ESRI Shapefiles in pure Python.

GeospatialPython
โญ 1,160 Python ๐Ÿ‘ค GeospatialPython User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2014-03-04
31
nikic/iter

Iteration primitives using generators

nikic
โญ 1,145 PHP ๐Ÿ‘ค nikic User ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2013-07-01
32
felixonmars/fcitx5-pinyin-zhwiki

Fcitx 5 Pinyin Dictionary from zh.wikipedia.org

felixonmars
โญ 1,108 Python ๐Ÿ‘ค felixonmars User ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2020-05-23
33
tadija/AEXML

Swift minion for simple and lightweight XML parsing

tadija
โญ 1,037 Swift ๐Ÿ‘ค tadija User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2014-10-16
34
google-deepmind/tree

tree is a library for working with nested data structures

google-deepmind
โญ 1,027 Python ๐Ÿ‘ค google-deepmind Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2019-10-08
35
thepycoach/automation

Automation with Python (Excel, scripts, etc)

thepycoach
โญ 1,010 Jupyter Notebook ๐Ÿ‘ค thepycoach User ๐Ÿ• Updated 2024-05-19 ๐Ÿ“… Created 2021-05-02
36
negokaz/excel-mcp-server

A Model Context Protocol (MCP) server that reads and writes MS Excel data

negokaz
โญ 991 Go ๐Ÿ‘ค negokaz User ๐Ÿ• Updated 2025-07-19 ๐Ÿ“… Created 2025-02-16
37
fireducks-dev/fireducks

Create an issue on FireDucks

fireducks-dev
โญ 929 Jupyter Notebook ๐Ÿ‘ค fireducks-dev Organization ๐Ÿ• Updated 2025-05-22 ๐Ÿ“… Created 2023-10-12
38
samtools/bcftools

This is the official development repository for BCFtools. See installation instructions and other documentation here http://samtools.github.io/bcftools/howtos/install.html

samtools
โญ 905 C ๐Ÿ‘ค samtools Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2013-07-12
39
bastibe/python-soundfile

SoundFile is an audio library based on libsndfile, CFFI, and NumPy

bastibe
โญ 872 Python ๐Ÿ‘ค bastibe User ๐Ÿ• Updated 2026-10-24 ๐Ÿ“… Created 2013-08-27
40
TeamMsgExtractor/msg-extractor

Extracts emails and attachments saved in Microsoft Outlook's .msg files

TeamMsgExtractor
โญ 858 Python ๐Ÿ‘ค TeamMsgExtractor Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2013-07-24
41
MohElshamy1994/n8n-course-Projetcs

n8n Projects for my YouTube course.

MohElshamy1994
โญ 854 ๐Ÿ‘ค MohElshamy1994 User ๐Ÿ• Updated 2025-10-19 ๐Ÿ“… Created 2025-07-13
42
0xfdf/toraniko

A multi-factor equity risk model for quantitative trading.

0xfdf
โญ 853 Python ๐Ÿ‘ค 0xfdf User ๐Ÿ• Updated 2024-08-16 ๐Ÿ“… Created 2024-07-03
43
benibela/xidel

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

benibela
โญ 836 Pascal ๐Ÿ‘ค benibela User ๐Ÿ• Updated 2025-02-22 ๐Ÿ“… Created 2015-06-11
44
bugen/pypipe

Python pipe command line tool

bugen
โญ 802 Python ๐Ÿ‘ค bugen User ๐Ÿ• Updated 2024-01-04 ๐Ÿ“… Created 2023-09-20
45
influxdata/flux

Flux is a lightweight scripting language for querying databases (like InfluxDB) and working with data. It's part of InfluxDB 1.7 and 2.0, but can be run independently of those.

influxdata
โญ 794 FLUX ๐Ÿ‘ค influxdata Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2018-06-14
46
PreSenseRadar/OpenRadar

An open source library for interacting with and processing radar data, specialized for MIMO mmWave radars

PreSenseRadar
โญ 776 Jupyter Notebook ๐Ÿ‘ค PreSenseRadar User ๐Ÿ• Updated 2024-04-30 ๐Ÿ“… Created 2019-09-20
47
ChenghaoMou/text-dedup

All-in-one text de-duplication

ChenghaoMou
โญ 765 Python ๐Ÿ‘ค ChenghaoMou User ๐Ÿ• Updated 2026-03-09 ๐Ÿ“… Created 2021-03-13
48
JuliaParallel/Dagger.jl

A framework for out-of-core and parallel execution

JuliaParallel
โญ 734 Julia ๐Ÿ‘ค JuliaParallel Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2015-12-03
49
hstreamdb/hstream

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

hstreamdb
โญ 728 Haskell ๐Ÿ‘ค hstreamdb Organization ๐Ÿ• Updated 2024-12-26 ๐Ÿ“… Created 2020-08-31
50
omarabid/rust-companies

A list of companies using Rust in production.

omarabid
โญ 721 ๐Ÿ‘ค omarabid User ๐Ÿ• Updated 2026-09-11 ๐Ÿ“… Created 2020-02-05

Frequently Asked Questions

1

What is the most popular #data-processing repository on GitHub?

The top ranked open source #data-processing project on GitHub is pathwaycom/pathway with 61,976 stars. Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

2

Which programming languages are most used in #data-processing repositories?

Among the top ranked #data-processing projects on GitHub, the most common programming languages are Python, C++, Jupyter Notebook, Rust.

3

How is the #data-processing repository ranking calculated?

We track 72 active open source repositories tagged with #data-processing (500+ stars), totaling 211,976 stars with an average of 2,944 stars per repository. Rankings update daily from the GitHub API.