Git Stars Git Stars

#data-deduplication Repositories

Discover 10 active open source GitHub repositories tagged with #data-deduplication, accumulating 20,707 total stars (averaging 2,070 stars per project). Leading this topic is bup/bup with 7,369 stars.

10
Total Repositories
20,707
Total Stars
2,070
Avg Stars/Repo
Programming Languages: Python C Go C#
1
bup/bup

Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).

bup
โญ 7,369 Python ๐Ÿ‘ค bup Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2012-09-03
2
huggingface/datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

huggingface
โญ 3,428 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-06-14
3
adrianlopezroche/fdupes

FDUPES is a program for identifying or deleting duplicate files residing within specified directories.

adrianlopezroche
โญ 3,039 C ๐Ÿ‘ค adrianlopezroche User ๐Ÿ• Updated 2026-04-14 ๐Ÿ“… Created 2014-05-15
4
tomnomnom/anew

A tool for adding new lines to files, skipping duplicates

tomnomnom
โญ 1,599 Go ๐Ÿ‘ค tomnomnom User ๐Ÿ• Updated 2024-01-12 ๐Ÿ“… Created 2019-04-02
5
s3ql/s3ql

a full featured file system for online data storage

s3ql
โญ 1,282 Python ๐Ÿ‘ค s3ql Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-07-12
6
Tarsnap/tarsnap

Command-line client code for Tarsnap.

Tarsnap
โญ 961 C ๐Ÿ‘ค Tarsnap Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2015-05-18
7
OpenDataBox/awesome-data-llm

Official Repository of "LLM ร— DATA" Survey Paper

OpenDataBox
โญ 836 ๐Ÿ‘ค OpenDataBox Organization ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2024-08-04
8
ssdeep-project/ssdeep

Fuzzy hashing API and fuzzy hashing tool

ssdeep-project
โญ 804 C ๐Ÿ‘ค ssdeep-project Organization ๐Ÿ• Updated 2026-08-13 ๐Ÿ“… Created 2017-06-05
9
JakeBayer/FuzzySharp

C# .NET fuzzy string matching implementation of Seat Geek's well known python FuzzyWuzzy algorithm.

JakeBayer
โญ 776 C# ๐Ÿ‘ค JakeBayer User ๐Ÿ• Updated 2024-08-20 ๐Ÿ“… Created 2018-04-10
10
anhaidgroup/deepmatcher

Python package for performing Entity and Text Matching using Deep Learning.

anhaidgroup
โญ 613 Python ๐Ÿ‘ค anhaidgroup User ๐Ÿ• Updated 2024-06-18 ๐Ÿ“… Created 2017-12-01

Frequently Asked Questions

1

What is the most popular #data-deduplication repository on GitHub?

The top ranked open source #data-deduplication project on GitHub is bup/bup with 7,369 stars. Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).

2

Which programming languages are most used in #data-deduplication repositories?

Among the top ranked #data-deduplication projects on GitHub, the most common programming languages are Python, C, Go, C#.

3

How is the #data-deduplication repository ranking calculated?

We track 10 active open source repositories tagged with #data-deduplication (500+ stars), totaling 20,707 stars with an average of 2,070 stars per repository. Rankings update daily from the GitHub API.