Git Stars Git Stars

#deduplication Repositories

Discover 26 active open source GitHub repositories tagged with #deduplication, accumulating 133,156 total stars (averaging 5,121 stars per project). Leading this topic is restic/restic with 37,333 stars.

26
Total Repositories
133,156
Total Stars
5,121
Avg Stars/Repo
1
restic/restic

Fast, secure, efficient backup program

restic
โญ 37,333 Go ๐Ÿ‘ค restic Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2014-04-27
2
kopia/kopia

Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

kopia
โญ 14,766 Go ๐Ÿ‘ค kopia Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-12-19
3
borgbackup/borg

Deduplicating archiver with compression and authenticated encryption.

borgbackup
โญ 13,940 Python ๐Ÿ‘ค borgbackup Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2015-05-12
4
prometheus/alertmanager

Prometheus Alertmanager

prometheus
โญ 8,701 Go ๐Ÿ‘ค prometheus Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2013-07-16
5
arsenetar/dupeguru

Find duplicate files

arsenetar
โญ 7,976 Python ๐Ÿ‘ค arsenetar User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2013-06-22
6
gilbertchen/duplicacy

A new generation cloud backup tool

gilbertchen
โญ 5,707 Go ๐Ÿ‘ค gilbertchen User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2016-02-23
7
togethercomputer/RedPajama-Data

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

togethercomputer
โญ 4,997 Python ๐Ÿ‘ค togethercomputer Organization ๐Ÿ• Updated 2026-06-03 ๐Ÿ“… Created 2023-04-14
8
openvenues/libpostal

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

openvenues
โญ 4,926 C ๐Ÿ‘ค openvenues Organization ๐Ÿ• Updated 2026-05-13 ๐Ÿ“… Created 2015-03-03
9
windirstat/windirstat

Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.

windirstat
โญ 4,592 C++ ๐Ÿ‘ค windirstat Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2016-04-04
10
rustic-rs/rustic

rustic - fast, encrypted, and deduplicated backups powered by Rust

rustic-rs
โญ 3,441 Rust ๐Ÿ‘ค rustic-rs Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2022-03-14
11
Ephemeral-AI-Lab/layerfs

Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind, or merge. Try it with LayerFS ร— DSH.

Ephemeral-AI-Lab
โญ 2,837 Rust ๐Ÿ‘ค Ephemeral-AI-Lab Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-08-10
12
mhx/dwarfs

A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows

mhx
โญ 2,638 C++ ๐Ÿ‘ค mhx User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-11-21
13
moj-analytical-services/splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

moj-analytical-services
โญ 2,592 Python ๐Ÿ‘ค moj-analytical-services Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2019-11-22
14
sahib/rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem

sahib
โญ 2,480 C ๐Ÿ‘ค sahib User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2010-09-27
15
borgmatic-collective/borgmatic

Mirror of borgmatic: Simple, configuration-driven backup software for servers and workstations

borgmatic-collective
โญ 2,371 Python ๐Ÿ‘ค borgmatic-collective Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2014-11-19
16
PlakarKorp/plakar

plakar is a backup solution powered by Kloset and ptar

PlakarKorp
โญ 2,183 Go ๐Ÿ‘ค PlakarKorp Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-03-19
17
cupcakearmy/autorestic

Config driven, easy backup cli for restic.

cupcakearmy
โญ 1,883 Go ๐Ÿ‘ค cupcakearmy User ๐Ÿ• Updated 2026-04-05 ๐Ÿ“… Created 2019-06-20
18
NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

NVIDIA-NeMo
โญ 1,866 Python ๐Ÿ‘ค NVIDIA-NeMo Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-03-14
19
karanhudia/borg-ui

Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.

karanhudia
โญ 1,692 Python ๐Ÿ‘ค karanhudia User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-11
20
zinggAI/zingg

AI-based entity resolution, identity resolution, master data management, and deduplication at scale for humans and agents

zinggAI
โญ 1,275 Java ๐Ÿ‘ค zinggAI Organization ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2021-08-25
21
J535D165/recordlinkage

A powerful and modular toolkit for record linkage and duplicate detection in Python

J535D165
โญ 1,039 Python ๐Ÿ‘ค J535D165 User ๐Ÿ• Updated 2024-02-21 ๐Ÿ“… Created 2015-10-18
22
MinishLab/semhash

Fast Multimodal Semantic Deduplication & Filtering

MinishLab
โญ 979 Python ๐Ÿ‘ค MinishLab Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2024-10-13
23
data-prep-kit/data-prep-kit

Open source project for data preparation for GenAI applications

data-prep-kit
โญ 968 HTML ๐Ÿ‘ค data-prep-kit Organization ๐Ÿ• Updated 2026-09-29 ๐Ÿ“… Created 2024-04-08
24
rin-nas/postgresql-patterns-library

Collection of ready-made SQL queries for PostgreSQL for common tasks (retrieving and modifying data, speeding up queries, database maintenance)

rin-nas
โญ 784 PLpgSQL ๐Ÿ‘ค rin-nas User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2018-12-26
25
Yomguithereal/talisman

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

Yomguithereal
โญ 733 JavaScript ๐Ÿ‘ค Yomguithereal User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2016-03-25
26
fcorbelli/zpaqfranz

Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

fcorbelli
โญ 513 C++ ๐Ÿ‘ค fcorbelli User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-01-20

Frequently Asked Questions

1

What is the most popular #deduplication repository on GitHub?

The top ranked open source #deduplication project on GitHub is restic/restic with 37,333 stars. Fast, secure, efficient backup program

2

Which programming languages are most used in #deduplication repositories?

Among the top ranked #deduplication projects on GitHub, the most common programming languages are Python, Go, C++, C.

3

How is the #deduplication repository ranking calculated?

We track 26 active open source repositories tagged with #deduplication (500+ stars), totaling 133,156 stars with an average of 5,121 stars per repository. Rankings update daily from the GitHub API.