Git Stars Git Stars

#distributed-training Repositories

147
Total Repositories
615,835
Total Stars
4,189
Avg Stars/Repo
51
google/tunix

A Lightweight LLM Post-Training Library

google
โญ 2,516 Python ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-02
52
Unakar/Logic-RL

Reproduce R1 Zero on Logic Puzzle

Unakar
โญ 2,451 Python ๐Ÿ‘ค Unakar User ๐Ÿ• Updated 2025-03-20 ๐Ÿ“… Created 2025-02-02
53
aigc-apps/VideoX-Fun

๐Ÿ“น A more flexible framework that can generate videos at any resolution and creates videos from images.

aigc-apps
โญ 2,337 Python ๐Ÿ‘ค aigc-apps Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-09-10
54
AmberLJC/LLMSys-PaperList

Large Language Model (LLM) Systems Paper List

AmberLJC
โญ 2,332 Python ๐Ÿ‘ค AmberLJC User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2023-06-06
55
ByteDance-Seed/VeOmni

VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

ByteDance-Seed
โญ 2,299 Python ๐Ÿ‘ค ByteDance-Seed Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-03-28
56
PrimeIntellect-ai/prime-rl

Agentic RL Training at Scale

PrimeIntellect-ai
โญ 2,268 Python ๐Ÿ‘ค PrimeIntellect-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-02-18
57
cfregly/ai-performance-engineering

Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and full-stack tuning.

cfregly
โญ 2,233 Python ๐Ÿ‘ค cfregly User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-04-21
58
NVIDIA-NeMo/RL

Scalable toolkit for efficient model reinforcement

NVIDIA-NeMo
โญ 2,146 Python ๐Ÿ‘ค NVIDIA-NeMo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-03-16
59
km1994/LLMsNineStoryDemonTower

ใ€LLMs Nine-Layered Demon Towerใ€‘Sharing LLMs practical experiences and knowledge in fields such as Natural Language Processing (ChatGLM, Chinese-LLaMA-Alpaca, Xiaoyangtuo Vicuna, LLaMA, GPT4ALL, etc.), Information Retrieval (langchain), Language Synthesis, Language Recognition, Multimodal (Stable Diffusion, MiniGPT-4, VisualGLM-6B, Ziya-Visual, etc.).

km1994
โญ 2,140 ๐Ÿ‘ค km1994 User ๐Ÿ• Updated 2024-03-30 ๐Ÿ“… Created 2023-05-01
60
epfLLM/meditron

Meditron is a suite of open-source medical Large Language Models (LLMs).

epfLLM
โญ 2,125 Python ๐Ÿ‘ค epfLLM Organization ๐Ÿ• Updated 2024-04-10 ๐Ÿ“… Created 2023-11-23
61
aceliuchanghong/FAQ_Of_LLM_Interview

Large Model Algorithm Position Interview Questions (with Answers): Common Questions and Concept Analysis "Large Model Interview Questions", "Algorithm Position Interview", "Common Interview Questions", "Large Model Algorithm Interview", "Large Model Application Fundamentals"

aceliuchanghong
โญ 2,064 Jupyter Notebook ๐Ÿ‘ค aceliuchanghong User ๐Ÿ• Updated 2026-09-17 ๐Ÿ“… Created 2024-03-13
62
NUS-HPC-AI-Lab/VideoSys

VideoSys: An easy and efficient system for video generation

NUS-HPC-AI-Lab
โญ 2,022 Python ๐Ÿ‘ค NUS-HPC-AI-Lab Organization ๐Ÿ• Updated 2025-08-27 ๐Ÿ“… Created 2024-02-17
63
LTH14/mar

PyTorch implementation of MAR+DiffLoss https://arxiv.org/abs/2406.11838

LTH14
โญ 1,956 Python ๐Ÿ‘ค LTH14 User ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2024-07-24
64
flexflow/flexflow-train

Automatically Discovering Fast Parallelization Strategies for Distributed Deep Neural Network Training

flexflow
โญ 1,900 C++ ๐Ÿ‘ค flexflow Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-12-08
65
laekov/fastmoe

A fast MoE impl for PyTorch

laekov
โญ 1,853 Python ๐Ÿ‘ค laekov User ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2021-01-25
โญ 1,837 Python ๐Ÿ‘ค ytongbai User ๐Ÿ• Updated 2024-06-28 ๐Ÿ“… Created 2023-04-05
67
allenai/Olmo-core

PyTorch building blocks for the OLMo ecosystem

allenai
โญ 1,810 Python ๐Ÿ‘ค allenai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-03-23
68
facebookresearch/coconut

Training Large Language Model to Reason in a Continuous Latent Space

facebookresearch
โญ 1,752 Python ๐Ÿ‘ค facebookresearch Organization ๐Ÿ• Updated 2026-07-02 ๐Ÿ“… Created 2025-01-15
โญ 1,711 Cuda ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-10-05 ๐Ÿ“… Created 2017-08-08
70
intelligent-machine-learning/dlrover

DLRover: An Automatic Distributed Deep Learning System

intelligent-machine-learning
โญ 1,695 Python ๐Ÿ‘ค intelligent-machine-learning Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2022-06-24
71
gensyn-ai/rl-swarm

A fully open source framework for creating RL training swarms over the internet.

gensyn-ai
โญ 1,673 Python ๐Ÿ‘ค gensyn-ai Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2025-02-25
72
tczhangzhi/pytorch-distributed

A quickstart and benchmark for pytorch distributed training.

tczhangzhi
โญ 1,665 Python ๐Ÿ‘ค tczhangzhi User ๐Ÿ• Updated 2024-07-25 ๐Ÿ“… Created 2019-12-20
73
XueFuzhao/OpenMoE

A family of open-sourced Mixture-of-Experts (MoE) Large Language Models

XueFuzhao
โญ 1,648 Python ๐Ÿ‘ค XueFuzhao User ๐Ÿ• Updated 2024-03-08 ๐Ÿ“… Created 2023-08-08
74
alibaba/Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.

alibaba
โญ 1,601 Python ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2025-12-15 ๐Ÿ“… Created 2023-09-04
75
EvolvingLMMs-Lab/open-r1-multimodal

A fork to add multimodal model training to open-r1

EvolvingLMMs-Lab
โญ 1,556 Python ๐Ÿ‘ค EvolvingLMMs-Lab Organization ๐Ÿ• Updated 2025-02-08 ๐Ÿ“… Created 2025-01-27
76
AnswerDotAI/fsdp_qlora

Training LLMs with QLoRA + FSDP

AnswerDotAI
โญ 1,534 Jupyter Notebook ๐Ÿ‘ค AnswerDotAI Organization ๐Ÿ• Updated 2024-11-09 ๐Ÿ“… Created 2024-01-14
77
Tencent/TFace

A trusty face analysis research platform developed by Tencent Youtu Lab

Tencent
โญ 1,524 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2025-12-11 ๐Ÿ“… Created 2021-05-20
78
MoonshotAI/Moonlight

Muon is Scalable for LLM Training

MoonshotAI
โญ 1,523 ๐Ÿ‘ค MoonshotAI Organization ๐Ÿ• Updated 2025-08-03 ๐Ÿ“… Created 2025-02-22
79
fastnlp/fitlog

fitlog is a tool used in deep learning training to assist users in logging and managing code.

fastnlp
โญ 1,517 Python ๐Ÿ‘ค fastnlp Organization ๐Ÿ• Updated 2024-01-16 ๐Ÿ“… Created 2019-04-03
80
Lightning-AI/lightning-thunder

PyTorch compiler that accelerates training and inference. Get built-in optimizations for performance, memory, parallelism, and easily write your own.

Lightning-AI
โญ 1,476 Python ๐Ÿ‘ค Lightning-AI Organization ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2024-03-18
81
pytorch/gloo

Collective communications library with various primitives for multi-machine training.

pytorch
โญ 1,469 C++ ๐Ÿ‘ค pytorch Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2017-02-03
82
kubeflow/examples

A repository to host extended examples and tutorials

kubeflow
โญ 1,463 Jsonnet ๐Ÿ‘ค kubeflow Organization ๐Ÿ• Updated 2025-04-14 ๐Ÿ“… Created 2018-02-01
โญ 1,368 ๐Ÿ‘ค meituan-longcat Organization ๐Ÿ• Updated 2026-06-23 ๐Ÿ“… Created 2025-08-30
85
mapillary/inplace_abn

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

mapillary
โญ 1,331 Python ๐Ÿ‘ค mapillary Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2017-11-23
โญ 1,323 Python ๐Ÿ‘ค ChenmienTan User ๐Ÿ• Updated 2026-05-20 ๐Ÿ“… Created 2025-04-04
87
XueFuzhao/awesome-mixture-of-experts

A collection of AWESOME things about mixture-of-experts

XueFuzhao
โญ 1,247 ๐Ÿ‘ค XueFuzhao User ๐Ÿ• Updated 2024-12-08 ๐Ÿ“… Created 2022-03-30
88
LTH14/fractalgen

PyTorch implementation of FractalGen https://arxiv.org/abs/2502.17437

LTH14
โญ 1,238 Python ๐Ÿ‘ค LTH14 User ๐Ÿ• Updated 2025-02-25 ๐Ÿ“… Created 2025-02-18
โญ 1,216 Python ๐Ÿ‘ค Cerebras Organization ๐Ÿ• Updated 2026-09-01 ๐Ÿ“… Created 2022-04-08
90
DeepRec-AI/DeepRec

DeepRec is a high-performance recommendation deep learning framework based on TensorFlow. It is hosted in incubation in LF AI & Data Foundation.

DeepRec-AI
โญ 1,183 C++ ๐Ÿ‘ค DeepRec-AI Organization ๐Ÿ• Updated 2025-01-21 ๐Ÿ“… Created 2021-12-24
91
deepseek-ai/profile-data

Analyze computation-communication overlap in V3/R1.

deepseek-ai
โญ 1,159 ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-03-21 ๐Ÿ“… Created 2025-02-26
92
gpu-mode/awesomeMLSys

An ML Systems Onboarding list

gpu-mode
โญ 1,154 ๐Ÿ‘ค gpu-mode Organization ๐Ÿ• Updated 2026-02-19 ๐Ÿ“… Created 2024-05-06
93
HITsz-TMG/Uni-MoE

Uni-MoE: Lychee's Large Multimodal Model Family.

HITsz-TMG
โญ 1,120 Python ๐Ÿ‘ค HITsz-TMG Organization ๐Ÿ• Updated 2026-08-06 ๐Ÿ“… Created 2024-03-04
94
arcee-ai/DistillKit

An Open Source Toolkit For LLM Distillation

arcee-ai
โญ 1,095 Python ๐Ÿ‘ค arcee-ai Organization ๐Ÿ• Updated 2026-05-12 ๐Ÿ“… Created 2024-07-31
95
open-gigaai/giga-train

GigaTrain: An Efficient and Scalable Training Framework for AI Models

open-gigaai
โญ 1,074 Python ๐Ÿ‘ค open-gigaai Organization ๐Ÿ• Updated 2026-08-25 ๐Ÿ“… Created 2025-09-26
96
zhuzilin/ring-flash-attention

Ring attention implementation with flash attention

zhuzilin
โญ 1,054 Python ๐Ÿ‘ค zhuzilin User ๐Ÿ• Updated 2025-09-10 ๐Ÿ“… Created 2024-02-21
97
mryab/efficient-dl-systems

Efficient Deep Learning Systems course materials

mryab
โญ 1,052 Jupyter Notebook ๐Ÿ‘ค mryab User ๐Ÿ• Updated 2026-05-28 ๐Ÿ“… Created 2021-12-06
98
isaac-sim/OmniIsaacGymEnvs

Reinforcement Learning Environments for Omniverse Isaac Gym

isaac-sim
โญ 1,020 Python ๐Ÿ‘ค isaac-sim Organization ๐Ÿ• Updated 2024-06-06 ๐Ÿ“… Created 2022-06-01
99
PsycheFoundation/nousnet

An open infrastructure to democratize and decentralize the development of superintelligence for humanity.

PsycheFoundation
โญ 983 Rust ๐Ÿ‘ค PsycheFoundation Organization ๐Ÿ• Updated 2026-03-24 ๐Ÿ“… Created 2025-01-23
100
SandAI-org/MagiAttention

A Distributed Attention Towards Linear Scalability for Ultra-Long Context, Heterogeneous Data Training

SandAI-org
โญ 975 Python ๐Ÿ‘ค SandAI-org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-20