Git Stars Git Stars

#pretraining Repositories

Discover 22 active open source GitHub repositories tagged with #pretraining, accumulating 150,107 total stars (averaging 6,823 stars per project). Leading this topic is rasbt/LLMs-from-scratch with 108,745 stars.

22
Total Repositories
150,107
Total Stars
6,823
Avg Stars/Repo
Programming Languages: Python Jupyter Notebook
1
rasbt/LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

rasbt
โญ 108,745 Jupyter Notebook ๐Ÿ‘ค rasbt User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-07-23
2
LlamaChinese/Llama-Chinese

Llama Chinese Community, real-time aggregation of the latest Llama learning materials, building the best Chinese Llama large model open-source ecosystem, fully open-source and commercially available

LlamaChinese
โญ 14,712 Python ๐Ÿ‘ค LlamaChinese Organization ๐Ÿ• Updated 2025-04-06 ๐Ÿ“… Created 2023-07-19
3
microsoft/LMOps

General technology for enabling AI capabilities w/ LLMs and MLLMs

microsoft
โญ 4,484 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-09-15 ๐Ÿ“… Created 2022-12-13
4
OFA-Sys/OFA

Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

OFA-Sys
โญ 2,550 Python ๐Ÿ‘ค OFA-Sys Organization ๐Ÿ• Updated 2024-04-24 ๐Ÿ“… Created 2022-01-29
5
X-PLUG/mPLUG-Owl

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

X-PLUG
โญ 2,541 Python ๐Ÿ‘ค X-PLUG Organization ๐Ÿ• Updated 2025-04-02 ๐Ÿ“… Created 2023-04-25
6
sapientinc/HRM-Text

HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.

sapientinc
โญ 2,341 Python ๐Ÿ‘ค sapientinc Organization ๐Ÿ• Updated 2026-09-21 ๐Ÿ“… Created 2026-05-18
7
ChandlerBang/awesome-self-supervised-gnn

Papers about pretraining and self-supervised learning on Graph Neural Networks (GNNs).

ChandlerBang
โญ 1,704 Python ๐Ÿ‘ค ChandlerBang User ๐Ÿ• Updated 2024-02-02 ๐Ÿ“… Created 2020-05-27
8
keyu-tian/SparK

[ICLR'23 Spotlight๐Ÿ”ฅ] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

keyu-tian
โญ 1,363 Python ๐Ÿ‘ค keyu-tian User ๐Ÿ• Updated 2024-01-23 ๐Ÿ“… Created 2023-01-05
9
InternScience/GraphGen

GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

InternScience
โญ 1,236 Python ๐Ÿ‘ค InternScience Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2025-01-08
10
OpenWAM-Official/OpenWAM

Official repository for "OpenWAM: An Open, Modular Exploration Towards Systematic Worldโ€“Action Model Pretraining".

OpenWAM-Official
โญ 1,210 Python ๐Ÿ‘ค OpenWAM-Official Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2026-09-06
11
deepmodeling/Uni-Mol

Official Repository for the Uni-Mol Series Methods

deepmodeling
โญ 1,112 Python ๐Ÿ‘ค deepmodeling Organization ๐Ÿ• Updated 2025-05-29 ๐Ÿ“… Created 2022-05-22
12
seal-rg/recurrent-pretraining

Pretraining and inference code for a large-scale depth-recurrent language model

seal-rg
โญ 993 Python ๐Ÿ‘ค seal-rg Organization ๐Ÿ• Updated 2025-12-29 ๐Ÿ“… Created 2025-02-07
13
baidu-baige/LoongForge

A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3, and more.

baidu-baige
โญ 881 Python ๐Ÿ‘ค baidu-baige Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-23
14
PKU-YuanGroup/LanguageBind

ใ€ICLR 2024๐Ÿ”ฅใ€‘ Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

PKU-YuanGroup
โญ 856 Python ๐Ÿ‘ค PKU-YuanGroup Organization ๐Ÿ• Updated 2024-03-25 ๐Ÿ“… Created 2023-10-02
15
zubair-irshad/Awesome-Robotics-3D

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites

zubair-irshad
โญ 824 ๐Ÿ‘ค zubair-irshad User ๐Ÿ• Updated 2025-12-17 ๐Ÿ“… Created 2024-08-12
16
zcablii/SARDet_100K

[NeurIPS 2024 spotlight] Offical implementation of MSFA and release of SARDet_100K dataset for Large-Scale Synthetic Aperture Radar (SAR) Object Detection

zcablii
โญ 743 Python ๐Ÿ‘ค zcablii User ๐Ÿ• Updated 2025-05-06 ๐Ÿ“… Created 2024-02-20
17
AGI-Arena/MARS

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models

AGI-Arena
โญ 725 Python ๐Ÿ‘ค AGI-Arena Organization ๐Ÿ• Updated 2026-03-26 ๐Ÿ“… Created 2024-11-17
18
Coobiw/MPP-LLaVA

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.

Coobiw
โญ 678 Jupyter Notebook ๐Ÿ‘ค Coobiw User ๐Ÿ• Updated 2025-03-10 ๐Ÿ“… Created 2023-10-24
19
m-bain/webvid

Large-scale text-video dataset. 10 million captioned short videos.

m-bain
โญ 670 Python ๐Ÿ‘ค m-bain User ๐Ÿ• Updated 2024-08-14 ๐Ÿ“… Created 2021-06-07
20
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17
21
westlake-repl/SaProt

Saprot: Protein Language Model with Structural Alphabet (AA+3Di)

westlake-repl
โญ 646 Python ๐Ÿ‘ค westlake-repl Organization ๐Ÿ• Updated 2026-03-08 ๐Ÿ“… Created 2023-10-02
22
PITI-Synthesis/PITI

PITI: Pretraining is All You Need for Image-to-Image Translation

PITI-Synthesis
โญ 501 Python ๐Ÿ‘ค PITI-Synthesis User ๐Ÿ• Updated 2024-06-02 ๐Ÿ“… Created 2022-05-27

Frequently Asked Questions

1

What is the most popular #pretraining repository on GitHub?

The top ranked open source #pretraining project on GitHub is rasbt/LLMs-from-scratch with 108,745 stars. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

2

Which programming languages are most used in #pretraining repositories?

Among the top ranked #pretraining projects on GitHub, the most common programming languages are Python, Jupyter Notebook.

3

How is the #pretraining repository ranking calculated?

We track 22 active open source repositories tagged with #pretraining (500+ stars), totaling 150,107 stars with an average of 6,823 stars per repository. Rankings update daily from the GitHub API.