Git Stars Git Stars

#pre-training Repositories

Discover 18 active open source GitHub repositories tagged with #pre-training, accumulating 43,729 total stars (averaging 2,429 stars per project). Leading this topic is RUCAIBox/LLMSurvey with 12,165 stars.

18
Total Repositories
43,729
Total Stars
2,429
Avg Stars/Repo
Programming Languages: Python Jupyter Notebook
1
RUCAIBox/LLMSurvey

The official GitHub page for the survey paper "A Survey of Large Language Models".

RUCAIBox
โญ 12,165 Python ๐Ÿ‘ค RUCAIBox Organization ๐Ÿ• Updated 2025-03-11 ๐Ÿ“… Created 2023-03-14
2
datajuicer/data-juicer

Data processing for and with foundation models! ๐ŸŽ ๐Ÿ‹ ๐ŸŒฝ โžก๏ธ โžก๏ธ๐Ÿธ ๐Ÿน ๐Ÿท

datajuicer
โญ 7,266 Python ๐Ÿ‘ค datajuicer Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-08-01
3
bojone/bert4keras

Keras implementation of Transformers for humans

bojone
โญ 5,422 Python ๐Ÿ‘ค bojone User ๐Ÿ• Updated 2024-11-11 ๐Ÿ“… Created 2019-08-26
4
dbiir/UER-py

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo

dbiir
โญ 3,102 Python ๐Ÿ‘ค dbiir Organization ๐Ÿ• Updated 2024-05-09 ๐Ÿ“… Created 2019-04-10
5
EgoAlpha/prompt-in-context-learning

Awesome resources for in-context learning and prompt engineering: Mastery of the LLMs such as ChatGPT, GPT-3, and FlanT5, with up-to-date and cutting-edge updates.

EgoAlpha
โญ 2,251 Jupyter Notebook ๐Ÿ‘ค EgoAlpha User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-03-08
6
ChandlerBang/awesome-self-supervised-gnn

Papers about pretraining and self-supervised learning on Graph Neural Networks (GNNs).

ChandlerBang
โญ 1,704 Python ๐Ÿ‘ค ChandlerBang User ๐Ÿ• Updated 2024-02-02 ๐Ÿ“… Created 2020-05-27
7
SalesforceAIResearch/uni2ts

Unified Training of Universal Time Series Forecasting Transformers

SalesforceAIResearch
โญ 1,631 Jupyter Notebook ๐Ÿ‘ค SalesforceAIResearch Organization ๐Ÿ• Updated 2026-06-02 ๐Ÿ“… Created 2024-02-07
8
LirongWu/awesome-graph-self-supervised-learning

Code for TKDE paper "Self-supervised learning on graphs: Contrastive, generative, or predictive"

LirongWu
โญ 1,431 ๐Ÿ‘ค LirongWu User ๐Ÿ• Updated 2024-08-15 ๐Ÿ“… Created 2019-05-24
9
zjunlp/KnowLM

An Open-sourced Knowledgable Large Language Model Framework.

zjunlp
โญ 1,387 Python ๐Ÿ‘ค zjunlp Organization ๐Ÿ• Updated 2025-01-11 ๐Ÿ“… Created 2023-04-01
10
yzhuoning/Awesome-CLIP

Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).

yzhuoning
โญ 1,231 ๐Ÿ‘ค yzhuoning User ๐Ÿ• Updated 2024-06-28 ๐Ÿ“… Created 2021-09-05
11
qingsongedu/Awesome-TimeSeries-SpatioTemporal-LM-LLM

A professional list on Large (Language) Models and Foundation Models (LLM, LM, FM) for Time Series, Spatiotemporal, and Event Data.

qingsongedu
โญ 1,184 ๐Ÿ‘ค qingsongedu User ๐Ÿ• Updated 2024-12-22 ๐Ÿ“… Created 2023-06-10
12
Tencent/TencentPretrain

Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo

Tencent
โญ 1,086 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2024-08-04 ๐Ÿ“… Created 2022-09-26
13
nancheng58/Awesome-LLM4RS-Papers

Large Language Model-enhanced Recommender System Papers

nancheng58
โญ 765 ๐Ÿ‘ค nancheng58 User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-05-11
14
linwhitehat/ET-BERT

The repository of ET-BERT, a network traffic classification model on encrypted traffic. The work has been accepted as The Web Conference (WWW) 2022 accepted paper.

linwhitehat
โญ 687 Python ๐Ÿ‘ค linwhitehat User ๐Ÿ• Updated 2026-07-23 ๐Ÿ“… Created 2022-02-09
15
cxcscmu/Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

cxcscmu
โญ 654 Python ๐Ÿ‘ค cxcscmu Organization ๐Ÿ• Updated 2025-02-24 ๐Ÿ“… Created 2025-02-17
16
princeton-nlp/LLM-Shearing

[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

princeton-nlp
โญ 635 Python ๐Ÿ‘ค princeton-nlp Organization ๐Ÿ• Updated 2024-03-04 ๐Ÿ“… Created 2023-10-16
17
Shen-Lab/GraphCL

[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen

Shen-Lab
โญ 620 Python ๐Ÿ‘ค Shen-Lab User ๐Ÿ• Updated 2024-07-17 ๐Ÿ“… Created 2020-09-25
18
microsoft/XPretrain

Multi-modality pre-training

microsoft
โญ 510 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-03-15

Frequently Asked Questions

1

What is the most popular #pre-training repository on GitHub?

The top ranked open source #pre-training project on GitHub is RUCAIBox/LLMSurvey with 12,165 stars. The official GitHub page for the survey paper "A Survey of Large Language Models".

2

Which programming languages are most used in #pre-training repositories?

Among the top ranked #pre-training projects on GitHub, the most common programming languages are Python, Jupyter Notebook.

3

How is the #pre-training repository ranking calculated?

We track 18 active open source repositories tagged with #pre-training (500+ stars), totaling 43,729 stars with an average of 2,429 stars per repository. Rankings update daily from the GitHub API.