#pretraining Repositories
Discover 22 active open source GitHub repositories tagged with #pretraining, accumulating 150,107 total stars (averaging 6,823 stars per project). Leading this topic is rasbt/LLMs-from-scratch with 108,745 stars.
22 repositories total
Llama Chinese Community, real-time aggregation of the latest Llama learning materials, building the best Chinese Llama large model open-source ecosystem, fully open-source and commercially available
Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNNs).
[ICLR'23 Spotlight๐ฅ] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
Official repository for "OpenWAM: An Open, Modular Exploration Towards Systematic WorldโAction Model Pretraining".
Pretraining and inference code for a large-scale depth-recurrent language model
A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3, and more.
ใICLR 2024๐ฅใ Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites
[NeurIPS 2024 spotlight] Offical implementation of MSFA and release of SARDet_100K dataset for Large-Scale Synthetic Aperture Radar (SAR) Object Detection
The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models
Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
Frequently Asked Questions
What is the most popular #pretraining repository on GitHub?
The top ranked open source #pretraining project on GitHub is rasbt/LLMs-from-scratch with 108,745 stars. Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Which programming languages are most used in #pretraining repositories?
Among the top ranked #pretraining projects on GitHub, the most common programming languages are Python, Jupyter Notebook.
How is the #pretraining repository ranking calculated?
We track 22 active open source repositories tagged with #pretraining (500+ stars), totaling 150,107 stars with an average of 6,823 stars per repository. Rankings update daily from the GitHub API.