Git Stars Git Stars

#mixture-of-experts Repositories

78
Total Repositories
392,641
Total Stars
5,033
Avg Stars/Repo
51
hans0809/MiniMind-in-Depth

Source code analysis of the lightweight large language model MiniMind, including the complete processes for tokenizer, RoPE, MoE, KV Cache, pretraining, SFT, LoRA, and DPO.

hans0809
โญ 1,072 ๐Ÿ‘ค hans0809 User ๐Ÿ• Updated 2025-06-16 ๐Ÿ“… Created 2025-06-16
52
Helldez/BigMoeOnEdge

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, or edge device. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

Helldez
โญ 1,066 C++ ๐Ÿ‘ค Helldez User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-07-10
53
jondurbin/airoboros

Customizable implementation of the self-instruct paper.

jondurbin
โญ 1,050 Python ๐Ÿ‘ค jondurbin User ๐Ÿ• Updated 2024-03-07 ๐Ÿ“… Created 2023-04-29
54
microsoft/Tutel

Tutel MoE: Optimized Mixture-of-Experts Library, Supporting GLM/Kimi/GPT/DeepSeek/Qwen with FP8/NVFP4/MXFP4

microsoft
โญ 1,027 C ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2021-08-06
55
NVIDIA/cudnn-frontend

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

NVIDIA
โญ 1,008 Cuda ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-01-22
56
pjlab-sys4nlp/llama-moe

โ›ท๏ธ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

pjlab-sys4nlp
โญ 1,004 Python ๐Ÿ‘ค pjlab-sys4nlp Organization ๐Ÿ• Updated 2024-12-06 ๐Ÿ“… Created 2023-07-24
57
myshell-ai/JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

myshell-ai
โญ 988 Python ๐Ÿ‘ค myshell-ai Organization ๐Ÿ• Updated 2024-07-23 ๐Ÿ“… Created 2024-04-02
58
SMTorg/smt

SMT: The Surrogate Modeling Toolbox

SMTorg
โญ 927 Jupyter Notebook ๐Ÿ‘ค SMTorg Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2016-11-08
59
InternLM/Intern-S1

A Scientific Multimodal Foundation Model

InternLM
โญ 876 ๐Ÿ‘ค InternLM Organization ๐Ÿ• Updated 2026-09-13 ๐Ÿ“… Created 2025-07-25
60
ARahim3/kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

ARahim3
โญ 812 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-08-31
โญ 812 ๐Ÿ‘ค ByteDance-Seed Organization ๐Ÿ• Updated 2025-06-09 ๐Ÿ“… Created 2025-04-10
62
Dao-AILab/sonic-moe

Accelerating MoE with IO and Tile-aware Optimizations

Dao-AILab
โญ 791 Python ๐Ÿ‘ค Dao-AILab Organization ๐Ÿ• Updated 2026-08-29 ๐Ÿ“… Created 2025-12-15
63
AviSoori1x/makeMoE

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

AviSoori1x
โญ 783 Jupyter Notebook ๐Ÿ‘ค AviSoori1x User ๐Ÿ• Updated 2024-10-30 ๐Ÿ“… Created 2024-01-22
64
datawhalechina/unlock-deepseek

Insights, extensions, and reproductions of DeepSeek series works.

datawhalechina
โญ 743 Python ๐Ÿ‘ค datawhalechina Organization ๐Ÿ• Updated 2026-03-09 ๐Ÿ“… Created 2025-01-30
65
prabanta-dev/janas

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

prabanta-dev
โญ 738 C ๐Ÿ‘ค prabanta-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-20
66
deepseek-ai/ESFT

Expert Specialized Fine-Tuning

deepseek-ai
โญ 737 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-05-22 ๐Ÿ“… Created 2024-07-04
โญ 729 Jupyter Notebook ๐Ÿ‘ค google-research Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2021-10-08
68
inclusionAI/Ming

Ming - facilitating advanced multimodal understanding and generation capabilities built upon the Ling LLM.

inclusionAI
โญ 683 Jupyter Notebook ๐Ÿ‘ค inclusionAI Organization ๐Ÿ• Updated 2026-07-27 ๐Ÿ“… Created 2025-04-21
69
carloslfu/slotstream

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16โ€“64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

carloslfu
โญ 674 Swift ๐Ÿ‘ค carloslfu User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-28
70
leonickson1/Swiftlet

Swiftlet is a Swift and Metal runtime that runs large Qwen Mixture-of-Experts models locally on Apple devices by streaming expert weights from storage, enabling 35B and 80B models to run with low RAM, including on iPhone.

leonickson1
โญ 667 Swift ๐Ÿ‘ค leonickson1 User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-03
71
codecaution/Awesome-Mixture-of-Experts-Papers

A curated reading list of research in Mixture-of-Experts(MoE).

codecaution
โญ 654 ๐Ÿ‘ค codecaution User ๐Ÿ• Updated 2024-10-30 ๐Ÿ“… Created 2022-02-24
72
perplexityai/pplx-kernels

Perplexity GPU Kernels

perplexityai
โญ 614 C++ ๐Ÿ‘ค perplexityai Organization ๐Ÿ• Updated 2025-11-07 ๐Ÿ“… Created 2025-03-24
73
ymcui/Chinese-Mixtral

Chinese Mixtral Mixture-of-Experts LLMs (Chinese Mixtral MoE LLMs)

ymcui
โญ 613 Python ๐Ÿ‘ค ymcui User ๐Ÿ• Updated 2026-04-19 ๐Ÿ“… Created 2024-01-11
74
fe1ixxu/ALMA

State-of-the-art LLM-based translation models.

fe1ixxu
โญ 585 Ruby ๐Ÿ‘ค fe1ixxu User ๐Ÿ• Updated 2025-04-09 ๐Ÿ“… Created 2023-09-13
โญ 557 Python ๐Ÿ‘ค Snowflake-Labs Organization ๐Ÿ• Updated 2024-08-16 ๐Ÿ“… Created 2024-04-22
76
Indras-Mirror/Xolotl

Xolotl: run DeepSeek-V4.1-Flash, all 552B parameters, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s agent, ~41 fast, 512K context. Plus Inkling-2-Mini (Thinking Machines, own engine + MTP drafter), GLM-5.3-Flash, Qwen3.8, LFM3-Omni. MoE experts tiered across VRAM, RAM, and NVMe (Xolotl Direct). OpenAI + Anthropic API.

Indras-Mirror
โญ 549 C++ ๐Ÿ‘ค Indras-Mirror User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-06
77
brontoguana/krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

brontoguana
โญ 530 C++ ๐Ÿ‘ค brontoguana User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2026-02-09
78
Leeroo-AI/mergoo

A library for easily merging multiple LLM experts, and efficiently train the merged LLM.

Leeroo-AI
โญ 500 Python ๐Ÿ‘ค Leeroo-AI Organization ๐Ÿ• Updated 2024-08-26 ๐Ÿ“… Created 2024-04-08