Git Stars Git Stars

#speculative-decoding Repositories

Discover 37 active open source GitHub repositories tagged with #speculative-decoding, accumulating 83,160 total stars (averaging 2,247 stars per project). Leading this topic is OpenBMB/MiniCPM with 11,898 stars.

37
Total Repositories
83,160
Total Stars
2,247
Avg Stars/Repo
1
OpenBMB/MiniCPM

MiniCPM5: SOTA on-device LLMs, small yet powerful.

OpenBMB
โญ 11,898 Jupyter Notebook ๐Ÿ‘ค OpenBMB Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2024-01-29
2
karminski/one-small-step

This is a simple technical็ง‘ๆ™ฎ tutorial project, mainly focusing on explaining some interesting, cutting-edge technical concepts and principles. Each article strives to be completed within 5 minutes of reading.

karminski
โญ 7,006 Python ๐Ÿ‘ค karminski User ๐Ÿ• Updated 2026-03-08 ๐Ÿ“… Created 2025-01-16
3
z-lab/dflash

DFlash: Block Diffusion for Flash Speculative Decoding

z-lab
โญ 6,254 Python ๐Ÿ‘ค z-lab Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2026-01-04
4
meta-pytorch/gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

meta-pytorch
โญ 6,244 Python ๐Ÿ‘ค meta-pytorch Organization ๐Ÿ• Updated 2025-08-22 ๐Ÿ“… Created 2023-10-17
5
NVIDIA/Model-Optimizer

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

NVIDIA
โญ 5,646 Python ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-23
6
raullenchai/Rapid-MLX

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4ร— faster than Apple's MLX (mlx-lm) on the same weights.

raullenchai
โญ 4,168 Python ๐Ÿ‘ค raullenchai User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-25
7
radixark/miles

Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.

radixark
โญ 3,386 Python ๐Ÿ‘ค radixark Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-10-09
8
Luce-Org/lucebox

Lucebox inference engine for heterogeneous hardware & consumer GPUs

Luce-Org
โญ 3,024 C++ ๐Ÿ‘ค Luce-Org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-03
9
syv-ai/HyperQwen

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

syv-ai
โญ 2,778 Python ๐Ÿ‘ค syv-ai Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-08-15
10
youssofal/MTPLX

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

youssofal
โญ 2,729 Python ๐Ÿ‘ค youssofal User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-05-02
11
incoai/splash

A local inference engine for Apple silicon, built around the model.

incoai
โญ 2,591 C++ ๐Ÿ‘ค incoai Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-09-18
12
SafeAILab/EAGLE

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

SafeAILab
โญ 2,581 Python ๐Ÿ‘ค SafeAILab Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2023-12-07
13
dphnAI/sonar

Large-scale LLM inference engine

dphnAI
โญ 1,911 Python ๐Ÿ‘ค dphnAI Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-06-23
14
Tencent/AngelSlim

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

Tencent
โญ 1,811 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-07-04
15
sybil-solutions/inference-optimisation-skill

Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

sybil-solutions
โญ 1,425 Python ๐Ÿ‘ค sybil-solutions Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-11-16
16
peonist-ai/halogen-flash-server

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

peonist-ai
โญ 1,412 Shell ๐Ÿ‘ค peonist-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-08-26
17
synchopate/llama-modchip

Zero-GPU CPU inference for 30Bโ€“284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.

synchopate
โญ 1,398 Python ๐Ÿ‘ค synchopate User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-29
18
hao-ai-lab/LookaheadDecoding

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

hao-ai-lab
โญ 1,337 Python ๐Ÿ‘ค hao-ai-lab Organization ๐Ÿ• Updated 2025-03-06 ๐Ÿ“… Created 2023-11-21
19
simplexai-labs/PolyStrata

Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder, and MiniCPM-V on a second engine built for PolyStrata. One-click installation for Windows and Linux, with an OpenAI/Anthropic API on localhost.

simplexai-labs
โญ 1,324 C++ ๐Ÿ‘ค simplexai-labs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-10-06
20
hemingkx/SpeculativeDecodingPapers

๐Ÿ“ฐ Must-read papers and blogs on Speculative Decoding โšก๏ธ

hemingkx
โญ 1,309 ๐Ÿ‘ค hemingkx User ๐Ÿ• Updated 2026-06-27 ๐Ÿ“… Created 2023-09-20
21
eagle-hq/EAGLE-4

Official Inference Code of EAGLE-4

eagle-hq
โญ 1,220 Python ๐Ÿ‘ค eagle-hq Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2026-10-20
22
StayLameBro/backburner

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

StayLameBro
โญ 1,195 Python ๐Ÿ‘ค StayLameBro User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-10-01
23
vllm-project/speculators

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

vllm-project
โญ 972 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-04-08
24
chrisliu298/awesome-on-policy-distillation

A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models

chrisliu298
โญ 926 ๐Ÿ‘ค chrisliu298 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-03-22
25
feifeibear/LLMSpeculativeSampling

Fast inference from large language models via speculative decoding

feifeibear
โญ 875 Python ๐Ÿ‘ค feifeibear User ๐Ÿ• Updated 2024-08-22 ๐Ÿ“… Created 2023-08-16
26
nace-ai/kapitsa

Blazing-fast Apple Silicon inference engine optimized for Qwen & Nemotron. Compatible with the OpenAI and Anthropic APIs. Built by the team at nace.ai

nace-ai
โญ 840 C++ ๐Ÿ‘ค nace-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-16
27
ARahim3/kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

ARahim3
โญ 812 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-08-31
28
ARahim3/mlx-dspark

Up to 4ร— faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ€” Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

ARahim3
โญ 743 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2026-06-29
29
prabanta-dev/janas

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

prabanta-dev
โญ 738 C ๐Ÿ‘ค prabanta-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-20
30
Avarok-Cybersecurity/atlas

Pure Rust Inference Engine

Avarok-Cybersecurity
โญ 737 Rust ๐Ÿ‘ค Avarok-Cybersecurity Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-05-05
32
MDK8888/GPTFast

Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.

MDK8888
โญ 687 Python ๐Ÿ‘ค MDK8888 User ๐Ÿ• Updated 2024-08-22 ๐Ÿ“… Created 2024-02-18
33
Nehanth/pooled

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

Nehanth
โญ 658 JavaScript ๐Ÿ‘ค Nehanth User ๐Ÿ• Updated 2026-11-03 ๐Ÿ“… Created 2026-09-01
35
zengxiao-he/tessera

From teacher to tiles โ€” a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

zengxiao-he
โญ 556 Python ๐Ÿ‘ค zengxiao-he User ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2026-06-05
36
turboderp-org/exui

Web UI for ExLlamaV2

turboderp-org
โญ 513 JavaScript ๐Ÿ‘ค turboderp-org Organization ๐Ÿ• Updated 2025-02-05 ๐Ÿ“… Created 2023-10-29
37
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.

AEON-7
โญ 501 Python ๐Ÿ‘ค AEON-7 User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2026-04-25

Frequently Asked Questions

1

What is the most popular #speculative-decoding repository on GitHub?

The top ranked open source #speculative-decoding project on GitHub is OpenBMB/MiniCPM with 11,898 stars. MiniCPM5: SOTA on-device LLMs, small yet powerful.

2

Which programming languages are most used in #speculative-decoding repositories?

Among the top ranked #speculative-decoding projects on GitHub, the most common programming languages are Python, C++, Jupyter Notebook, JavaScript.

3

How is the #speculative-decoding repository ranking calculated?

We track 37 active open source repositories tagged with #speculative-decoding (500+ stars), totaling 83,160 stars with an average of 2,247 stars per repository. Rankings update daily from the GitHub API.