#speculative-decoding Repositories
Discover 37 active open source GitHub repositories tagged with #speculative-decoding, accumulating 83,160 total stars (averaging 2,247 stars per project). Leading this topic is OpenBMB/MiniCPM with 11,898 stars.
37 repositories total
This is a simple technical็งๆฎ tutorial project, mainly focusing on explaining some interesting, cutting-edge technical concepts and principles. Each article strives to be completed within 5 minutes of reading.
Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4ร faster than Apple's MLX (mlx-lm) on the same weights.
Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
Zero-GPU CPU inference for 30Bโ284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.
[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder, and MiniCPM-V on a second engine built for PolyStrata. One-click installation for Windows and Linux, with an OpenAI/Anthropic API on localhost.
๐ฐ Must-read papers and blogs on Speculative Decoding โก๏ธ
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM
A curated collection of papers, technical reports, frameworks, and tools for on-policy distillation (OPD) of large language models
Fast inference from large language models via speculative decoding
Blazing-fast Apple Silicon inference engine optimized for Qwen & Nemotron. Compatible with the OpenAI and Anthropic APIs. Built by the team at nace.ai
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
Up to 4ร faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.
Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
From teacher to tiles โ a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.
Frequently Asked Questions
What is the most popular #speculative-decoding repository on GitHub?
The top ranked open source #speculative-decoding project on GitHub is OpenBMB/MiniCPM with 11,898 stars. MiniCPM5: SOTA on-device LLMs, small yet powerful.
Which programming languages are most used in #speculative-decoding repositories?
Among the top ranked #speculative-decoding projects on GitHub, the most common programming languages are Python, C++, Jupyter Notebook, JavaScript.
How is the #speculative-decoding repository ranking calculated?
We track 37 active open source repositories tagged with #speculative-decoding (500+ stars), totaling 83,160 stars with an average of 2,247 stars per repository. Rankings update daily from the GitHub API.