Git Stars Git Stars

#llm-serving Repositories

Discover 44 active open source GitHub repositories tagged with #llm-serving, accumulating 381,346 total stars (averaging 8,666 stars per project). Leading this topic is vllm-project/vllm with 97,302 stars.

44
Total Repositories
381,346
Total Stars
8,666
Avg Stars/Repo
1
vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project
โญ 97,302 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-02-09
2
exo-explore/exo

Run frontier AI locally.

exo-explore
โญ 48,516 Python ๐Ÿ‘ค exo-explore Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2024-06-24
3
ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-project
โญ 44,352 Python ๐Ÿ‘ค ray-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-10-25
4
lm-sys/FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

lm-sys
โญ 39,550 Python ๐Ÿ‘ค lm-sys Organization ๐Ÿ• Updated 2026-05-01 ๐Ÿ“… Created 2023-03-19
5
liguodongiot/llm-action

This project aims to share the technical principles and practical experience related to large models (large model engineering, practical application of large models).

liguodongiot
โญ 25,297 HTML ๐Ÿ‘ค liguodongiot User ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2023-05-23
6
NVIDIA/TensorRT-LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

NVIDIA
โญ 15,112 Python ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-08-16
7
bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml
โญ 12,610 Python ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-04-19
8
skypilot-org/skypilot

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

skypilot-org
โญ 10,869 Python ๐Ÿ‘ค skypilot-org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-08-11
9
bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

bentoml
โญ 8,944 Python ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2019-04-02
10
bojieli/ai-infra-book

ใ€ŠIn-Depth Understanding of AI Infrastructure: Quantitative Analysis and System Designใ€‹ (by Li Bojie), an open-source book manuscript: Starting from hardware constraints and model architectures, it quantitatively derives the design of LLM inference and training systems. Includes the full text, PDF, accompanying computational tools, and experiments.

bojieli
โญ 8,179 Python ๐Ÿ‘ค bojieli User ๐Ÿ• Updated 2026-10-01 ๐Ÿ“… Created 2026-08-22
11
z-lab/dflash

DFlash: Block Diffusion for Flash Speculative Decoding

z-lab
โญ 6,254 Python ๐Ÿ‘ค z-lab Organization ๐Ÿ• Updated 2026-08-18 ๐Ÿ“… Created 2026-01-04
12
gpustack/gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

gpustack
โญ 6,097 Python ๐Ÿ‘ค gpustack Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-11
13
sgl-project/mini-sglang

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

sgl-project
โญ 5,623 Python ๐Ÿ‘ค sgl-project Organization ๐Ÿ• Updated 2026-05-17 ๐Ÿ“… Created 2025-09-01
14
superduper-io/superduper

Superduper: End-to-end framework for building custom AI applications and agents.

superduper-io
โญ 5,317 Python ๐Ÿ‘ค superduper-io Organization ๐Ÿ• Updated 2025-09-01 ๐Ÿ“… Created 2022-08-30
15
SylphAI-Inc/LLM-engineer-handbook

A curated list of Large Language Model resources, covering model training, serving, fine-tuning, and building LLM applications.

SylphAI-Inc
โญ 5,014 ๐Ÿ‘ค SylphAI-Inc Organization ๐Ÿ• Updated 2025-08-18 ๐Ÿ“… Created 2024-11-04
16
predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

predibase
โญ 3,836 Python ๐Ÿ‘ค predibase Organization ๐Ÿ• Updated 2026-05-28 ๐Ÿ“… Created 2023-10-20
17
PaddlePaddle/FastDeploy

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

PaddlePaddle
โญ 3,722 Python ๐Ÿ‘ค PaddlePaddle Organization ๐Ÿ• Updated 2026-09-24 ๐Ÿ“… Created 2022-06-27
18
vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

vllm-project
โญ 3,128 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-01-29
19
thu-pacman/chitu

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

thu-pacman
โญ 3,005 Python ๐Ÿ‘ค thu-pacman Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-02-20
20
MoonshotAI/MoBA

MoBA: Mixture of Block Attention for Long-Context LLMs

MoonshotAI
โญ 2,122 Python ๐Ÿ‘ค MoonshotAI Organization ๐Ÿ• Updated 2025-04-03 ๐Ÿ“… Created 2025-02-17
21
microsoft/aici

AICI: Prompts as (Wasm) Programs

microsoft
โญ 2,077 Rust ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-01-22 ๐Ÿ“… Created 2023-09-26
22
SemiAnalysisAI/InferenceX

Open-Source Inference Research Platform

SemiAnalysisAI
โญ 2,074 Python ๐Ÿ‘ค SemiAnalysisAI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-12
23
ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

ovg-project
โญ 2,051 Python ๐Ÿ‘ค ovg-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-27
24
HuangOwen/Awesome-LLM-Compression

Awesome LLM compression research papers and tools.

HuangOwen
โญ 1,899 ๐Ÿ‘ค HuangOwen User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-05-30
25
RunanywhereAI/trnrt

The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

RunanywhereAI
โญ 1,653 Python ๐Ÿ‘ค RunanywhereAI Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-10-30
26
GradientHQ/parallax

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

GradientHQ
โญ 1,429 Python ๐Ÿ‘ค GradientHQ Organization ๐Ÿ• Updated 2026-07-01 ๐Ÿ“… Created 2025-09-22
27
peonist-ai/halogen-flash-server

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

peonist-ai
โญ 1,412 Shell ๐Ÿ‘ค peonist-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-08-26
28
alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

alibaba
โญ 1,402 Python ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-27
29
microsoft/MInference

[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

microsoft
โญ 1,230 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-05-22
30
efeslab/Nanoflow

A throughput-oriented high-performance serving framework for LLMs

efeslab
โญ 978 Jupyter Notebook ๐Ÿ‘ค efeslab Organization ๐Ÿ• Updated 2026-03-29 ๐Ÿ“… Created 2024-08-19
31
zhihu/ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

zhihu
โญ 906 C++ ๐Ÿ‘ค zhihu Organization ๐Ÿ• Updated 2026-03-18 ๐Ÿ“… Created 2024-12-06
32
mosecorg/mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

mosecorg
โญ 905 Python ๐Ÿ‘ค mosecorg Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2021-03-13
33
triton-inference-server/tutorials

This repository contains tutorials and examples for Triton Inference Server

triton-inference-server
โญ 869 Python ๐Ÿ‘ค triton-inference-server Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2022-12-16
34
mit-han-lab/omniserve

[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention

mit-han-lab
โญ 840 C++ ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2025-03-06 ๐Ÿ“… Created 2024-05-06
35
helixml/helix

โ™พ๏ธ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude, Codex, Gemini and open models on a full private AI Stack โ™พ๏ธ

helixml
โญ 833 Go ๐Ÿ‘ค helixml Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-17
36
LLMServe/DistServe

Disaggregated serving system for Large Language Models (LLMs).

LLMServe
โญ 817 Jupyter Notebook ๐Ÿ‘ค LLMServe Organization ๐Ÿ• Updated 2025-04-06 ๐Ÿ“… Created 2024-01-16
37
ARahim3/kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

ARahim3
โญ 812 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-08-31
38
nunchaku-ai/deepcompressor

Model Compression Toolbox for Large Language Models and Diffusion Models

nunchux-ai
โญ 799 Python ๐Ÿ‘ค nunchux-ai Organization ๐Ÿ• Updated 2025-08-14 ๐Ÿ“… Created 2024-05-06
39
pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine โ€” no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

pegainfer-project
โญ 748 Rust ๐Ÿ‘ค pegainfer-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-17
40
owenliang/qwen-vllm

Tongyi Qianwen VLLM Inference Deployment DEMO

owenliang
โญ 637 Python ๐Ÿ‘ค owenliang User ๐Ÿ• Updated 2024-03-28 ๐Ÿ“… Created 2024-01-21
42
rohan-paul/LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

rohan-paul
โญ 574 Jupyter Notebook ๐Ÿ‘ค rohan-paul User ๐Ÿ• Updated 2025-04-01 ๐Ÿ“… Created 2023-10-22
43
llumnix-project/llumnix-ray

Efficient and easy multi-instance LLM serving

llumnix-project
โญ 561 Python ๐Ÿ‘ค llumnix-project Organization ๐Ÿ• Updated 2026-03-12 ๐Ÿ“… Created 2024-05-20
44
OmniJev/Flide

๐Ÿ”ฅ๐Ÿ”ฅ The extremely fast inference engine for decision models

OmniJev
โญ 550 Python ๐Ÿ‘ค OmniJev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-24

Frequently Asked Questions

1

What is the most popular #llm-serving repository on GitHub?

The top ranked open source #llm-serving project on GitHub is vllm-project/vllm with 97,302 stars. A high-throughput and memory-efficient inference and serving engine for LLMs

2

Which programming languages are most used in #llm-serving repositories?

Among the top ranked #llm-serving projects on GitHub, the most common programming languages are Python, Jupyter Notebook, Rust, C++.

3

How is the #llm-serving repository ranking calculated?

We track 44 active open source repositories tagged with #llm-serving (500+ stars), totaling 381,346 stars with an average of 8,666 stars per repository. Rankings update daily from the GitHub API.