Git Stars Git Stars

#model-serving Repositories

Discover 51 active open source GitHub repositories tagged with #model-serving, accumulating 211,758 total stars (averaging 4,152 stars per project). Leading this topic is vllm-project/vllm with 97,302 stars.

51
Total Repositories
211,758
Total Stars
4,152
Avg Stars/Repo
1
vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project
โญ 97,302 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-02-09
2
bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

bentoml
โญ 8,944 Python ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2019-04-02
3
vllm-project/vllm-omni

A framework for efficient model inference with omni-modality models

vllm-project
โญ 7,884 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-09-11
4
flyteorg/flyte

Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s โ€” in pure Python with async/await, also other languages (Rust, Go, and TypeScript)

flyteorg
โญ 7,801 Go ๐Ÿ‘ค flyteorg Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-10-21
5
zhaochenyang20/Awesome-ML-SYS-Tutorial

My learning notes for ML SYS.

zhaochenyang20
โญ 7,695 HTML ๐Ÿ‘ค zhaochenyang20 User ๐Ÿ• Updated 2026-10-21 ๐Ÿ“… Created 2024-11-09
6
kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

kserve
โญ 6,253 Go ๐Ÿ‘ค kserve Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-27
7
beclab/Olares

Open-Source Personal Cloud OS for Always-On Agents

beclab
โญ 5,373 Go ๐Ÿ‘ค beclab Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-29
8
ahkarami/Deep-Learning-in-Production

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

ahkarami
โญ 4,380 ๐Ÿ‘ค ahkarami User ๐Ÿ• Updated 2024-11-09 ๐Ÿ“… Created 2018-05-03
9
ModelTC/LightLLM

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

ModelTC
โญ 4,368 Python ๐Ÿ‘ค ModelTC Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-07-22
10
HuaizhengZhang/AI-Infra-from-Zero-to-Hero

๐Ÿš€ Awesome System for Machine Learning โšก๏ธ AI System Papers and Industry Practice. โšก๏ธ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). ๐Ÿป OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. ๐Ÿ—ƒ๏ธ Llama3, Mistral, etc. ๐Ÿง‘โ€๐Ÿ’ป Video Tutorials.

HuaizhengZhang
โญ 4,224 ๐Ÿ‘ค HuaizhengZhang User ๐Ÿ• Updated 2025-07-25 ๐Ÿ“… Created 2019-01-07
11
FedML-AI/FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

FedML-AI
โญ 4,068 Python ๐Ÿ‘ค FedML-AI Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2020-07-21
12
predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

predibase
โญ 3,836 Python ๐Ÿ‘ค predibase Organization ๐Ÿ• Updated 2026-05-28 ๐Ÿ“… Created 2023-10-20
13
vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

vllm-project
โญ 3,128 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-01-29
14
openlake-project/openlake

OpenLake is a high performance storage engine for efficient LLM inference and GPU Training

openlake-project
โญ 3,089 Rust ๐Ÿ‘ค openlake-project Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-04-27
15
thu-pacman/chitu

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

thu-pacman
โญ 3,005 Python ๐Ÿ‘ค thu-pacman Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-02-20
16
AmberLJC/LLMSys-PaperList

Large Language Model (LLM) Systems Paper List

AmberLJC
โญ 2,332 Python ๐Ÿ‘ค AmberLJC User ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2023-06-06
17
tensorchord/envd

๐Ÿ•๏ธ Reproducible development environment for humans and agents

tensorchord
โญ 2,249 Go ๐Ÿ‘ค tensorchord Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2022-04-11
18
microsoft/aici

AICI: Prompts as (Wasm) Programs

microsoft
โญ 2,077 Rust ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-01-22 ๐Ÿ“… Created 2023-09-26
19
tile-ai/TileRT

Tile-Based Runtime for Ultra-Low-Latency LLM Inference

tile-ai
โญ 1,906 Python ๐Ÿ‘ค tile-ai Organization ๐Ÿ• Updated 2026-09-24 ๐Ÿ“… Created 2025-11-12
20
sgl-project/sglang-omni

High-Performance Serving Runtime for Interactive Superintelligence Systems

sgl-project
โญ 1,882 Python ๐Ÿ‘ค sgl-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-01-07
21
janhq/awesome-local-ai

An awesome repository of local AI tools

janhq
โญ 1,791 ๐Ÿ‘ค janhq Organization ๐Ÿ• Updated 2024-11-13 ๐Ÿ“… Created 2023-09-06
22
mlrun/mlrun

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

mlrun
โญ 1,703 Python ๐Ÿ‘ค mlrun Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-09-01
23
ollaya-dev/ollaya

Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.

ollaya-dev
โญ 1,473 Rust ๐Ÿ‘ค ollaya-dev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-09-23
24
kitops-ml/kitops

An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

kitops-ml
โญ 1,451 Go ๐Ÿ‘ค kitops-ml Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-02-02
25
alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

alibaba
โญ 1,402 Python ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-27
26
hemingkx/SpeculativeDecodingPapers

๐Ÿ“ฐ Must-read papers and blogs on Speculative Decoding โšก๏ธ

hemingkx
โญ 1,309 ๐Ÿ‘ค hemingkx User ๐Ÿ• Updated 2026-06-27 ๐Ÿ“… Created 2023-09-20
27
logicalclocks/hopsworks

Hopsworks - Data-Intensive AI platform with a Feature Store

logicalclocks
โญ 1,299 Java ๐Ÿ‘ค logicalclocks Organization ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2018-07-26
28
DataExpert-io/ai-engineer-handbook

All the links, books, and creators you need to follow to stay up to date with AI!

DataExpert-io
โญ 1,261 ๐Ÿ‘ค DataExpert-io Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-02-16
29
basetenlabs/truss

The simplest way to serve AI/ML models in production

basetenlabs
โญ 1,236 Python ๐Ÿ‘ค basetenlabs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-07-06
30
gpu-mode/awesomeMLSys

An ML Systems Onboarding list

gpu-mode
โญ 1,154 ๐Ÿ‘ค gpu-mode Organization ๐Ÿ• Updated 2026-02-19 ๐Ÿ“… Created 2024-05-06
31
efeslab/Nanoflow

A throughput-oriented high-performance serving framework for LLMs

efeslab
โญ 978 Jupyter Notebook ๐Ÿ‘ค efeslab Organization ๐Ÿ• Updated 2026-03-29 ๐Ÿ“… Created 2024-08-19
32
openvinotoolkit/model_server

A scalable inference server for models optimized with OpenVINOโ„ข

openvinotoolkit
โญ 961 C++ ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-09-26
33
triton-inference-server/tensorrtllm_backend

The Triton TensorRT-LLM Backend

triton-inference-server
โญ 944 ๐Ÿ‘ค triton-inference-server Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2023-09-12
34
replicate/replicate-python

Python client for Replicate

replicate
โญ 909 Python ๐Ÿ‘ค replicate Organization ๐Ÿ• Updated 2025-08-26 ๐Ÿ“… Created 2022-05-11
35
zhihu/ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

zhihu
โญ 906 C++ ๐Ÿ‘ค zhihu Organization ๐Ÿ• Updated 2026-03-18 ๐Ÿ“… Created 2024-12-06
36
mosecorg/mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

mosecorg
โญ 905 Python ๐Ÿ‘ค mosecorg Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2021-03-13
37
trainindata/deploying-machine-learning-models

Code for the online course "Deployment of Machine Learning Models"

trainindata
โญ 879 Jupyter Notebook ๐Ÿ‘ค trainindata Organization ๐Ÿ• Updated 2024-08-20 ๐Ÿ“… Created 2019-01-09
38
InternLM/Intern-S1

A Scientific Multimodal Foundation Model

InternLM
โญ 876 ๐Ÿ‘ค InternLM Organization ๐Ÿ• Updated 2026-09-13 ๐Ÿ“… Created 2025-07-25
39
bentoml/Yatai

Model Deployment at Scale on Kubernetes ๐Ÿฆ„๏ธ

bentoml
โญ 843 TypeScript ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-05-30 ๐Ÿ“… Created 2021-07-30
40
Tejas-TA/predikit

The missing bridge between your ML models and your AI agents.

Tejas-TA
โญ 840 Python ๐Ÿ‘ค Tejas-TA User ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2026-05-25
41
pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine โ€” no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

pegainfer-project
โญ 748 Rust ๐Ÿ‘ค pegainfer-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-17
42
abhishekkrthakur
โญ 728 Python ๐Ÿ‘ค abhishekkrthakur User ๐Ÿ• Updated 2024-09-14 ๐Ÿ“… Created 2021-10-26
43
ServerlessLLM/ServerlessLLM

Serverless LLM Serving for Everyone.

ServerlessLLM
โญ 720 Python ๐Ÿ‘ค ServerlessLLM Organization ๐Ÿ• Updated 2026-09-04 ๐Ÿ“… Created 2024-01-23
44
kossisoroyce/timber

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

kossisoroyce
โญ 692 Python ๐Ÿ‘ค kossisoroyce User ๐Ÿ• Updated 2026-04-16 ๐Ÿ“… Created 2026-02-27
46
mlfoundations/evalchemy

Automatic evals for LLMs

mlfoundations
โญ 617 HTML ๐Ÿ‘ค mlfoundations Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2024-11-15
47
eightBEC/fastapi-ml-skeleton

FastAPI Skeleton App to serve machine learning models production-ready.

eightBEC
โญ 601 Python ๐Ÿ‘ค eightBEC User ๐Ÿ• Updated 2026-01-08 ๐Ÿ“… Created 2020-02-15
48
predibase/llm_distillation_playbook

Best practices for distilling large language models.

predibase
โญ 596 Jupyter Notebook ๐Ÿ‘ค predibase Organization ๐Ÿ• Updated 2024-02-01 ๐Ÿ“… Created 2023-12-06
49
fufankeji/DeepSeek-OCR-Web

Out-of-the-box DeepSeek OCR document parsing Web Studio

fufankeji
โญ 568 TypeScript ๐Ÿ‘ค fufankeji User ๐Ÿ• Updated 2025-10-27 ๐Ÿ“… Created 2025-10-23
โญ 553 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-08-27 ๐Ÿ“… Created 2024-05-22

Frequently Asked Questions

1

What is the most popular #model-serving repository on GitHub?

The top ranked open source #model-serving project on GitHub is vllm-project/vllm with 97,302 stars. A high-throughput and memory-efficient inference and serving engine for LLMs

2

Which programming languages are most used in #model-serving repositories?

Among the top ranked #model-serving projects on GitHub, the most common programming languages are Python, Go, Rust, Jupyter Notebook.

3

How is the #model-serving repository ranking calculated?

We track 51 active open source repositories tagged with #model-serving (500+ stars), totaling 211,758 stars with an average of 4,152 stars per repository. Rankings update daily from the GitHub API.