Git Stars Git Stars

#inference Repositories

Discover 144 active open source GitHub repositories tagged with #inference, accumulating 870,238 total stars (averaging 6,043 stars per project). Leading this topic is vllm-project/vllm with 97,302 stars.

144
Total Repositories
870,238
Total Stars
6,043
Avg Stars/Repo
1
vllm-project/vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project
โญ 97,302 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-02-09
2
ultralytics/yolov5

Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.

ultralytics
โญ 58,290 Python ๐Ÿ‘ค ultralytics Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2020-05-18
3
ggml-org/whisper.cpp

Port of OpenAI's Whisper model in C/C++

ggml-org
โญ 55,392 C++ ๐Ÿ‘ค ggml-org Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-09-25
4
deepspeedai/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

deepspeedai
โญ 43,348 Python ๐Ÿ‘ค deepspeedai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-01-23
5
hpcaitech/ColossalAI

Making large AI models cheaper, faster and more accessible

hpcaitech
โญ 41,421 Python ๐Ÿ‘ค hpcaitech Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2021-10-28
6
sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

sgl-project
โญ 38,899 Python ๐Ÿ‘ค sgl-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-01-08
7
google-ai-edge/mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

google-ai-edge
โญ 37,769 C++ ๐Ÿ‘ค google-ai-edge Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-06-13
8
SYSTRAN/faster-whisper

Faster Whisper transcription with CTranslate2

SYSTRAN
โญ 26,924 Python ๐Ÿ‘ค SYSTRAN Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2023-02-11
9
Tencent/ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Tencent
โญ 24,082 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2017-06-30
10
stas00/ml-engineering

Machine Learning Engineering Open Book

stas00
โญ 19,738 Python ๐Ÿ‘ค stas00 User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-09-02
11
FlashML-org/FreeToken

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

FlashML-org
โญ 17,522 Python ๐Ÿ‘ค FlashML-org User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-07-20
โญ 16,519 Python ๐Ÿ‘ค GeeeekExplorer User ๐Ÿ• Updated 2026-04-26 ๐Ÿ“… Created 2025-06-09
13
gvergnaud/ts-pattern

๐ŸŽจ The exhaustive Pattern Matching library for TypeScript, with smart type inference.

gvergnaud
โญ 15,224 TypeScript ๐Ÿ‘ค gvergnaud User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-05-24
14
NVIDIA/TensorRT

NVIDIAยฎ TensorRTโ„ข is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

NVIDIA
โญ 13,475 C++ ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2019-05-02
15
LMCache/LMCache

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

LMCache
โญ 12,405 Python ๐Ÿ‘ค LMCache Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-28
16
openvinotoolkit/openvino

OpenVINOโ„ข is an open source toolkit for optimizing and deploying AI inference

openvinotoolkit
โญ 11,869 C++ ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-10-15
17
triton-inference-server/server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

triton-inference-server
โญ 11,156 Python ๐Ÿ‘ค triton-inference-server Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2018-10-04
18
aws/amazon-sagemaker-examples

Example ๐Ÿ““ Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using ๐Ÿง  Amazon SageMaker.

aws
โญ 11,020 Jupyter Notebook ๐Ÿ‘ค aws Organization ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2017-10-23
19
huggingface/text-generation-inference

Large Language Model Text Generation Inference

huggingface
โญ 10,812 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-01-08 ๐Ÿ“… Created 2022-10-08
20
ultralytics/yolov3

PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.

ultralytics
โญ 10,647 Python ๐Ÿ‘ค ultralytics Organization ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2018-08-26
21
RunanywhereAI/runanywhere-sdks

Production ready toolkit to run AI locally

RunanywhereAI
โญ 10,354 C++ ๐Ÿ‘ค RunanywhereAI Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2025-07-22
22
xorbitsai/inference

Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.

xorbitsai
โญ 9,667 Python ๐Ÿ‘ค xorbitsai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-06-14
23
oumi-ai/oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

oumi-ai
โญ 9,441 Python ๐Ÿ‘ค oumi-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-07
24
dusty-nv/jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.

dusty-nv
โญ 9,015 C++ ๐Ÿ‘ค dusty-nv User ๐Ÿ• Updated 2025-10-16 ๐Ÿ“… Created 2016-07-30
25
vllm-project/vllm-omni

A framework for efficient model inference with omni-modality models

vllm-project
โญ 7,884 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-09-11
26
Human-Agent-Society/reef

Infrastructure for continually self-improving agents

Human-Agent-Society
โญ 7,652 Python ๐Ÿ‘ค Human-Agent-Society Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-08-31
27
mizorewww/laya-mlx

Native MLX runtime for Laya typed decision models โ€” 7โ€“14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.

mizorewww
โญ 7,115 Python ๐Ÿ‘ค mizorewww User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-09-19
28
kvcache-ai/Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

kvcache-ai
โญ 7,100 C++ ๐Ÿ‘ค kvcache-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-06-25
29
gcanti/io-ts

Runtime type system for IO decoding/encoding

gcanti
โญ 6,812 TypeScript ๐Ÿ‘ค gcanti User ๐Ÿ• Updated 2024-12-10 ๐Ÿ“… Created 2017-01-28
30
vllm-project/semantic-router

An open, programmable decision layer for models and compute.

vllm-project
โญ 6,514 Go ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-26
31
argmaxinc/argmax-oss-swift

On-device Speech AI for Apple Silicon

argmaxinc
โญ 6,471 Swift ๐Ÿ‘ค argmaxinc Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2024-01-26
32
Trusted-AI/adversarial-robustness-toolbox

Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams

Trusted-AI
โญ 6,322 Python ๐Ÿ‘ค Trusted-AI Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2018-03-15
33
gpustack/gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

gpustack
โญ 6,097 Python ๐Ÿ‘ค gpustack Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-11
34
llm-d/llm-d

State-of-the-art inference performance with modern accelerators on Kubernetes

llm-d
โญ 5,325 Python ๐Ÿ‘ค llm-d Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-29
35
superduper-io/superduper

Superduper: End-to-end framework for building custom AI applications and agents.

superduper-io
โญ 5,317 Python ๐Ÿ‘ค superduper-io Organization ๐Ÿ• Updated 2025-09-01 ๐Ÿ“… Created 2022-08-30
36
tencentmusic/cube-studio

cube studio Open-Source Cloud-Native All-in-One Machine Learning/Deep Learning/Large Model AI Platform, MLOps algorithm link full process, computing power rental platform, notebook online development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference service VGPU virtualization, edge computing, annotation platform automated annotation, deepseek and other large models sft fine-tuning/reward model/reinforcement learning training, vllm/ollama/mindie large model multi-machine inference, private knowledge base, AI model market, supports domestic CPU/GPU/NPU Ascend ecosystem, supports RDMA, supports distributed training such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano

tencentmusic
โญ 5,085 ๐Ÿ‘ค tencentmusic Organization ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2021-08-17
37
NVIDIA-AI-IOT/torch2trt

An easy to use PyTorch to TensorRT converter

NVIDIA-AI-IOT
โญ 4,840 Python ๐Ÿ‘ค NVIDIA-AI-IOT Organization ๐Ÿ• Updated 2024-08-17 ๐Ÿ“… Created 2019-04-27
38
hao-ai-lab/FastVideo

A unified inference and post-training framework for accelerated video generation.

hao-ai-lab
โญ 4,833 Python ๐Ÿ‘ค hao-ai-lab Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-10-24
39
OpenNMT/CTranslate2

Fast inference engine for Transformer models

OpenNMT
โญ 4,758 C++ ๐Ÿ‘ค OpenNMT Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-09-23
40
Tencent/TNN

TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile, desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts

Tencent
โญ 4,635 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2025-05-09 ๐Ÿ“… Created 2020-05-29
41
typedb/typedb

TypeDB: Built for systems, not records

typedb
โญ 4,524 Rust ๐Ÿ‘ค typedb Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2016-07-11
42
openvinotoolkit/open_model_zoo

Pre-trained Deep Learning models and demos (high quality and extremely fast)

openvinotoolkit
โญ 4,440 Python ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2018-10-15
43
zml/zml

Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild

zml
โญ 4,292 Zig ๐Ÿ‘ค zml Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-09-17
44
uxlfoundation/oneDNN

Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.

uxlfoundation
โญ 4,094 C++ ๐Ÿ‘ค uxlfoundation Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-05-09
45
OpenCSGs/csghub

CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! โญ๏ธ

OpenCSGs
โญ 4,083 Vue ๐Ÿ‘ค OpenCSGs Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2024-01-12
46
FlashML-org/FreeVideo

Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.

FlashML-org
โญ 4,049 Python ๐Ÿ‘ค FlashML-org User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-13
47
PaddlePaddle/FastDeploy

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

PaddlePaddle
โญ 3,722 Python ๐Ÿ‘ค PaddlePaddle Organization ๐Ÿ• Updated 2026-09-24 ๐Ÿ“… Created 2022-06-27
48
superlinked/sie

Open-source inference server and production cluster for all the models your agent needs.

superlinked
โญ 3,534 Python ๐Ÿ‘ค superlinked Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-07
49
huggingface/optimum

๐Ÿš€ Accelerate inference and training of ๐Ÿค— Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools

huggingface
โญ 3,525 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2021-07-20
50
zjhellofss/KuiperInfer

Good projects for campus recruitment, autumn recruitment, spring recruitment, internships! Guiding you to implement a high-performance deep learning inference library from scratch, supporting inference for large models like llama2, Unet, YoloV5, Resnet, etc. Implement a high-performance deep learning inference library step by step

zjhellofss
โญ 3,444 C++ ๐Ÿ‘ค zjhellofss User ๐Ÿ• Updated 2025-06-22 ๐Ÿ“… Created 2022-11-21

Frequently Asked Questions

1

What is the most popular #inference repository on GitHub?

The top ranked open source #inference project on GitHub is vllm-project/vllm with 97,302 stars. A high-throughput and memory-efficient inference and serving engine for LLMs

2

Which programming languages are most used in #inference repositories?

Among the top ranked #inference projects on GitHub, the most common programming languages are Python, C++, TypeScript, Jupyter Notebook.

3

How is the #inference repository ranking calculated?

We track 144 active open source repositories tagged with #inference (500+ stars), totaling 870,238 stars with an average of 6,043 stars per repository. Rankings update daily from the GitHub API.