Git Stars Git Stars

#inference Repositories

144
Total Repositories
870,238
Total Stars
6,043
Avg Stars/Repo
51
trymirai/uzu

A high-performance inference engine for AI models

trymirai
โญ 3,378 Rust ๐Ÿ‘ค trymirai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-06-23
52
openvinotoolkit/openvino_notebooks

๐Ÿ“š Jupyter notebook tutorials for OpenVINOโ„ข

openvinotoolkit
โญ 3,245 Jupyter Notebook ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-03-11
53
vllm-project/vllm-ascend

Community maintained hardware plugin for vLLM on Huawei Ascend

vllm-project
โญ 3,128 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-01-29
54
pytorch/ao

PyTorch native quantization for training and inference

pytorch
โญ 3,032 Python ๐Ÿ‘ค pytorch Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-11-03
55
pykeio/ort

Fast ML inference & training for ONNX models in Rust

pykeio
โญ 2,626 Rust ๐Ÿ‘ค pykeio Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2022-11-26
56
data-infra/cube-studio

CubeStudio is an open-source cloud-native one-stop machine learning/deep learning/large model AI platform/MaaS/MLOps/artificial intelligence platform/training and inference platform, covering the full algorithm pipeline, multi-tenancy, computing power leasing platform, token relay, drag-and-drop task flow pipeline orchestration, multi-machine multi-card distributed training, hyperparameter search, inference services, VGPU virtualization, cloud-edge-device collaboration, edge computing, automated annotation platform, SFT fine-tuning/reward model/reinforcement learning training for large models like DeepSeek, multi-machine inference for large models with vLLM/Ollama/MindIE, private knowledge base LLMOps intelligent agents, AI model marketplace, support for domestic heterogeneous computing power scheduling (Ascend, Cambricon, Hygon, Moore Threads, Muxi, etc.), support for IB/RoCE/RDMA, and Xinchuang support.

data-infra
โญ 2,615 Python ๐Ÿ‘ค data-infra User ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2024-07-02
57
huggingface/huggingface.js

Use Hugging Face with JavaScript

huggingface
โญ 2,613 TypeScript ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-02-06
58
ddalcu/mlx-serve

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation.

ddalcu
โญ 2,528 Zig ๐Ÿ‘ค ddalcu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-17
59
google/XNNPACK

High-efficiency floating-point neural network inference operators for mobile, server, and Web

google
โญ 2,506 C ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-09-13
60
roboflow/inference

Turn any computer or edge device into a command center for your computer vision projects.

roboflow
โญ 2,500 Python ๐Ÿ‘ค roboflow Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-07-31
61
theagentrouter/agent-router

Manages Unified Access to Generative AI Services built on Envoy Gateway

theagentrouter
โญ 2,428 Go ๐Ÿ‘ค theagentrouter Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-10-21
62
adithya-s-k/AI-Engineering.academy

Mastering Applied AI, One Concept at a Time

adithya-s-k
โญ 2,388 Jupyter Notebook ๐Ÿ‘ค adithya-s-k User ๐Ÿ• Updated 2026-02-27 ๐Ÿ“… Created 2023-10-05
63
dstackai/dstack

A unified orchestration layer for heterogeneous AI compute. It standardizes how to manage compute and run training and inference on GPU clouds, Kubernetes, VMs, or bare-metal clusters.

dstackai
โญ 2,319 Python ๐Ÿ‘ค dstackai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-01-04
64
mozilla-ai/any-llm

Communicate with an LLM provider using a single interface

mozilla-ai
โญ 2,265 Python ๐Ÿ‘ค mozilla-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-14
65
RightNow-AI/picolm

Run a 1-billion parameter LLM on a $10 board with 256MB RAM.

RightNow-AI
โญ 2,218 C ๐Ÿ‘ค RightNow-AI Organization ๐Ÿ• Updated 2026-02-22 ๐Ÿ“… Created 2026-02-19
66
magenta/magenta-js

Magenta.js: Music and Art Generation with Machine Learning in the browser

magenta
โญ 2,136 TypeScript ๐Ÿ‘ค magenta Organization ๐Ÿ• Updated 2026-06-22 ๐Ÿ“… Created 2018-03-26
67
tairov/llama2.mojo

Inference Llama 2 in one file of pure ๐Ÿ”ฅ

tairov
โญ 2,126 Mojo ๐Ÿ‘ค tairov User ๐Ÿ• Updated 2026-09-30 ๐Ÿ“… Created 2023-09-10
68
deepspeedai/DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

deepspeedai
โญ 2,107 Python ๐Ÿ‘ค deepspeedai Organization ๐Ÿ• Updated 2025-06-30 ๐Ÿ“… Created 2022-03-23
69
microsoft/aici

AICI: Prompts as (Wasm) Programs

microsoft
โญ 2,077 Rust ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-01-22 ๐Ÿ“… Created 2023-09-26
70
SemiAnalysisAI/InferenceX

Open-Source Inference Research Platform

SemiAnalysisAI
โญ 2,074 Python ๐Ÿ‘ค SemiAnalysisAI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-12
71
sgl-project/sglang-omni

High-Performance Serving Runtime for Interactive Superintelligence Systems

sgl-project
โญ 1,882 Python ๐Ÿ‘ค sgl-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-01-07
72
AgibotTech/agibot_x1_infer

The inference module for AgiBot X1.

AgibotTech
โญ 1,817 C++ ๐Ÿ‘ค AgibotTech Organization ๐Ÿ• Updated 2025-04-03 ๐Ÿ“… Created 2024-10-23
73
StartLuxLabs/StartLux-Decision

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

StartLuxLabs
โญ 1,780 Python ๐Ÿ‘ค StartLuxLabs Organization ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2026-09-29
74
theopenco/llmgateway

Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.

theopenco
โญ 1,755 TypeScript ๐Ÿ‘ค theopenco Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-04-12
75
Maratyszcza/NNPACK

Acceleration package for neural networks on multi-core CPUs

Maratyszcza
โญ 1,702 C ๐Ÿ‘ค Maratyszcza User ๐Ÿ• Updated 2024-06-11 ๐Ÿ“… Created 2016-03-21
76
ELS-RD/transformer-deploy

Efficient, scalable and enterprise-grade CPU/GPU inference server for ๐Ÿค— Hugging Face transformer models ๐Ÿš€

ELS-RD
โญ 1,689 Python ๐Ÿ‘ค ELS-RD Organization ๐Ÿ• Updated 2024-10-23 ๐Ÿ“… Created 2021-10-31
77
RunanywhereAI/trnrt

The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.

RunanywhereAI
โญ 1,653 Python ๐Ÿ‘ค RunanywhereAI Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-10-30
78
xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.

xLLM-AI
โญ 1,626 C++ ๐Ÿ‘ค xLLM-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-12
79
Delta-ML/delta

DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/

Delta-ML
โญ 1,602 Python ๐Ÿ‘ค Delta-ML Organization ๐Ÿ• Updated 2025-04-16 ๐Ÿ“… Created 2019-05-29
80
utkuozdemir/nvidia_gpu_exporter

NVIDIA GPU exporter for Prometheus using the nvidia-smi binary OR NVML

utkuozdemir
โญ 1,599 Go ๐Ÿ‘ค utkuozdemir User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-06-08
81
Tencent/TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

Tencent
โญ 1,546 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2025-07-18 ๐Ÿ“… Created 2020-04-20
โญ 1,503 Python ๐Ÿ‘ค Stability-AI Organization ๐Ÿ• Updated 2025-01-08 ๐Ÿ“… Created 2024-10-09
83
sybil-solutions/inference-optimisation-skill

Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

sybil-solutions
โญ 1,425 Python ๐Ÿ‘ค sybil-solutions Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-11-16
84
alibaba/rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

alibaba
โญ 1,402 Python ๐Ÿ‘ค alibaba Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-27
85
ebhy/budgetml

Deploy a ML inference service on a budget in less than 10 lines of code.

ebhy
โญ 1,345 Python ๐Ÿ‘ค ebhy Organization ๐Ÿ• Updated 2024-02-12 ๐Ÿ“… Created 2020-12-27
86
StarlightSearch/EmbedAnything

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust ๐Ÿฆ€

StarlightSearch
โญ 1,334 Rust ๐Ÿ‘ค StarlightSearch Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-03-31
87
qualcomm/ai-hub-models

Qualcommยฎ AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcommยฎ devices.

qualcomm
โญ 1,269 Python ๐Ÿ‘ค qualcomm Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-20
88
NVIDIA/kvpress

LLM KV cache compression made easy

NVIDIA
โญ 1,258 Python ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2024-11-06
89
FenTechSolutions/CausalDiscoveryToolbox

Package for causal inference in graphs and in the pairwise settings. Tools for graph structure recovery and dependencies are included.

FenTechSolutions
โญ 1,242 Python ๐Ÿ‘ค FenTechSolutions Organization ๐Ÿ• Updated 2025-10-13 ๐Ÿ“… Created 2017-05-31
90
run-house/kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

run-house
โญ 1,228 Python ๐Ÿ‘ค run-house Organization ๐Ÿ• Updated 2026-05-29 ๐Ÿ“… Created 2022-05-10
91
google
โญ 1,208 Go ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-21
92
jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

jmaczan
โญ 1,207 C++ ๐Ÿ‘ค jmaczan User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2026-02-09
93
chrishayuk/larql

Run, serve, query, and edit AI models โ€” a Rust model runtime built on VINDEX3, an executable, inspectable model format.

chrishayuk
โญ 1,196 Rust ๐Ÿ‘ค chrishayuk User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-27
94
usamahz/cpu-performance-engineering

A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.

usamahz
โญ 1,160 Python ๐Ÿ‘ค usamahz User ๐Ÿ• Updated 2026-10-08 ๐Ÿ“… Created 2026-09-17
95
receptron/laya

Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime

receptron
โญ 1,092 TypeScript ๐Ÿ‘ค receptron Organization ๐Ÿ• Updated 2026-10-17 ๐Ÿ“… Created 2026-09-19
96
Helldez/BigMoeOnEdge

Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, or edge device. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.

Helldez
โญ 1,066 C++ ๐Ÿ‘ค Helldez User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-07-10
97
jjang-ai/mlxstudio

MLX Studio - Easiest way to run LLMs on your Mac. All-in-one engine.

jjang-ai
โญ 992 ๐Ÿ‘ค jjang-ai User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-03-05
98
efeslab/Nanoflow

A throughput-oriented high-performance serving framework for LLMs

efeslab
โญ 978 Jupyter Notebook ๐Ÿ‘ค efeslab Organization ๐Ÿ• Updated 2026-03-29 ๐Ÿ“… Created 2024-08-19
99
openvinotoolkit/model_server

A scalable inference server for models optimized with OpenVINOโ„ข

openvinotoolkit
โญ 961 C++ ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-09-26
100
huawei-noah/bolt

Bolt is a deep learning library with high performance and heterogeneous flexibility.

huawei-noah
โญ 958 C++ ๐Ÿ‘ค huawei-noah Organization ๐Ÿ• Updated 2025-04-11 ๐Ÿ“… Created 2019-12-02