Git Stars Git Stars

#inference-engine Repositories

Discover 40 active open source GitHub repositories tagged with #inference-engine, accumulating 102,063 total stars (averaging 2,551 stars per project). Leading this topic is huggingface/candle with 21,367 stars.

40
Total Repositories
102,063
Total Stars
2,551
Avg Stars/Repo
1
huggingface/candle

Minimalist ML framework for Rust

huggingface
โญ 21,367 Rust ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-06-19
2
halfrost/Halfrost-Field

โœ๐Ÿป Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field: Source Code Analysis, System Design & Engineering Practice Notes

halfrost
โญ 13,259 Go ๐Ÿ‘ค halfrost User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2017-04-02
3
FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

FareedKhan-dev
โญ 12,523 C ๐Ÿ‘ค FareedKhan-dev User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-08-01
4
bbycroft/llm-viz

3D Visualization of an GPT-style LLM

bbycroft
โญ 5,599 TypeScript ๐Ÿ‘ค bbycroft User ๐Ÿ• Updated 2026-08-10 ๐Ÿ“… Created 2023-03-01
5
FedML-AI/FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

FedML-AI
โญ 4,068 Python ๐Ÿ‘ค FedML-AI Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2020-07-21
6
zjhellofss/KuiperInfer

Good projects for campus recruitment, autumn recruitment, spring recruitment, internships! Guiding you to implement a high-performance deep learning inference library from scratch, supporting inference for large models like llama2, Unet, YoloV5, Resnet, etc. Implement a high-performance deep learning inference library step by step

zjhellofss
โญ 3,444 C++ ๐Ÿ‘ค zjhellofss User ๐Ÿ• Updated 2025-06-22 ๐Ÿ“… Created 2022-11-21
7
Luce-Org/lucebox

Lucebox inference engine for heterogeneous hardware & consumer GPUs

Luce-Org
โญ 3,024 C++ ๐Ÿ‘ค Luce-Org Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-03
8
xdit-project/xDiT

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism

xdit-project
โญ 2,755 Python ๐Ÿ‘ค xdit-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-03-19
9
youssofal/MTPLX

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

youssofal
โญ 2,729 Python ๐Ÿ‘ค youssofal User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-05-02
10
hyperjumptech/grule-rule-engine

Rule engine implementation in Golang

hyperjumptech
โญ 2,539 Go ๐Ÿ‘ค hyperjumptech Organization ๐Ÿ• Updated 2026-02-10 ๐Ÿ“… Created 2019-12-13
11
datawhalechina/zero-to-sglang

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

datawhalechina
โญ 2,306 Python ๐Ÿ‘ค datawhalechina Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-27
12
ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

ovg-project
โญ 2,051 Python ๐Ÿ‘ค ovg-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-27
13
siliconflow/onediff

OneDiff: An out-of-the-box acceleration library for diffusion models.

siliconflow
โญ 1,961 Jupyter Notebook ๐Ÿ‘ค siliconflow Organization ๐Ÿ• Updated 2025-12-04 ๐Ÿ“… Created 2022-09-21
14
dphnAI/sonar

Large-scale LLM inference engine

dphnAI
โญ 1,911 Python ๐Ÿ‘ค dphnAI Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-06-23
15
nobodywho-ooo/nobodywho

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

nobodywho-ooo
โญ 1,699 Rust ๐Ÿ‘ค nobodywho-ooo Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-09-03
16
xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.

xLLM-AI
โญ 1,626 C++ ๐Ÿ‘ค xLLM-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-12
17
peonist-ai/halogen-flash-server

Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.

peonist-ai
โญ 1,412 Shell ๐Ÿ‘ค peonist-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-08-26
18
simplexai-labs/PolyStrata

Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder, and MiniCPM-V on a second engine built for PolyStrata. One-click installation for Windows and Linux, with an OpenAI/Anthropic API on localhost.

simplexai-labs
โญ 1,324 C++ ๐Ÿ‘ค simplexai-labs Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-10-06
19
qualcomm/ai-hub-models

Qualcommยฎ AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcommยฎ devices.

qualcomm
โญ 1,269 Python ๐Ÿ‘ค qualcomm Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-20
20
RainBowLuoCS/Gamer

Official Implementation of Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agents

RainBowLuoCS
โญ 1,179 ๐Ÿ‘ค RainBowLuoCS User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-10-30
21
PaddlePaddle/Paddle.js

Paddle.js is a web project for Baidu PaddlePaddle, which is an open source deep learning framework running in the browser. Paddle.js can either load a pre-trained model, or transforming a model from paddle-hub with model transforming tools provided by Paddle.js. It could run in every browser with WebGL/WebGPU/WebAssembly supported. It could also run in Baidu Smartprogram and WX miniprogram.

PaddlePaddle
โญ 1,071 JavaScript ๐Ÿ‘ค PaddlePaddle Organization ๐Ÿ• Updated 2024-04-03 ๐Ÿ“… Created 2020-03-26
22
wangeai/K3Flight

Run Kimi K3 locally on CPU with ~55GB measured runtime RAM. A single-file Linux inference server powered by cPilot Runtime.

wangeai
โญ 1,019 HTML ๐Ÿ‘ค wangeai Organization ๐Ÿ• Updated 2026-07-31 ๐Ÿ“… Created 2026-07-30
23
jjang-ai/mlxstudio

MLX Studio - Easiest way to run LLMs on your Mac. All-in-one engine.

jjang-ai
โญ 992 ๐Ÿ‘ค jjang-ai User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-03-05
24
zhihu/ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

zhihu
โญ 906 C++ ๐Ÿ‘ค zhihu Organization ๐Ÿ• Updated 2026-03-18 ๐Ÿ“… Created 2024-12-06
25
insight-platform/Savant

Python Computer Vision & Video Analytics Framework With Batteries Included

insight-platform
โญ 868 Python ๐Ÿ‘ค insight-platform Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2022-05-07
26
mit-han-lab/omniserve

[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention

mit-han-lab
โญ 840 C++ ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2025-03-06 ๐Ÿ“… Created 2024-05-06
27
pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine โ€” no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

pegainfer-project
โญ 748 Rust ๐Ÿ‘ค pegainfer-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-17
28
ARahim3/mlx-dspark

Up to 4ร— faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ€” Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

ARahim3
โญ 743 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2026-06-29
29
prabanta-dev/janas

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.

prabanta-dev
โญ 738 C ๐Ÿ‘ค prabanta-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-20
30
zhongkaifu/TensorSharp

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, an iPhone app, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows, macOS, iOS, and Linux with full GPU capability.

zhongkaifu
โญ 681 C# ๐Ÿ‘ค zhongkaifu User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-03
31
carloslfu/slotstream

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16โ€“64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

carloslfu
โญ 674 Swift ๐Ÿ‘ค carloslfu User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-28
32
andrewkchan/yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

andrewkchan
โญ 597 C++ ๐Ÿ‘ค andrewkchan User ๐Ÿ• Updated 2025-09-13 ๐Ÿ“… Created 2024-10-02
33
pylint-dev/astroid

A common base representation of python source code for pylint and other projects

pylint-dev
โญ 589 Python ๐Ÿ‘ค pylint-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2015-12-08
34
zjhellofss/KuiperLLama

Campus recruitment, fall recruitment, spring recruitment, great internship projects, guiding you to implement from scratch a large model inference framework supporting Llama2/3 and Qwen2.5.

zjhellofss
โญ 581 C++ ๐Ÿ‘ค zjhellofss User ๐Ÿ• Updated 2025-10-28 ๐Ÿ“… Created 2024-04-25
35
StartLuxLabs/StartLux-Prism

Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files run on int8 and 4-bit tensor cores behind the /v1/systemone API, delivering up to 5.3ร— lower latency than vLLM in FP8 with bit-identical answers.

StartLuxLabs
โญ 560 Python ๐Ÿ‘ค StartLuxLabs Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2026-10-13
36
zengxiao-he/tessera

From teacher to tiles โ€” a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

zengxiao-he
โญ 556 Python ๐Ÿ‘ค zengxiao-he User ๐Ÿ• Updated 2026-10-09 ๐Ÿ“… Created 2026-06-05
37
SearchSavior/OpenArc

Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.

SearchSavior
โญ 551 Python ๐Ÿ‘ค SearchSavior User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-02-11
38
OmniJev/Flide

๐Ÿ”ฅ๐Ÿ”ฅ The extremely fast inference engine for decision models

OmniJev
โญ 550 Python ๐Ÿ‘ค OmniJev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-24
39
Generated-Computers/Locals

Run many AI agents at once on your Mac, up to 3x faster.

Generated-Computers
โญ 538 Python ๐Ÿ‘ค Generated-Computers Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-12
40
brontoguana/krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

brontoguana
โญ 530 C++ ๐Ÿ‘ค brontoguana User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2026-02-09

Frequently Asked Questions

1

What is the most popular #inference-engine repository on GitHub?

The top ranked open source #inference-engine project on GitHub is huggingface/candle with 21,367 stars. Minimalist ML framework for Rust

2

Which programming languages are most used in #inference-engine repositories?

Among the top ranked #inference-engine projects on GitHub, the most common programming languages are Python, C++, Rust, Go.

3

How is the #inference-engine repository ranking calculated?

We track 40 active open source repositories tagged with #inference-engine (500+ stars), totaling 102,063 stars with an average of 2,551 stars per repository. Rankings update daily from the GitHub API.