#inference-engine Repositories
Discover 40 active open source GitHub repositories tagged with #inference-engine, accumulating 102,063 total stars (averaging 2,551 stars per project). Leading this topic is huggingface/candle with 21,367 stars.
40 repositories total
โ๐ป Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field: Source Code Analysis, System Design & Engineering Practice Notes
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
Good projects for campus recruitment, autumn recruitment, spring recruitment, internships! Guiding you to implement a high-performance deep learning inference library from scratch, supporting inference for large models like llama2, Unet, YoloV5, Resnet, etc. Implement a high-performance deep learning inference library step by step
xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder, and MiniCPM-V on a second engine built for PolyStrata. One-click installation for Windows and Linux, with an OpenAI/Anthropic API on localhost.
Qualcommยฎ AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcommยฎ devices.
Official Implementation of Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agents
Paddle.js is a web project for Baidu PaddlePaddle, which is an open source deep learning framework running in the browser. Paddle.js can either load a pre-trained model, or transforming a model from paddle-hub with model transforming tools provided by Paddle.js. It could run in every browser with WebGL/WebGPU/WebAssembly supported. It could also run in Baidu Smartprogram and WX miniprogram.
Run Kimi K3 locally on CPU with ~55GB measured runtime RAM. A single-file Linux inference server powered by cPilot Runtime.
Python Computer Vision & Video Analytics Framework With Batteries Included
[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
Pure Rust + CUDA LLM inference engine โ no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Up to 4ร faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, an iPhone app, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows, macOS, iOS, and Linux with full GPU capability.
Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16โ64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
A common base representation of python source code for pylint and other projects
Campus recruitment, fall recruitment, spring recruitment, great internship projects, guiding you to implement from scratch a large model inference framework supporting Llama2/3 and Qwen2.5.
Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files run on int8 and 4-bit tensor cores behind the /v1/systemone API, delivering up to 5.3ร lower latency than vLLM in FP8 with bit-identical answers.
From teacher to tiles โ a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
Frequently Asked Questions
What is the most popular #inference-engine repository on GitHub?
The top ranked open source #inference-engine project on GitHub is huggingface/candle with 21,367 stars. Minimalist ML framework for Rust
Which programming languages are most used in #inference-engine repositories?
Among the top ranked #inference-engine projects on GitHub, the most common programming languages are Python, C++, Rust, Go.
How is the #inference-engine repository ranking calculated?
We track 40 active open source repositories tagged with #inference-engine (500+ stars), totaling 102,063 stars with an average of 2,551 stars per repository. Rankings update daily from the GitHub API.