Git Stars Git Stars

#llm-inference Repositories

202
Total Repositories
1.0M
Total Stars
5,180
Avg Stars/Repo
151
nace-ai/kapitsa

Blazing-fast Apple Silicon inference engine optimized for Qwen & Nemotron. Compatible with the OpenAI and Anthropic APIs. Built by the team at nace.ai

nace-ai
โญ 840 C++ ๐Ÿ‘ค nace-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-16
152
warpfront/hipfire

RDNA-native LLM inference engine in Rust.

warpfront
โญ 826 Rust ๐Ÿ‘ค warpfront Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-21
153
mukel/llama3.java

Llama 3+ inference in pure Java

mukel
โญ 818 Java ๐Ÿ‘ค mukel User ๐Ÿ• Updated 2026-04-24 ๐Ÿ“… Created 2024-04-25
154
vince-lam/awesome-local-llms

Compare compatible open-source local LLM projects by their metrics to assess popularity and activeness.

vince-lam
โญ 816 Python ๐Ÿ‘ค vince-lam User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2024-02-29
155
ARahim3/kaggle-tpu-lab

Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

ARahim3
โญ 812 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2026-08-31
156
tetherto/qvac

Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.

tetherto
โญ 806 TypeScript ๐Ÿ‘ค tetherto Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-01-03
157
SimpleBerry/LLaMA-O1

Large Reasoning Models

SimpleBerry
โญ 806 Python ๐Ÿ‘ค SimpleBerry Organization ๐Ÿ• Updated 2024-12-03 ๐Ÿ“… Created 2024-10-10
158
D-Keqi/mtla

MTLA: Multi-head Temporal Latent Attention

D-Keqi
โญ 761 Python ๐Ÿ‘ค D-Keqi User ๐Ÿ• Updated 2025-10-06 ๐Ÿ“… Created 2025-05-19
159
microsoft/BitBLAS

BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment.

microsoft
โญ 751 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-08-06 ๐Ÿ“… Created 2024-02-23
160
pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine โ€” no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

pegainfer-project
โญ 748 Rust ๐Ÿ‘ค pegainfer-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-17
161
ARahim3/mlx-dspark

Up to 4ร— faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ€” Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.

ARahim3
โญ 743 Python ๐Ÿ‘ค ARahim3 User ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2026-06-29
162
Avarok-Cybersecurity/atlas

Pure Rust Inference Engine

Avarok-Cybersecurity
โญ 737 Rust ๐Ÿ‘ค Avarok-Cybersecurity Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-05-05
163
ghimiresunil/LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

ghimiresunil
โญ 734 Jupyter Notebook ๐Ÿ‘ค ghimiresunil User ๐Ÿ• Updated 2026-03-13 ๐Ÿ“… Created 2023-05-28
164
stoyan-stoyanov/llmflows

LLMFlows - Simple, Explicit and Transparent LLM Apps

stoyan-stoyanov
โญ 706 Python ๐Ÿ‘ค stoyan-stoyanov User ๐Ÿ• Updated 2025-02-20 ๐Ÿ“… Created 2023-06-26
165
feifeibear/long-context-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

feifeibear
โญ 697 Python ๐Ÿ‘ค feifeibear User ๐Ÿ• Updated 2026-09-16 ๐Ÿ“… Created 2024-03-27
โญ 696 Go ๐Ÿ‘ค deanxv User ๐Ÿ• Updated 2026-02-27 ๐Ÿ“… Created 2024-12-23
167
MDK8888/GPTFast

Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.

MDK8888
โญ 687 Python ๐Ÿ‘ค MDK8888 User ๐Ÿ• Updated 2024-08-22 ๐Ÿ“… Created 2024-02-18
168
google-deepmind/recurrentgemma

Open weights language model from Google DeepMind, based on Griffin.

google-deepmind
โญ 686 Python ๐Ÿ‘ค google-deepmind Organization ๐Ÿ• Updated 2026-02-06 ๐Ÿ“… Created 2024-03-25
169
carloslfu/slotstream

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16โ€“64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

carloslfu
โญ 674 Swift ๐Ÿ‘ค carloslfu User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-28
170
run-ai/genv

GPU environment and cluster management with LLM support

run-ai
โญ 657 Python ๐Ÿ‘ค run-ai Organization ๐Ÿ• Updated 2024-05-16 ๐Ÿ“… Created 2022-09-04
172
NightMean/OlliteRT

Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source

NightMean
โญ 645 Kotlin ๐Ÿ‘ค NightMean User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-04-06
173
NotPunchnox/rkllama

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support (rkllm)

NotPunchnox
โญ 631 Python ๐Ÿ‘ค NotPunchnox User ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2024-12-30
174
sixgpt/miner

Official Miner for SixGPT

sixgpt
โญ 611 Shell ๐Ÿ‘ค sixgpt Organization ๐Ÿ• Updated 2024-12-19 ๐Ÿ“… Created 2024-10-06
175
felladrin/MiniSearch

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space

felladrin
โญ 611 TypeScript ๐Ÿ‘ค felladrin User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-10-14
176
jaylfc/taOS

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).

jaylfc
โญ 609 Python ๐Ÿ‘ค jaylfc User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-05
177
andrewkchan/yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

andrewkchan
โญ 597 C++ ๐Ÿ‘ค andrewkchan User ๐Ÿ• Updated 2025-09-13 ๐Ÿ“… Created 2024-10-02
178
huggingface/swift-chat

Mac app to demonstrate swift-transformers

huggingface
โญ 592 Swift ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2024-07-29 ๐Ÿ“… Created 2023-05-11
179
zjhellofss/KuiperLLama

Campus recruitment, fall recruitment, spring recruitment, great internship projects, guiding you to implement from scratch a large model inference framework supporting Llama2/3 and Qwen2.5.

zjhellofss
โญ 581 C++ ๐Ÿ‘ค zjhellofss User ๐Ÿ• Updated 2025-10-28 ๐Ÿ“… Created 2024-04-25
180
kacper-daftcode/vLLM-Moet

Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and 2-bit experts on vLLM v0.24.0

kacper-daftcode
โญ 580 Sass ๐Ÿ‘ค kacper-daftcode User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-06-15
181
SakanaAI/treequest

A Tree Search Library with Flexible API for LLM Inference-Time Scaling

SakanaAI
โญ 580 Python ๐Ÿ‘ค SakanaAI Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2025-06-30
182
devflowinc/uzi

CLI for running large numbers of coding agents in parallel with git worktrees

devflowinc
โญ 579 Go ๐Ÿ‘ค devflowinc Organization ๐Ÿ• Updated 2025-06-04 ๐Ÿ“… Created 2025-05-27
183
rohan-paul/LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

rohan-paul
โญ 574 Jupyter Notebook ๐Ÿ‘ค rohan-paul User ๐Ÿ• Updated 2025-04-01 ๐Ÿ“… Created 2023-10-22
184
StartLuxLabs/StartLux-Prism

Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files run on int8 and 4-bit tensor cores behind the /v1/systemone API, delivering up to 5.3ร— lower latency than vLLM in FP8 with bit-identical answers.

StartLuxLabs
โญ 560 Python ๐Ÿ‘ค StartLuxLabs Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2026-10-13
185
yassa9/qwen600

Static suckless single batch CUDA-only qwen3-0.6B mini inference engine

yassa9
โญ 559 Cuda ๐Ÿ‘ค yassa9 User ๐Ÿ• Updated 2025-09-08 ๐Ÿ“… Created 2025-09-06
186
01-ai/Yi-1.5

Yi-1.5 is an upgraded version of Yi, delivering stronger performance in coding, math, reasoning, and instruction-following capability.

01-ai
โญ 557 ๐Ÿ‘ค 01-ai Organization ๐Ÿ• Updated 2024-11-11 ๐Ÿ“… Created 2024-05-10
โญ 557 Python ๐Ÿ‘ค Snowflake-Labs Organization ๐Ÿ• Updated 2024-08-16 ๐Ÿ“… Created 2024-04-22
โญ 553 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-08-27 ๐Ÿ“… Created 2024-05-22
189
Noeda/rllama

Rust+OpenCL+AVX2 implementation of LLaMA inference code

Noeda
โญ 551 Rust ๐Ÿ‘ค Noeda User ๐Ÿ• Updated 2024-02-12 ๐Ÿ“… Created 2023-03-11
190
OmniJev/Flide

๐Ÿ”ฅ๐Ÿ”ฅ The extremely fast inference engine for decision models

OmniJev
โญ 550 Python ๐Ÿ‘ค OmniJev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-24
191
Indras-Mirror/Xolotl

Xolotl: run DeepSeek-V4.1-Flash, all 552B parameters, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s agent, ~41 fast, 512K context. Plus Inkling-2-Mini (Thinking Machines, own engine + MTP drafter), GLM-5.3-Flash, Qwen3.8, LFM3-Omni. MoE experts tiered across VRAM, RAM, and NVMe (Xolotl Direct). OpenAI + Anthropic API.

Indras-Mirror
โญ 549 C++ ๐Ÿ‘ค Indras-Mirror User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-06
192
mit-han-lab/duo-attention

[ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

mit-han-lab
โญ 541 Python ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2025-02-10 ๐Ÿ“… Created 2024-10-15
193
microsoft/sarathi-serve

A low-latency & high-throughput serving engine for LLMs

microsoft
โญ 539 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-11-02
194
v-modal/awesome-jev-tools

A curated list of tools built for Jev โ€” TypeSafe AI's System One model for typed decisions.

v-modal
โญ 537 ๐Ÿ‘ค v-modal Organization ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2026-09-19
195
brontoguana/krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

brontoguana
โญ 530 C++ ๐Ÿ‘ค brontoguana User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2026-02-09
196
ome-projects/ome

Open Model Engine (OME) โ€” Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

ome-projects
โญ 529 Go ๐Ÿ‘ค ome-projects Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-05-19
197
amitshekhariitbhu/llm-inference-engineering

Learn LLM Inference Engineering step by stepโ€”from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

amitshekhariitbhu
โญ 523 Markdown ๐Ÿ‘ค amitshekhariitbhu User ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2026-08-18
198
huawei-csl/KVarN

KVarN is a native vLLM KV-cache quantization backend for your agents: 3โ€“5ร— more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

huawei-csl
โญ 521 Python ๐Ÿ‘ค huawei-csl Organization ๐Ÿ• Updated 2026-06-22 ๐Ÿ“… Created 2026-05-29
199
expectedparrot/edsl

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.

expectedparrot
โญ 521 Python ๐Ÿ‘ค expectedparrot Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-01-02
200
turboderp-org/exui

Web UI for ExLlamaV2

turboderp-org
โญ 513 JavaScript ๐Ÿ‘ค turboderp-org Organization ๐Ÿ• Updated 2025-02-05 ๐Ÿ“… Created 2023-10-29