Git Stars Git Stars

#llm-inference Repositories

Discover 202 active open source GitHub repositories tagged with #llm-inference, accumulating 1,046,424 total stars (averaging 5,180 stars per project). Leading this topic is deepseek-ai/DeepSeek-V3 with 104,417 stars.

202
Total Repositories
1.0M
Total Stars
5,180
Avg Stars/Repo
โญ 104,417 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-08-28 ๐Ÿ“… Created 2024-12-26
2
nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

nomic-ai
โญ 77,352 C++ ๐Ÿ‘ค nomic-ai Organization ๐Ÿ• Updated 2025-05-27 ๐Ÿ“… Created 2023-03-27
3
oobabooga/textgen

Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.

oobabooga
โญ 47,736 Python ๐Ÿ‘ค oobabooga User ๐Ÿ• Updated 2026-08-17 ๐Ÿ“… Created 2022-12-21
4
ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-project
โญ 44,352 Python ๐Ÿ‘ค ray-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2016-10-25
5
microsoft/BitNet

Official inference framework for 1-bit LLMs

microsoft
โญ 42,358 C++ ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-07-27 ๐Ÿ“… Created 2024-08-05
6
gitleaks/gitleaks

Find secrets with Gitleaks ๐Ÿ”‘

gitleaks
โญ 31,234 Go ๐Ÿ‘ค gitleaks Organization ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2018-01-27
7
QwenLM/Qwen3

Qwen3 is the large language model series developed by Qwen team, Alibaba Cloud.

QwenLM
โญ 27,753 Python ๐Ÿ‘ค QwenLM Organization ๐Ÿ• Updated 2026-01-09 ๐Ÿ“… Created 2024-02-05
8
Hannibal046/Awesome-LLM

Awesome-LLM: a curated list of Large Language Model

Hannibal046
โญ 27,212 ๐Ÿ‘ค Hannibal046 User ๐Ÿ• Updated 2025-07-31 ๐Ÿ“… Created 2023-02-17
9
google-ai-edge/gallery

A gallery that showcases on-device ML/GenAI use cases and allows people to try and use models locally.

google-ai-edge
โญ 25,398 Kotlin ๐Ÿ‘ค google-ai-edge Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-03-31
10
liguodongiot/llm-action

This project aims to share the technical principles and practical experience related to large models (large model engineering, practical application of large models).

liguodongiot
โญ 25,297 HTML ๐Ÿ‘ค liguodongiot User ๐Ÿ• Updated 2026-10-14 ๐Ÿ“… Created 2023-05-23
11
deepseek-ai/DeepSeek-OCR

Contexts Optical Compression

deepseek-ai
โญ 24,180 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2026-01-27 ๐Ÿ“… Created 2025-10-17
12
kvcache-ai/ktransformers

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

kvcache-ai
โญ 19,687 Python ๐Ÿ‘ค kvcache-ai Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2024-07-26
13
karpathy/llama2.c

Inference Llama 2 in one file of pure C

karpathy
โญ 19,062 C ๐Ÿ‘ค karpathy User ๐Ÿ• Updated 2024-08-06 ๐Ÿ“… Created 2023-07-23
14
Stability-AI/StableLM

StableLM: Stability AI Language Models

Stability-AI
โญ 15,784 Jupyter Notebook ๐Ÿ‘ค Stability-AI Organization ๐Ÿ• Updated 2024-04-08 ๐Ÿ“… Created 2023-04-19
15
mindverse/Second-Me

Train your AI self, amplify you, bridge the world

mindverse
โญ 15,688 Python ๐Ÿ‘ค mindverse Organization ๐Ÿ• Updated 2025-09-30 ๐Ÿ“… Created 2025-03-16
16
Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Lightning-AI
โญ 13,757 Python ๐Ÿ‘ค Lightning-AI Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2023-05-04
17
deepseek-ai/FlashMLA

FlashMLA: Efficient Multi-head Latent Attention Kernels

deepseek-ai
โญ 13,275 C++ ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2026-10-27 ๐Ÿ“… Created 2025-02-21
18
bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml
โญ 12,610 Python ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-04-19
19
FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

FareedKhan-dev
โญ 12,523 C ๐Ÿ‘ค FareedKhan-dev User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-08-01
20
openvinotoolkit/openvino

OpenVINOโ„ข is an open source toolkit for optimizing and deploying AI inference

openvinotoolkit
โญ 11,869 C++ ๐Ÿ‘ค openvinotoolkit Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-10-15
21
mistralai/mistral-inference

Official inference library for Mistral models

mistralai
โญ 10,809 Jupyter Notebook ๐Ÿ‘ค mistralai Organization ๐Ÿ• Updated 2026-04-20 ๐Ÿ“… Created 2023-09-27
22
abetlen/llama-cpp-python

Python bindings for llama.cpp

abetlen
โญ 10,691 Python ๐Ÿ‘ค abetlen User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-03-23
23
Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

Tiiny-AI
โญ 9,893 C++ ๐Ÿ‘ค Tiiny-AI Organization ๐Ÿ• Updated 2026-05-11 ๐Ÿ“… Created 2023-12-15
24
bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

bentoml
โญ 8,944 Python ๐Ÿ‘ค bentoml Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2019-04-02
25
ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

ai-dynamo
โญ 8,542 Rust ๐Ÿ‘ค ai-dynamo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-03-03
26
bojieli/ai-infra-book

ใ€ŠIn-Depth Understanding of AI Infrastructure: Quantitative Analysis and System Designใ€‹ (by Li Bojie), an open-source book manuscript: Starting from hardware constraints and model architectures, it quantitatively derives the design of LLM inference and training systems. Includes the full text, PDF, accompanying computational tools, and experiments.

bojieli
โญ 8,179 Python ๐Ÿ‘ค bojieli User ๐Ÿ• Updated 2026-10-01 ๐Ÿ“… Created 2026-08-22
27
InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

InternLM
โญ 8,153 Python ๐Ÿ‘ค InternLM Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-06-15
28
algorithmicsuperintelligence/openevolve

Open-source implementation of AlphaEvolve

algorithmicsuperintelligence
โญ 7,835 Python ๐Ÿ‘ค algorithmicsuperintelligence Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-15
29
drumih/turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

drumih
โญ 7,432 Swift ๐Ÿ‘ค drumih User ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2026-07-17
30
mit-han-lab/streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

mit-han-lab
โญ 7,165 Python ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2024-07-11 ๐Ÿ“… Created 2023-09-29
31
katanemo/plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.

katanemo
โญ 7,118 Rust ๐Ÿ‘ค katanemo Organization ๐Ÿ• Updated 2026-10-24 ๐Ÿ“… Created 2024-07-09
32
google/gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

google
โญ 7,089 C++ ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-02-13
33
deepseek-ai/DeepSeek-Coder-V2

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

deepseek-ai
โญ 7,030 ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-11-11 ๐Ÿ“… Created 2024-06-14
34
karminski/one-small-step

This is a simple technical็ง‘ๆ™ฎ tutorial project, mainly focusing on explaining some interesting, cutting-edge technical concepts and principles. Each article strives to be completed within 5 minutes of reading.

karminski
โญ 7,006 Python ๐Ÿ‘ค karminski User ๐Ÿ• Updated 2026-03-08 ๐Ÿ“… Created 2025-01-16
35
flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

flashinfer-ai
โญ 6,982 Cuda ๐Ÿ‘ค flashinfer-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-07-22
36
simplescaling/s1

s1: Simple test-time scaling

simplescaling
โญ 6,656 Python ๐Ÿ‘ค simplescaling Organization ๐Ÿ• Updated 2025-06-25 ๐Ÿ“… Created 2025-02-01
37
lemonade-sdk/lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

lemonade-sdk
โญ 6,262 C++ ๐Ÿ‘ค lemonade-sdk Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-05-15
38
kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

kserve
โญ 6,253 Go ๐Ÿ‘ค kserve Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2019-03-27
39
meta-pytorch/gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

meta-pytorch
โญ 6,244 Python ๐Ÿ‘ค meta-pytorch Organization ๐Ÿ• Updated 2025-08-22 ๐Ÿ“… Created 2023-10-17
40
cactus-compute/engine

2-bit quantization, kernels, and runtime for consumer devices.

cactus-compute
โญ 6,200 C++ ๐Ÿ‘ค cactus-compute Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2025-04-23
41
gpustack/gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

gpustack
โญ 6,097 Python ๐Ÿ‘ค gpustack Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-05-11
42
Michael-A-Kuykendall/shimmy

โšก Pure-Rust WebGPU inference engine โ€” OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

Michael-A-Kuykendall
โญ 6,020 Rust ๐Ÿ‘ค Michael-A-Kuykendall User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2025-08-28
43
alexzhang13/rlm

General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.

alexzhang13
โญ 5,776 Python ๐Ÿ‘ค alexzhang13 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-12-20
44
sgl-project/mini-sglang

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

sgl-project
โญ 5,623 Python ๐Ÿ‘ค sgl-project Organization ๐Ÿ• Updated 2026-05-17 ๐Ÿ“… Created 2025-09-01
45
xlite-dev/Awesome-LLM-Inference

๐Ÿ“šA curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.๐ŸŽ‰

xlite-dev
โญ 5,565 Python ๐Ÿ‘ค xlite-dev Organization ๐Ÿ• Updated 2026-10-20 ๐Ÿ“… Created 2023-08-27
46
superduper-io/superduper

Superduper: End-to-end framework for building custom AI applications and agents.

superduper-io
โญ 5,317 Python ๐Ÿ‘ค superduper-io Organization ๐Ÿ• Updated 2025-09-01 ๐Ÿ“… Created 2022-08-30
47
ztxz16/fastllm

fastllm is a high-performance large language model inference library with no backend dependencies, helping you run inference efficiently on most devices!

ztxz16
โญ 5,133 C++ ๐Ÿ‘ค ztxz16 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-13
48
deepseek-ai/DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

deepseek-ai
โญ 4,988 ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2024-09-25 ๐Ÿ“… Created 2024-04-22
49
FellouAI/eko

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

FellouAI
โญ 4,963 TypeScript ๐Ÿ‘ค FellouAI Organization ๐Ÿ• Updated 2026-03-03 ๐Ÿ“… Created 2024-11-23
50
ruvnet/RuVector

RuVector provides high-performance, real-time decision-making and agent memory, self-learning AI, and a vector GNN database built in Rust.

ruvnet
โญ 4,713 Rust ๐Ÿ‘ค ruvnet User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-11-19

Frequently Asked Questions

1

What is the most popular #llm-inference repository on GitHub?

The top ranked open source #llm-inference project on GitHub is deepseek-ai/DeepSeek-V3 with 104,417 stars.

2

Which programming languages are most used in #llm-inference repositories?

Among the top ranked #llm-inference projects on GitHub, the most common programming languages are Python, C++, Rust, Go.

3

How is the #llm-inference repository ranking calculated?

We track 202 active open source repositories tagged with #llm-inference (500+ stars), totaling 1,046,424 stars with an average of 5,180 stars per repository. Rankings update daily from the GitHub API.