Git Stars Git Stars

#llm-inference Repositories

202
Total Repositories
1.0M
Total Stars
5,180
Avg Stars/Repo
51
turboderp-org/exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

turboderp-org
โญ 4,653 Python ๐Ÿ‘ค turboderp-org Organization ๐Ÿ• Updated 2026-03-04 ๐Ÿ“… Created 2023-08-30
52
openai/harmony

Renderer for the harmony response format to be used with gpt-oss

openai
โญ 4,548 Rust ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-04-08 ๐Ÿ“… Created 2025-07-31
53
CalvinXKY/InfraTech

Sharing AI Infra knowledge & code practice: PyTorch, vLLM/SGLang, slime/vime framework introduction โšก๏ธ, performance acceleration ๐Ÿš€, large model basics ๐Ÿง , AI software and hardware ๐Ÿ”ง, etc.

CalvinXKY
โญ 4,411 Jupyter Notebook ๐Ÿ‘ค CalvinXKY User ๐Ÿ• Updated 2026-09-20 ๐Ÿ“… Created 2025-11-14
54
algorithmicsuperintelligence/optillm

Optimizing inference proxy for LLMs

algorithmicsuperintelligence
โญ 4,343 Python ๐Ÿ‘ค algorithmicsuperintelligence Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2024-08-22
55
NVIDIA/GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

NVIDIA
โญ 4,235 Jupyter Notebook ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2023-10-19
56
raullenchai/Rapid-MLX

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4ร— faster than Apple's MLX (mlx-lm) on the same weights.

raullenchai
โญ 4,168 Python ๐Ÿ‘ค raullenchai User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-25
57
predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

predibase
โญ 3,836 Python ๐Ÿ‘ค predibase Organization ๐Ÿ• Updated 2026-05-28 ๐Ÿ“… Created 2023-10-20
58
hemansnation/AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

hemansnation
โญ 3,782 Jupyter Notebook ๐Ÿ‘ค hemansnation User ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2022-03-29
59
amitshekhariitbhu/ai-engineering-interview-questions

Your Cheat Sheet for AI Engineering Interview โ€“ Questions and Answers.

amitshekhariitbhu
โญ 3,707 Markdown ๐Ÿ‘ค amitshekhariitbhu User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-21
60
xjdr-alt/entropix

Entropy Based Sampling and Parallel CoT Decoding

xjdr-alt
โญ 3,434 Python ๐Ÿ‘ค xjdr-alt User ๐Ÿ• Updated 2024-11-13 ๐Ÿ“… Created 2024-10-03
61
ikawrakow/ik_llama.cpp

llama.cpp fork with additional SOTA quants and improved performance

ikawrakow
โญ 3,411 C++ ๐Ÿ‘ค ikawrakow User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-06-27
โญ 3,377 Python ๐Ÿ‘ค danielgross User ๐Ÿ• Updated 2024-02-25 ๐Ÿ“… Created 2023-10-01
63
b4rtaz/distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

b4rtaz
โญ 3,141 C++ ๐Ÿ‘ค b4rtaz User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-12-04
64
mw00/project-maya

Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B), and Hy4-Flash (220B) on your own GPUs โ€” starting from 8 GB of VRAM, with NVIDIA and AMD support, one dashboard, and an OpenAI- and Anthropic-compatible API.

mw00
โญ 3,140 C++ ๐Ÿ‘ค mw00 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-06
65
spiceai/spiceai

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.

spiceai
โญ 3,132 Rust ๐Ÿ‘ค spiceai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2021-08-08
66
qwopqwop200/GPTQ-for-LLaMa

4 bits quantization of LLaMA using GPTQ

qwopqwop200
โญ 3,077 Python ๐Ÿ‘ค qwopqwop200 User ๐Ÿ• Updated 2024-07-13 ๐Ÿ“… Created 2023-03-06
67
luminal-ai/luminal

Inference at the speed of light.

luminal-ai
โญ 3,064 Rust ๐Ÿ‘ค luminal-ai Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-07-18
โญ 2,804 Python ๐Ÿ‘ค Mega4alik User ๐Ÿ• Updated 2026-07-27 ๐Ÿ“… Created 2025-08-16
69
syv-ai/HyperQwen

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

syv-ai
โญ 2,778 Python ๐Ÿ‘ค syv-ai Organization ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2026-08-15
70
youssofal/MTPLX

The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.

youssofal
โญ 2,729 Python ๐Ÿ‘ค youssofal User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-05-02
71
vllm-project/production-stack

vLLMโ€™s reference system for K8S-native cluster-wide deployment with community-driven performance optimization

vllm-project
โญ 2,729 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2025-01-21
72
FasterDecoding/Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

FasterDecoding
โญ 2,684 Jupyter Notebook ๐Ÿ‘ค FasterDecoding Organization ๐Ÿ• Updated 2024-06-25 ๐Ÿ“… Created 2023-09-10
73
mirage-project/mirage

Mirage Persistent Kernel: Compiling LLMs into a MegaKernel

mirage-project
โญ 2,625 Cuda ๐Ÿ‘ค mirage-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-08
74
incoai/splash

A local inference engine for Apple silicon, built around the model.

incoai
โญ 2,591 C++ ๐Ÿ‘ค incoai Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-09-18
75
Nano-Collective/nanocoder

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.

Nano-Collective
โญ 2,582 TypeScript ๐Ÿ‘ค Nano-Collective Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-30
76
SafeAILab/EAGLE

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

SafeAILab
โญ 2,581 Python ๐Ÿ‘ค SafeAILab Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2023-12-07
77
databricks/dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

databricks
โญ 2,578 Python ๐Ÿ‘ค databricks Organization ๐Ÿ• Updated 2024-05-01 ๐Ÿ“… Created 2024-03-26
78
Jittor/JittorLLMs

Jito large model inference library features high performance, low configuration requirements, good Chinese support, and portability.

Jittor
โญ 2,419 Python ๐Ÿ‘ค Jittor User ๐Ÿ• Updated 2025-02-22 ๐Ÿ“… Created 2023-04-02
79
ashhart/TensorFold

LLM Inference Engine for Metal, CUDA and Vulkan.

ashhart
โญ 2,364 Zig ๐Ÿ‘ค ashhart User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-06-19
80
datawhalechina/zero-to-sglang

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

datawhalechina
โญ 2,306 Python ๐Ÿ‘ค datawhalechina Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2026-08-27
โญ 2,288 Jupyter Notebook ๐Ÿ‘ค imanoop7 User ๐Ÿ• Updated 2025-03-17 ๐Ÿ“… Created 2024-11-28
82
cfregly/ai-performance-engineering

Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and full-stack tuning.

cfregly
โญ 2,233 Python ๐Ÿ‘ค cfregly User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-04-21
83
neuron-core/neuron-ai

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.

neuron-core
โญ 2,179 PHP ๐Ÿ‘ค neuron-core Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-03-02
84
pallavi-shekhar/ai-engineering-interview-questions-company-wise

Your Cheat Sheet For AI Engineering Interviews at Top AI Companies - Questions and Answers.

pallavi-shekhar
โญ 2,098 Markdown ๐Ÿ‘ค pallavi-shekhar User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-09-19
85
microsoft/aici

AICI: Prompts as (Wasm) Programs

microsoft
โญ 2,077 Rust ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-01-22 ๐Ÿ“… Created 2023-09-26
86
SemiAnalysisAI/InferenceX

Open-Source Inference Research Platform

SemiAnalysisAI
โญ 2,074 Python ๐Ÿ‘ค SemiAnalysisAI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-12
87
ovg-project/kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

ovg-project
โญ 2,051 Python ๐Ÿ‘ค ovg-project Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-27
88
AtomicBot-ai/Atomic-Chat

Local AI app and inference engine for agents. Run open-weight LLMs locally โ€” private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

AtomicBot-ai
โญ 2,040 TypeScript ๐Ÿ‘ค AtomicBot-ai Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-03-31
89
kimjammer/Neuro

A recreation of Neuro-Sama originally created in 7 days.

kimjammer
โญ 1,960 Python ๐Ÿ‘ค kimjammer User ๐Ÿ• Updated 2025-01-17 ๐Ÿ“… Created 2024-01-07
90
liltom-eth/llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

liltom-eth
โญ 1,954 Jupyter Notebook ๐Ÿ‘ค liltom-eth User ๐Ÿ• Updated 2024-03-22 ๐Ÿ“… Created 2023-07-20
91
tile-ai/TileRT

Tile-Based Runtime for Ultra-Low-Latency LLM Inference

tile-ai
โญ 1,906 Python ๐Ÿ‘ค tile-ai Organization ๐Ÿ• Updated 2026-09-24 ๐Ÿ“… Created 2025-11-12
92
XiongjieDai/GPU-Benchmarks-on-LLM-Inference

Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?

XiongjieDai
โญ 1,862 Jupyter Notebook ๐Ÿ‘ค XiongjieDai User ๐Ÿ• Updated 2024-05-13 ๐Ÿ“… Created 2023-07-19
93
beam-cloud/beta9

Ultrafast serverless GPU inference, sandboxes, and background jobs

beam-cloud
โญ 1,858 Go ๐Ÿ‘ค beam-cloud Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-15
94
turboderp-org/exllamav3

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

turboderp-org
โญ 1,853 Python ๐Ÿ‘ค turboderp-org Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-04-06
95
vllm-project/guidellm

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

vllm-project
โญ 1,800 Python ๐Ÿ‘ค vllm-project Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-29
96
software-mansion/react-native-executorch

High-performance, privacy-first on-device AI inference library for React Native, powered by PyTorch's ExecuTorch runtime

software-mansion
โญ 1,794 TypeScript ๐Ÿ‘ค software-mansion Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-10-16
97
janhq/awesome-local-ai

An awesome repository of local AI tools

janhq
โญ 1,791 ๐Ÿ‘ค janhq Organization ๐Ÿ• Updated 2024-11-13 ๐Ÿ“… Created 2023-09-06
โญ 1,677 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-11-18 ๐Ÿ“… Created 2025-09-29
99
LlamaEdge/LlamaEdge

The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

LlamaEdge
โญ 1,665 Rust ๐Ÿ‘ค LlamaEdge Organization ๐Ÿ• Updated 2026-02-08 ๐Ÿ“… Created 2023-10-09
100
Nutlope/turboseek

An AI search engine inspired by Perplexity

Nutlope
โญ 1,660 TypeScript ๐Ÿ‘ค Nutlope User ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2024-05-26