Git Stars Git Stars

#llm-inference Repositories

202
Total Repositories
1.0M
Total Stars
5,180
Avg Stars/Repo
101
AnswerDotAI/rerankers

A lightweight, low-dependency, unified API to use all common reranking and cross-encoder models.

AnswerDotAI
โญ 1,644 Python ๐Ÿ‘ค AnswerDotAI Organization ๐Ÿ• Updated 2025-12-20 ๐Ÿ“… Created 2024-03-06
102
xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.

xLLM-AI
โญ 1,626 C++ ๐Ÿ‘ค xLLM-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-12
103
autonomous-ai/Personal-AI-Computer

Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS, and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.

autonomous-ai
โญ 1,601 ๐Ÿ‘ค autonomous-ai Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2025-08-22
104
mit-han-lab/smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

mit-han-lab
โญ 1,576 Python ๐Ÿ‘ค mit-han-lab Organization ๐Ÿ• Updated 2024-07-12 ๐Ÿ“… Created 2022-11-17
105
jax-ml/scaling-book

Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs

jax-ml
โญ 1,537 SCSS ๐Ÿ‘ค jax-ml Organization ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2025-02-03
106
theroyallab/tabbyAPI

The official API server for Exllama. OAI compatible, lightweight, and fast.

theroyallab
โญ 1,531 Python ๐Ÿ‘ค theroyallab Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2023-11-10
107
felladrin/awesome-ai-web-search

List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search

felladrin
โญ 1,521 HTML ๐Ÿ‘ค felladrin User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-11-09
108
sauravpanda/BrowserAI

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

sauravpanda
โญ 1,451 TypeScript ๐Ÿ‘ค sauravpanda User ๐Ÿ• Updated 2026-07-21 ๐Ÿ“… Created 2025-01-08
109
Nutlope/pdftochat

Chat with your PDFs with AI

Nutlope
โญ 1,444 TypeScript ๐Ÿ‘ค Nutlope User ๐Ÿ• Updated 2026-10-23 ๐Ÿ“… Created 2023-09-09
110
taielab/awesome-hacking-lists

A curated collection of top-tier penetration testing tools and productivity utilities across multiple domains. Join us to explore, contribute, and enhance your hacking toolkit!

taielab
โญ 1,404 ๐Ÿ‘ค taielab User ๐Ÿ• Updated 2025-12-04 ๐Ÿ“… Created 2020-06-12
โญ 1,368 ๐Ÿ‘ค meituan-longcat Organization ๐Ÿ• Updated 2026-06-23 ๐Ÿ“… Created 2025-08-30
112
lean-dojo/LeanCopilot

LLMs as Copilots for Theorem Proving in Lean

lean-dojo
โญ 1,347 C++ ๐Ÿ‘ค lean-dojo Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2023-09-09
113
hemingkx/SpeculativeDecodingPapers

๐Ÿ“ฐ Must-read papers and blogs on Speculative Decoding โšก๏ธ

hemingkx
โญ 1,309 ๐Ÿ‘ค hemingkx User ๐Ÿ• Updated 2026-06-27 ๐Ÿ“… Created 2023-09-20
114
aigrantsindia/aigrantsindia

Get API keys for AI hackathons

aigrantsindia
โญ 1,287 ๐Ÿ‘ค aigrantsindia User ๐Ÿ• Updated 2025-06-09 ๐Ÿ“… Created 2025-02-22
115
wici-ai/inferssd

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

wici-ai
โญ 1,277 C++ ๐Ÿ‘ค wici-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-10-22
116
patchy631/time-to-first-token

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.

patchy631
โญ 1,245 HTML ๐Ÿ‘ค patchy631 User ๐Ÿ• Updated 2026-08-14 ๐Ÿ“… Created 2026-08-02
117
Azure/AzurePublicDataset

Microsoft Azure Traces

Azure
โญ 1,231 Jupyter Notebook ๐Ÿ‘ค Azure Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2017-08-18
118
eagle-hq/EAGLE-4

Official Inference Code of EAGLE-4

eagle-hq
โญ 1,220 Python ๐Ÿ‘ค eagle-hq Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2026-10-20
119
jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

jmaczan
โญ 1,207 C++ ๐Ÿ‘ค jmaczan User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2026-02-09
120
StayLameBro/backburner

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

StayLameBro
โญ 1,195 Python ๐Ÿ‘ค StayLameBro User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-10-01
121
Tencent/hpc-ops

High Performance LLM Inference Operator Library

Tencent
โญ 1,193 C++ ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-01-20
122
character-ai/prompt-poet

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

character-ai
โญ 1,160 Python ๐Ÿ‘ค character-ai Organization ๐Ÿ• Updated 2026-02-12 ๐Ÿ“… Created 2024-07-17
123
microsoft/onnxruntime-genai

Generative AI with ONNX Runtime

microsoft
โญ 1,158 C++ ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-11-13
โญ 1,090 Python ๐Ÿ‘ค thinking-machines-lab Organization ๐Ÿ• Updated 2025-11-04 ๐Ÿ“… Created 2025-09-10
125
zhuzilin/ring-flash-attention

Ring attention implementation with flash attention

zhuzilin
โญ 1,054 Python ๐Ÿ‘ค zhuzilin User ๐Ÿ• Updated 2025-09-10 ๐Ÿ“… Created 2024-02-21
126
jondurbin/airoboros

Customizable implementation of the self-instruct paper.

jondurbin
โญ 1,050 Python ๐Ÿ‘ค jondurbin User ๐Ÿ• Updated 2024-03-07 ๐Ÿ“… Created 2023-04-29
โญ 1,041 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-09-24 ๐Ÿ“… Created 2023-10-06
128
tspeterkim/flash-attention-minimal

Flash Attention in ~100 lines of CUDA (forward pass only)

tspeterkim
โญ 1,036 Cuda ๐Ÿ‘ค tspeterkim User ๐Ÿ• Updated 2024-12-30 ๐Ÿ“… Created 2024-03-07
โญ 1,033 CSS ๐Ÿ‘ค susiai Organization ๐Ÿ• Updated 2026-01-20 ๐Ÿ“… Created 2017-11-25
โญ 1,027 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2024-01-04 ๐Ÿ“… Created 2023-07-17
132
MoonshotAI/checkpoint-engine

Checkpoint-engine is a simple middleware to update model weights in LLM inference engines

MoonshotAI
โญ 1,026 Python ๐Ÿ‘ค MoonshotAI Organization ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2025-09-08
133
shyamsaktawat/OpenAlpha_Evolve

OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.

shyamsaktawat
โญ 1,021 Python ๐Ÿ‘ค shyamsaktawat User ๐Ÿ• Updated 2025-05-31 ๐Ÿ“… Created 2025-05-17
134
tingaicompass/AI-Compass

"AI-Compass" will guide the community in navigating the ocean of AI technology. Whether you are a beginner or an advanced developer, you can find a path towards all major AI directions here. It aims to help developers systematically understand AIโ€™s core concepts, mainstream technologies, cuttingโ€‘edge trends, and, through practice, master the entire process from theory to implementation.

tingaicompass
โญ 1,000 Python ๐Ÿ‘ค tingaicompass User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-05-27
135
likejazz/llama3.np

llama3.np is a pure NumPy implementation for Llama 3 model.

likejazz
โญ 991 Python ๐Ÿ‘ค likejazz User ๐Ÿ• Updated 2025-04-27 ๐Ÿ“… Created 2024-05-13
136
convergence-ai/proxy-lite

A mini, open-weights, version of our Proxy assistant.

convergence-ai
โญ 989 Python ๐Ÿ‘ค convergence-ai Organization ๐Ÿ• Updated 2025-02-26 ๐Ÿ“… Created 2025-02-10
137
feder-cr/jev

jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices, and scores with probabilities, on your laptop's CPU.

feder-cr
โญ 970 C++ ๐Ÿ‘ค feder-cr User ๐Ÿ• Updated 2026-10-18 ๐Ÿ“… Created 2024-08-15
138
dalisoft/awesome-hosting

List of awesome hosting sorted by minimal plan price

dalisoft
โญ 970 ๐Ÿ‘ค dalisoft User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2019-05-31
139
microsoft/T-MAC

Low-bit LLM inference on CPU/NPU with lookup table

microsoft
โญ 964 C++ ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2025-06-05 ๐Ÿ“… Created 2024-02-02
140
triton-inference-server/tensorrtllm_backend

The Triton TensorRT-LLM Backend

triton-inference-server
โญ 944 ๐Ÿ‘ค triton-inference-server Organization ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2023-09-12
141
foldl/chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

foldl
โญ 942 C++ ๐Ÿ‘ค foldl User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2023-12-02
142
guidance-ai/llguidance

Super-fast Structured Outputs

guidance-ai
โญ 917 Rust ๐Ÿ‘ค guidance-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-07-25
143
kubernetes-sigs/lws

LeaderWorkerSet & DisaggregatedSet: Kubernetes APIs for Distributed and Disaggregated AI Serving

kubernetes-sigs
โญ 909 Go ๐Ÿ‘ค kubernetes-sigs Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-02-22
144
zhihu/ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

zhihu
โญ 906 C++ ๐Ÿ‘ค zhihu Organization ๐Ÿ• Updated 2026-03-18 ๐Ÿ“… Created 2024-12-06
145
harleyszhang/llm_note

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

harleyszhang
โญ 897 Python ๐Ÿ‘ค harleyszhang User ๐Ÿ• Updated 2026-08-19 ๐Ÿ“… Created 2024-09-18
โญ 891 Python ๐Ÿ‘ค ByteDance-Seed Organization ๐Ÿ• Updated 2025-09-15 ๐Ÿ“… Created 2025-08-08
148
YassKhazzan/openperplex_backend_os

openperplex is an opensource AI search engine

YassKhazzan
โญ 889 Python ๐Ÿ‘ค YassKhazzan User ๐Ÿ• Updated 2024-08-04 ๐Ÿ“… Created 2024-08-03
149
HazyResearch/Megakernels

Kernels, of the mega variety :)

HazyResearch
โญ 885 Python ๐Ÿ‘ค HazyResearch Organization ๐Ÿ• Updated 2026-05-26 ๐Ÿ“… Created 2025-05-27
150
eastriverlee/LLM.swift

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

eastriverlee
โญ 883 Swift ๐Ÿ‘ค eastriverlee User ๐Ÿ• Updated 2026-07-19 ๐Ÿ“… Created 2023-12-02