#llm-inference Repositories
Discover 202 active open source GitHub repositories tagged with #llm-inference, accumulating 1,046,424 total stars (averaging 5,180 stars per project). Leading this topic is deepseek-ai/DeepSeek-V3 with 104,417 stars.
202 repositories total
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
A gallery that showcases on-device ML/GenAI use cases and allows people to try and use models locally.
This project aims to share the technical principles and practical experience related to large models (large model engineering, practical application of large models).
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
OpenVINOโข is an open source toolkit for optimizing and deploying AI inference
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
ใIn-Depth Understanding of AI Infrastructure: Quantitative Analysis and System Designใ (by Li Bojie), an open-source book manuscript: Starting from hardware constraints and model architectures, it quantitatively derives the design of LLM inference and training systems. Includes the full text, PDF, accompanying computational tools, and experiments.
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
This is a simple technical็งๆฎ tutorial project, mainly focusing on explaining some interesting, cutting-edge technical concepts and principles. Each article strives to be completed within 5 minutes of reading.
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
โก Pure-Rust WebGPU inference engine โ OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes.
A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.
๐A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.๐
Superduper: End-to-end framework for building custom AI applications and agents.
fastllm is a high-performance large language model inference library with no backend dependencies, helping you run inference efficiently on most devices!
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai
RuVector provides high-performance, real-time decision-making and agent memory, self-learning AI, and a vector GNN database built in Rust.
Frequently Asked Questions
What is the most popular #llm-inference repository on GitHub?
The top ranked open source #llm-inference project on GitHub is deepseek-ai/DeepSeek-V3 with 104,417 stars.
Which programming languages are most used in #llm-inference repositories?
Among the top ranked #llm-inference projects on GitHub, the most common programming languages are Python, C++, Rust, Go.
How is the #llm-inference repository ranking calculated?
We track 202 active open source repositories tagged with #llm-inference (500+ stars), totaling 1,046,424 stars with an average of 5,180 stars per repository. Rankings update daily from the GitHub API.