#llm-inference Repositories
202 repositories total
A fast inference library for running LLMs locally on modern consumer-class GPUs
Sharing AI Infra knowledge & code practice: PyTorch, vLLM/SGLang, slime/vime framework introduction โก๏ธ, performance acceleration ๐, large model basics ๐ง , AI software and hardware ๐ง, etc.
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4ร faster than Apple's MLX (mlx-lm) on the same weights.
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
Your Cheat Sheet for AI Engineering Interview โ Questions and Answers.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B), and Hy4-Flash (220B) on your own GPUs โ starting from 8 GB of VRAM, with NVIDIA and AMD support, one dashboard, and an OpenAI- and Anthropic-compatible API.
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
vLLMโs reference system for K8S-native cluster-wide deployment with community-driven performance optimization
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
Code examples and resources for DBRX, a large language model developed by Databricks
Jito large model inference library features high performance, low configuration requirements, good Chinese support, and portability.
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and full-stack tuning.
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that interact with your data and UI.
Your Cheat Sheet For AI Engineering Interviews at Top AI Companies - Questions and Answers.
Local AI app and inference engine for agents. Run open-weight LLMs locally โ private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V
Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.
Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
High-performance, privacy-first on-device AI inference library for React Native, powered by PyTorch's ExecuTorch runtime
The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge