#llm-inference Repositories
202 repositories total
A lightweight, low-dependency, unified API to use all common reranking and cross-encoder models.
A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.
Open-source AI computer builds, from 2-GPU desktops to 8-GPU servers: CAD, BOM, BIOS, and build guides. Get the case kit with PCIe risers, or ask us for a free fit check.
[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs
The official API server for Exllama. OAI compatible, lightweight, and fast.
List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search
Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser
A curated collection of top-tier penetration testing tools and productivity utilities across multiple domains. Join us to explore, contribute, and enhance your hacking toolkit!
๐ฐ Must-read papers and blogs on Speculative Decoding โก๏ธ
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
Checkpoint-engine is a simple middleware to update model weights in LLM inference engines
OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.
"AI-Compass" will guide the community in navigating the ocean of AI technology. Whether you are a beginner or an advanced developer, you can find a path towards all major AI directions here. It aims to help developers systematically understand AIโs core concepts, mainstream technologies, cuttingโedge trends, and, through practice, master the entire process from theory to implementation.
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices, and scores with probabilities, on your laptop's CPU.
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
LeaderWorkerSet & DisaggregatedSet: Kubernetes APIs for Distributed and Disaggregated AI Serving
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.