#llm-inference Repositories
202 repositories total
Blazing-fast Apple Silicon inference engine optimized for Qwen & Nemotron. Compatible with the OpenAI and Anthropic APIs. Built by the team at nace.ai
Compare compatible open-source local LLM projects by their metrics to assess popularity and activeness.
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
Open-source local AI SDK - run AI on-device with no cloud, no API keys. Supports GGUF, RAG, image, music, and video generation, speech-to-text, P2P inference, and more. Cross-platform: Linux, macOS, Windows, Android, iOS.
BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment.
Pure Rust + CUDA LLM inference engine โ no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Up to 4ร faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding โ Gemma-4, Qwen3.8, Muse-Glimmer, Nemotron, LFM2.5, Ornith-1.0, ternary Bonsai-27B.
LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.
USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
Accelerate your Hugging Face Transformers 7.6-9x. Native to Hugging Face and PyTorch.
Open weights language model from Google DeepMind, based on Griffin.
Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16โ64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.
Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support (rkllm)
Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O
Campus recruitment, fall recruitment, spring recruitment, great internship projects, guiding you to implement from scratch a large model inference framework supporting Llama2/3 and Qwen2.5.
Frontier MoE models on RTX PRO 6000 / RTX 5090 (sm_120): DeepSeek-V4.1-Flash on 4x RTX PRO 6000, as a GPU+CPU hybrid on 4x RTX 5090, and 2-bit experts on vLLM v0.24.0
Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files run on int8 and 4-bit tensor cores behind the /v1/systemone API, delivering up to 5.3ร lower latency than vLLM in FP8 with bit-identical answers.
Yi-1.5 is an upgraded version of Yi, delivering stronger performance in coding, math, reasoning, and instruction-following capability.
Xolotl: run DeepSeek-V4.1-Flash, all 552B parameters, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s agent, ~41 fast, 512K context. Plus Inkling-2-Mini (Thinking Machines, own engine + MTP drafter), GLM-5.3-Flash, Qwen3.8, LFM3-Omni. MoE experts tiered across VRAM, RAM, and NVMe (Xolotl Direct). OpenAI + Anthropic API.
[ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
A curated list of tools built for Jev โ TypeSafe AI's System One model for typed decisions.
Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware
Open Model Engine (OME) โ Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
Learn LLM Inference Engineering step by stepโfrom KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.
KVarN is a native vLLM KV-cache quantization backend for your agents: 3โ5ร more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents and LLMs.