#quantization Repositories
109 repositories total
PaddleSlim is an open-source library for deep model compression and architecture search.
FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Zero-GPU CPU inference for 30Bโ284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.
Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intelยฎ XPU (CUDA also supported).
LLM model quantization (compression) toolkit with hardware acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascend NPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang.
[ICML2025] SpargeAttention: A training-free sparse attention that accelerates any model inference.
Z80-ฮผLM is a 2-bit quantized language model small enough to run on an 8-bit Z80 processor. Train conversational models in Python, export them as CP/M .COM binaries, and chat with your vintage computer.
Maximizing the ability to run different LLMs on the smallest devices possible
A blueprint for building production-ready RAG systems that minimize hallucination, featuring switchable 3-step (Speed) and 4-step (Precision) pipelines.
A powerful model compression framework for LLMs and LVLMs, adapted for NVIDIA GPUs and Huawei Ascend NPUs.
A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.
FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
[ICLR2024 Spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.
Palette quantization library that powers pngquant and other PNG optimizers
[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning; [NeurIPS 2022] MCUNetV3: On-Device Training Under 256KB Memory
AMD Ryzenโข AI Software includes the tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzenโข AI powered PCs.
Halo is an open-source framework built by White Circle for training large language and multimodal models
Code for the ICML 2023 paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".
[MLSys'25] QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving; [MLSys'25] LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
List of papers related to neural network quantization in recent AI conferences and journals.
Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA. โLLM Algorithms and Systems Tutorialโ: model architectures and training, post-training, inference acceleration, performance and operator optimization, with companion Notebook exercises, answer explanations, tests, and experiments.
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU.
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment
Bilingual (Chinese + English) ML / LLM / diffusion / agent interview cheat sheets for AI autumn recruiting โ generated by ARIS /interview-cheatsheet, rendered by /render-html into single-file HTML that can be read anywhere โ plus a CVโDBLP fact-checked academic homepage generator and hand-authored long-form blogs ๐ฑ
Welcome to the official repository of SINQ! A novel, fast and high-quality quantization method designed to make any Large Language Model smaller while preserving accuracy [ICML 2026]
Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.
Summary, Code for Deep Neural Network Quantization