#quantization Repositories
109 repositories total
Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files run on int8 and 4-bit tensor cores behind the /v1/systemone API, delivering up to 5.3ร lower latency than vLLM in FP8 with bit-identical answers.
From teacher to tiles โ a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
Xolotl: run DeepSeek-V4.1-Flash, all 552B parameters, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s agent, ~41 fast, 512K context. Plus Inkling-2-Mini (Thinking Machines, own engine + MTP drafter), GLM-5.3-Flash, Qwen3.8, LFM3-Omni. MoE experts tiered across VRAM, RAM, and NVMe (Xolotl Direct). OpenAI + Anthropic API.
Awesome machine learning model compression research papers, quantization, tools, and learning material.
Offline semantic Text-to-Image and Image-to-Image search on Android powered by quantized state-of-the-art vision-language pretrained CLIP model and ONNX Runtime inference engine
KVarN is a native vLLM KV-cache quantization backend for your agents: 3โ5ร more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.