#inference Repositories
144 repositories total
CubeStudio is an open-source cloud-native one-stop machine learning/deep learning/large model AI platform/MaaS/MLOps/artificial intelligence platform/training and inference platform, covering the full algorithm pipeline, multi-tenancy, computing power leasing platform, token relay, drag-and-drop task flow pipeline orchestration, multi-machine multi-card distributed training, hyperparameter search, inference services, VGPU virtualization, cloud-edge-device collaboration, edge computing, automated annotation platform, SFT fine-tuning/reward model/reinforcement learning training for large models like DeepSeek, multi-machine inference for large models with vLLM/Ollama/MindIE, private knowledge base LLMOps intelligent agents, AI model marketplace, support for domestic heterogeneous computing power scheduling (Ascend, Cambricon, Hygon, Moore Threads, Muxi, etc.), support for IB/RoCE/RDMA, and Xinchuang support.
Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation.
High-efficiency floating-point neural network inference operators for mobile, server, and Web
Turn any computer or edge device into a command center for your computer vision projects.
Manages Unified Access to Generative AI Services built on Envoy Gateway
A unified orchestration layer for heterogeneous AI compute. It standardizes how to manage compute and run training and inference on GPU clouds, Kubernetes, VMs, or bare-metal clusters.
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
High-Performance Serving Runtime for Interactive Superintelligence Systems
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.
Efficient, scalable and enterprise-grade CPU/GPU inference server for ๐ค Hugging Face transformer models ๐
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
A high-performance inference engine for LLM, VLM, and DiT models, optimized for diverse AI accelerators. It is hosted by the OpenAtom Foundation.
DELTA is a deep learning based natural language and speech processing platform. LF AI & DATA Projects: https://lfaidata.foundation/projects/delta/
NVIDIA GPU exporter for Prometheus using the nvidia-smi binary OR NVML
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust ๐ฆ
Qualcommยฎ AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcommยฎ devices.
Package for causal inference in graphs and in the pairwise settings. Tools for graph structure recovery and dependencies are included.
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Run, serve, query, and edit AI models โ a Rust model runtime built on VINDEX3, an executable, inspectable model format.
A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.
Run Laya, the open-source Jev-compatible System-1 decision model, from Node.js / TypeScript via ONNX Runtime
Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, or edge device. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Bolt is a deep learning library with high performance and heterogeneous flexibility.