#inference Repositories
144 repositories total
๐ Introduction to Modern Statistics - A college-level open-source textbook with a modern approach highlighting multivariable relationships and simulation-based inference.
A CLI to estimate inference memory requirements for Hugging Face models, written in Python.
Don't memorize ML. Understand it from first principles, including inference, CUDA, deep learning, data science, and reinforcement learning.
PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
โก Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.
An open-source computer vision framework to build and deploy apps in minutes
AI Productivity Tool - Free and open source, improve user productivity, and protect privacy and data security. Including but not limited to: built-in local exclusive ChatGPT, DeepSeek, Phi, Qwen and other models, one-click batch intelligent processing of pictures, videos, audio, etc.
Small, dependency-free, fast Python package to infer binary file types checking the magic numbers signature
Example notebooks for working with SageMaker Studio Lab. Sign up for an account at the link below!
Pure Rust + CUDA LLM inference engine โ no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
yolort is a runtime stack for yolov5 on specialized accelerators such as tensorrt, libtorch, onnxruntime, tvm and ncnn.
Intelยฎ AI Reference Models: contains Intel optimizations for running deep learning workloads on Intelยฎ Xeonยฎ Scalable processors and Intelยฎ Data Center GPUs
Privacy Meter: An open-source library to audit data privacy in statistical and machine learning algorithms.
Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries โ atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.
Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, an iPhone app, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows, macOS, iOS, and Linux with full GPU capability.
Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.
Deep Learning Streamer (DL Streamer) Pipeline Framework is an open-source streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines for the Cloud or at the Edge.
High-performance serverless GPU task orchestration โ the scheduling layer behind wavespeed.ai
Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.
This repository contains the codes for LipGAN. LipGAN was published as a part of the paper titled "Towards Automatic Face-to-Face Translation".
convert mmdetection model to tensorrt, support fp16, int8, batch input, dynamic shape etc.
sparkrun - launch, manage, and stop LLM inference workloads on NVIDIA DGX Spark systems. Live support: https://discord.com/invite/GH5kRgv6ZD
Python library for YOLO small object detection and instance segmentation
Local inference for diffusion language models (LLaDA, Dream), written in Mojo
Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.
Deep learned, NVIDIA-accelerated 3D object pose estimation
A high-performance inference system for large language models, designed for production environments.