#inference Repositories
Discover 144 active open source GitHub repositories tagged with #inference, accumulating 870,238 total stars (averaging 6,043 stars per project). Leading this topic is vllm-project/vllm with 97,302 stars.
144 repositories total
Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
SGLang is a high-performance serving framework for large language models and multimodal models.
ncnn is a high-performance neural network inference framework optimized for the mobile platform
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
๐จ The exhaustive Pattern Matching library for TypeScript, with smart type inference.
NVIDIAยฎ TensorRTโข is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
OpenVINOโข is an open source toolkit for optimizing and deploying AI inference
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Example ๐ Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using ๐ง Amazon SageMaker.
PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
Native MLX runtime for Laya typed decision models โ 7โ14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Superduper: End-to-end framework for building custom AI applications and agents.
cube studio Open-Source Cloud-Native All-in-One Machine Learning/Deep Learning/Large Model AI Platform, MLOps algorithm link full process, computing power rental platform, notebook online development, drag-and-drop task flow pipeline orchestration, multi-machine multi-GPU distributed training, hyperparameter search, inference service VGPU virtualization, edge computing, annotation platform automated annotation, deepseek and other large models sft fine-tuning/reward model/reinforcement learning training, vllm/ollama/mindie large model multi-machine inference, private knowledge base, AI model market, supports domestic CPU/GPU/NPU Ascend ecosystem, supports RDMA, supports distributed training such as pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano
A unified inference and post-training framework for accelerated video generation.
TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile, desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts
Pre-trained Deep Learning models and demos (high quality and extremely fast)
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild
Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.
CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! โญ๏ธ
Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware.
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Open-source inference server and production cluster for all the models your agent needs.
๐ Accelerate inference and training of ๐ค Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Good projects for campus recruitment, autumn recruitment, spring recruitment, internships! Guiding you to implement a high-performance deep learning inference library from scratch, supporting inference for large models like llama2, Unet, YoloV5, Resnet, etc. Implement a high-performance deep learning inference library step by step
Frequently Asked Questions
What is the most popular #inference repository on GitHub?
The top ranked open source #inference project on GitHub is vllm-project/vllm with 97,302 stars. A high-throughput and memory-efficient inference and serving engine for LLMs
Which programming languages are most used in #inference repositories?
Among the top ranked #inference projects on GitHub, the most common programming languages are Python, C++, TypeScript, Jupyter Notebook.
How is the #inference repository ranking calculated?
We track 144 active open source repositories tagged with #inference (500+ stars), totaling 870,238 stars with an average of 6,043 stars per repository. Rankings update daily from the GitHub API.