#model-serving Repositories
Discover 51 active open source GitHub repositories tagged with #model-serving, accumulating 211,758 total stars (averaging 4,152 stars per project). Leading this topic is vllm-project/vllm with 97,302 stars.
51 repositories total
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
Type-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s โ in pure Python with async/await, also other languages (Rust, Go, and TypeScript)
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
๐ Awesome System for Machine Learning โก๏ธ AI System Papers and Industry Practice. โก๏ธ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). ๐ป OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. ๐๏ธ Llama3, Mistral, etc. ๐งโ๐ป Video Tutorials.
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
High-Performance Serving Runtime for Interactive Superintelligence Systems
MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
๐ฐ Must-read papers and blogs on Speculative Decoding โก๏ธ
All the links, books, and creators you need to follow to stay up to date with AI!
A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
Code for the online course "Deployment of Machine Learning Models"
Pure Rust + CUDA LLM inference engine โ no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.
FastAPI Skeleton App to serve machine learning models production-ready.
Frequently Asked Questions
What is the most popular #model-serving repository on GitHub?
The top ranked open source #model-serving project on GitHub is vllm-project/vllm with 97,302 stars. A high-throughput and memory-efficient inference and serving engine for LLMs
Which programming languages are most used in #model-serving repositories?
Among the top ranked #model-serving projects on GitHub, the most common programming languages are Python, Go, Rust, Jupyter Notebook.
How is the #model-serving repository ranking calculated?
We track 51 active open source repositories tagged with #model-serving (500+ stars), totaling 211,758 stars with an average of 4,152 stars per repository. Rankings update daily from the GitHub API.