#evaluation Repositories
Discover 89 active open source GitHub repositories tagged with #evaluation, accumulating 369,872 total stars (averaging 4,155 stars per project). Leading this topic is lm-sys/FastChat with 39,550 stars.
89 repositories total
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
๐ชข Open-source AI engineering platform: AI Gateway, agent observability, evaluations, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc., across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
่พพๅฐๆ.skill โโ A system that infinitely evolves your Skills: evaluateโimproveโtestโkeep or rollback | Autoresearch-inspired autonomous skill optimization for Claude Code. Evaluate, improve, test, keep or revert.
๐ง Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ๐
Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
MTEB: State-of-the-art evaluation of embeddings across languages and modalities
Klipse is a JavaScript plugin for embedding interactive code snippets in tech blogs.
An open-source visual programming environment for battle-testing prompts to LLMs.
YAO = Yielding AI Outcomes. A rigorous engineering, evaluation, governance, and portability system for reusable agent skills.
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
๐ค Evaluate: A library for easily evaluating machine learning models and datasets.
Assemble RAG: Manually piece together and merge GraphRAG, LightRAG, and Neo4j-llm-graph-builder for knowledge graph construction and search; integrate DeepSearch technology for private domain RAG reasoning; self-create an evaluation framework for GraphRAG.
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.
๐ฐ Must-read papers and blogs on LLM based Long Context Modeling ๐ฅ
Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.
A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.
(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"
WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Easter Egg.
Mental Health Large Language Models (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models
The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)
A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions
Frequently Asked Questions
What is the most popular #evaluation repository on GitHub?
The top ranked open source #evaluation project on GitHub is lm-sys/FastChat with 39,550 stars. An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Which programming languages are most used in #evaluation repositories?
Among the top ranked #evaluation projects on GitHub, the most common programming languages are Python, TypeScript, Jupyter Notebook, Go.
How is the #evaluation repository ranking calculated?
We track 89 active open source repositories tagged with #evaluation (500+ stars), totaling 369,872 stars with an average of 4,155 stars per repository. Rankings update daily from the GitHub API.