Git Stars Git Stars

#evaluation Repositories

Discover 89 active open source GitHub repositories tagged with #evaluation, accumulating 369,872 total stars (averaging 4,155 stars per project). Leading this topic is lm-sys/FastChat with 39,550 stars.

89
Total Repositories
369,872
Total Stars
4,155
Avg Stars/Repo
1
lm-sys/FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

lm-sys
โญ 39,550 Python ๐Ÿ‘ค lm-sys Organization ๐Ÿ• Updated 2026-05-01 ๐Ÿ“… Created 2023-03-19
2
Tencent/WeKnora

Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

Tencent
โญ 37,938 Go ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-22
3
langfuse/langfuse

๐Ÿชข Open-source AI engineering platform: AI Gateway, agent observability, evaluations, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.

langfuse
โญ 37,696 TypeScript ๐Ÿ‘ค langfuse Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-18
4
mlflow/mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow
โญ 28,920 Python ๐Ÿ‘ค mlflow Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-06-05
5
promptfoo/promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

promptfoo
โญ 27,516 TypeScript ๐Ÿ‘ค promptfoo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-04-28
6
comet-ml/opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

comet-ml
โญ 23,479 Java ๐Ÿ‘ค comet-ml Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-10
7
vibrantlabsai/ragas

Supercharge Your LLM Application Evaluations ๐Ÿš€

vibrantlabsai
โญ 16,395 Python ๐Ÿ‘ค vibrantlabsai Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2023-05-08
8
open-compass/opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc., across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

open-compass
โญ 7,610 Python ๐Ÿ‘ค open-compass Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-06-15
9
alchaincyf/darwin-skill

่พพๅฐ”ๆ–‡.skill โ€”โ€” A system that infinitely evolves your Skills: evaluateโ†’improveโ†’testโ†’keep or rollback | Autoresearch-inspired autonomous skill optimization for Claude Code. Evaluate, improve, test, keep or revert.

alchaincyf
โญ 7,015 HTML ๐Ÿ‘ค alchaincyf User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2026-04-13
10
Helicone/helicone

๐ŸงŠ Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ๐Ÿ“

Helicone
โญ 6,292 TypeScript ๐Ÿ‘ค Helicone Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-01-31
11
coze-dev/coze-loop

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

coze-dev
โญ 5,799 Go ๐Ÿ‘ค coze-dev Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2025-06-24
12
Kiln-AI/Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Kiln-AI
โญ 5,286 Python ๐Ÿ‘ค Kiln-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-07-23
13
langwatch/langwatch

Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

langwatch
โญ 5,050 TypeScript ๐Ÿ‘ค langwatch Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-09-09
14
EvolvingLMMs-Lab/lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

EvolvingLMMs-Lab
โญ 4,501 Python ๐Ÿ‘ค EvolvingLMMs-Lab Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-03-07
15
open-compass/VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

open-compass
โญ 4,499 Python ๐Ÿ‘ค open-compass Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-12-01
16
MichaelGrupp/evo

Python package for the evaluation of odometry and SLAM

MichaelGrupp
โญ 4,367 Python ๐Ÿ‘ค MichaelGrupp User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2017-09-13
17
NVIDIA/SkillEvaluator

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

NVIDIA
โญ 4,300 Python ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-06-24
18
modelscope/evalscope

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

modelscope
โญ 3,715 Python ๐Ÿ‘ค modelscope Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-12-07
19
embeddings-benchmark/mteb

MTEB: State-of-the-art evaluation of embeddings across languages and modalities

embeddings-benchmark
โญ 3,519 Python ๐Ÿ‘ค embeddings-benchmark Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-04-05
20
lmnr-ai/lmnr

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

lmnr-ai
โญ 3,516 TypeScript ๐Ÿ‘ค lmnr-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-08-29
21
CLUEbenchmark/SuperCLUE

SuperCLUE: A Benchmark for Foundation Models in Chinese

CLUEbenchmark
โญ 3,308 ๐Ÿ‘ค CLUEbenchmark Organization ๐Ÿ• Updated 2026-02-06 ๐Ÿ“… Created 2023-05-02
22
viebel/klipse

Klipse is a JavaScript plugin for embedding interactive code snippets in tech blogs.

viebel
โญ 3,134 HTML ๐Ÿ‘ค viebel User ๐Ÿ• Updated 2024-10-01 ๐Ÿ“… Created 2015-11-19
23
ianarawjo/ChainForge

An open-source visual programming environment for battle-testing prompts to LLMs.

ianarawjo
โญ 3,044 TypeScript ๐Ÿ‘ค ianarawjo User ๐Ÿ• Updated 2026-10-29 ๐Ÿ“… Created 2023-03-26
24
microsoft/promptbench

A unified evaluation framework for large language models

microsoft
โญ 2,786 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2023-06-13
25
yaojingang/yao-meta-skill

YAO = Yielding AI Outcomes. A rigorous engineering, evaluation, governance, and portability system for reusable agent skills.

yaojingang
โญ 2,739 Python ๐Ÿ‘ค yaojingang User ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2026-03-31
26
huggingface/lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

huggingface
โญ 2,587 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-01-26
27
huggingface/evaluate

๐Ÿค— Evaluate: A library for easily evaluating machine learning models and datasets.

huggingface
โญ 2,498 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2022-03-30
28
1517005260/graph-rag-agent

Assemble RAG: Manually piece together and merge GraphRAG, LightRAG, and Neo4j-llm-graph-builder for knowledge graph construction and search; integrate DeepSearch technology for private domain RAG reasoning; self-create an evaluation framework for GraphRAG.

1517005260
โญ 2,355 Python ๐Ÿ‘ค 1517005260 User ๐Ÿ• Updated 2025-11-05 ๐Ÿ“… Created 2025-02-12
29
uptrain-ai/uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

uptrain-ai
โญ 2,337 Python ๐Ÿ‘ค uptrain-ai Organization ๐Ÿ• Updated 2024-08-18 ๐Ÿ“… Created 2022-11-07
30
MCPJam/inspector

Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.

MCPJam
โญ 2,331 TypeScript ๐Ÿ‘ค MCPJam Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-05-23
31
Sahir619/fable-method

The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.

Sahir619
โญ 2,324 Python ๐Ÿ‘ค Sahir619 User ๐Ÿ• Updated 2026-10-03 ๐Ÿ“… Created 2026-07-06
32
Xnhyacinth/Awesome-LLM-Long-Context-Modeling

๐Ÿ“ฐ Must-read papers and blogs on LLM based Long Context Modeling ๐Ÿ”ฅ

Xnhyacinth
โญ 2,191 ๐Ÿ‘ค Xnhyacinth User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-09-17
33
huggingface/evaluation-guidebook

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

huggingface
โญ 2,166 Jupyter Notebook ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2025-12-03 ๐Ÿ“… Created 2024-10-09
34
ContinualAI/avalanche

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.

ContinualAI
โญ 2,054 Python ๐Ÿ‘ค ContinualAI Organization ๐Ÿ• Updated 2025-03-11 ๐Ÿ“… Created 2020-03-05
35
Cloud-CV/EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

Cloud-CV
โญ 2,049 Python ๐Ÿ‘ค Cloud-CV Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2016-10-21
36
tatsu-lab/alpaca_eval

An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

tatsu-lab
โญ 2,010 Jupyter Notebook ๐Ÿ‘ค tatsu-lab Organization ๐Ÿ• Updated 2025-08-09 ๐Ÿ“… Created 2023-05-25
37
trpc-group/trpc-agent-go

A Go framework for building production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.

trpc-group
โญ 1,940 Go ๐Ÿ‘ค trpc-group Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-05-14
38
xinshuoweng/AB3DMOT

(IROS 2020, ECCVW 2020) Official Python Implementation for "3D Multi-Object Tracking: A Baseline and New Evaluation Metrics"

xinshuoweng
โญ 1,805 Python ๐Ÿ‘ค xinshuoweng User ๐Ÿ• Updated 2024-04-03 ๐Ÿ“… Created 2019-06-19
39
onestardao/WFGY

WFGY is heading toward WFGY 5.0 Polaris Protocol, a major open-source release for AI reasoning, RAG, agents, and real-world workflows. Includes Problem Map, Global Debug Card, WFGY 4.0, and the CFV Easter Egg.

onestardao
โญ 1,804 Jupyter Notebook ๐Ÿ‘ค onestardao User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-06-04
40
SmartFlowAI/EmoLLM

Mental Health Large Language Models (LLM x Mental Health), Pre & Post-training & Dataset & Evaluation & Depoly & RAG, with InternLM / Qwen / Baichuan / DeepSeek / Mixtral / LLama / GLM series models

SmartFlowAI
โญ 1,797 Python ๐Ÿ‘ค SmartFlowAI Organization ๐Ÿ• Updated 2026-06-18 ๐Ÿ“… Created 2024-01-11
41
deepsense-ai/ragbits

Building blocks for rapid development of GenAI applications

deepsense-ai
โญ 1,678 Python ๐Ÿ‘ค deepsense-ai Organization ๐Ÿ• Updated 2026-05-18 ๐Ÿ“… Created 2024-09-02
42
MLGroupJLU/LLM-eval-survey

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

MLGroupJLU
โญ 1,615 ๐Ÿ‘ค MLGroupJLU User ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-07-02
43
sepandhaghighi/pycm

Multi-class confusion matrix library in Python

sepandhaghighi
โญ 1,507 Python ๐Ÿ‘ค sepandhaghighi User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2018-01-22
44
Maluuba/nlg-eval

Evaluation code for various unsupervised automated metrics for Natural Language Generation.

Maluuba
โญ 1,391 Python ๐Ÿ‘ค Maluuba Organization ๐Ÿ• Updated 2024-08-20 ๐Ÿ“… Created 2017-06-27
45
NVIDIA-NeMo/Gym

Evaluate and improve models and agents using environments

NVIDIA-NeMo
โญ 1,347 Python ๐Ÿ‘ค NVIDIA-NeMo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-25
46
ScalingIntelligence/KernelBench

KernelBench: Can LLMs Write GPU Kernels? - Benchmark + Toolkit with Torch -> CUDA (+ more DSLs)

ScalingIntelligence
โญ 1,336 Jupyter Notebook ๐Ÿ‘ค ScalingIntelligence Organization ๐Ÿ• Updated 2026-03-24 ๐Ÿ“… Created 2024-10-25
47
TIGER-AI-Lab/ClawBench

Open-source benchmark for browser AI agents on daily tasks.

TIGER-AI-Lab
โญ 1,325 Python ๐Ÿ‘ค TIGER-AI-Lab Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2026-04-10
48
abo-abo/lispy

Short and sweet LISP editing

abo-abo
โญ 1,297 Emacs Lisp ๐Ÿ‘ค abo-abo User ๐Ÿ• Updated 2026-01-23 ๐Ÿ“… Created 2014-01-10
49
EthicalML/xai

XAI - An eXplainability toolbox for machine learning

EthicalML
โญ 1,270 Python ๐Ÿ‘ค EthicalML Organization ๐Ÿ• Updated 2025-11-29 ๐Ÿ“… Created 2019-01-11
50
plurai-ai/intellagent

A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions

plurai-ai
โญ 1,266 Python ๐Ÿ‘ค plurai-ai Organization ๐Ÿ• Updated 2026-09-14 ๐Ÿ“… Created 2024-10-28

Frequently Asked Questions

1

What is the most popular #evaluation repository on GitHub?

The top ranked open source #evaluation project on GitHub is lm-sys/FastChat with 39,550 stars. An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

2

Which programming languages are most used in #evaluation repositories?

Among the top ranked #evaluation projects on GitHub, the most common programming languages are Python, TypeScript, Jupyter Notebook, Go.

3

How is the #evaluation repository ranking calculated?

We track 89 active open source repositories tagged with #evaluation (500+ stars), totaling 369,872 stars with an average of 4,155 stars per repository. Rankings update daily from the GitHub API.