Git Stars Git Stars

#evaluation Repositories

89
Total Repositories
369,872
Total Stars
4,155
Avg Stars/Repo
51
chunxiaoxx/nautilus-compass

Open memory and reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) โ€” by Nautilus Platform

chunxiaoxx
โญ 1,260 Python ๐Ÿ‘ค chunxiaoxx User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-04-27
52
run-house/kubetorch

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

run-house
โญ 1,228 Python ๐Ÿ‘ค run-house Organization ๐Ÿ• Updated 2026-05-29 ๐Ÿ“… Created 2022-05-10
53
google/fuzzbench

FuzzBench - Fuzzer benchmarking as a service.

google
โญ 1,211 Python ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-01-26 ๐Ÿ“… Created 2020-02-04
54
toshas/torch-fidelity

High-fidelity performance metrics for generative models in PyTorch

toshas
โญ 1,205 Python ๐Ÿ‘ค toshas User ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2020-04-23
55
ncalc/ncalc

NCalc is a fast and lightweight expression evaluator library for .NET, designed for flexibility and high performance. It supports a wide range of mathematical and logical operations.

ncalc
โญ 1,172 C# ๐Ÿ‘ค ncalc Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2017-11-15
56
xinxuxin/keystone-bench

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

xinxuxin
โญ 1,130 Python ๐Ÿ‘ค xinxuxin User ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2026-09-07
57
Jwuthri/Tracely-ai

Trace-native CI/CD for AI agents โ€” production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Jwuthri
โญ 1,122 Python ๐Ÿ‘ค Jwuthri User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-06-03
58
agentscope-ai/OpenJudge

OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

agentscope-ai
โญ 1,121 Python ๐Ÿ‘ค agentscope-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-07-08
59
prometheus-eval/prometheus-eval

Evaluate your LLM's response with Prometheus and GPT4 ๐Ÿ’ฏ

prometheus-eval
โญ 1,087 Python ๐Ÿ‘ค prometheus-eval Organization ๐Ÿ• Updated 2025-04-25 ๐Ÿ“… Created 2024-04-18
60
langchain-ai/langsmith-sdk

LangSmith Client SDK Implementations

langchain-ai
โญ 1,082 Python ๐Ÿ‘ค langchain-ai Organization ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2023-05-30
61
MAC-AutoML/MindPipe

A powerful model compression framework for LLMs and LVLMs, adapted for NVIDIA GPUs and Huawei Ascend NPUs.

MAC-AutoML
โญ 1,013 Python ๐Ÿ‘ค MAC-AutoML Organization ๐Ÿ• Updated 2026-08-11 ๐Ÿ“… Created 2026-01-22
62
nolabs-ai/deepfabric

Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline

nolabs-ai
โญ 896 Python ๐Ÿ‘ค nolabs-ai Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2024-10-25
63
PRBonn/semantic-kitti-api

SemanticKITTI API for visualizing dataset, processing data, and evaluating results.

PRBonn
โญ 895 Python ๐Ÿ‘ค PRBonn Organization ๐Ÿ• Updated 2025-04-03 ๐Ÿ“… Created 2019-07-24
64
suyoumo/ClawProBench

ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.

suyoumo
โญ 829 Rust ๐Ÿ‘ค suyoumo User ๐Ÿ• Updated 2026-08-25 ๐Ÿ“… Created 2025-03-02
65
PaesslerAG/gval

Expression evaluation in golang

PaesslerAG
โญ 819 Go ๐Ÿ‘ค PaesslerAG Organization ๐Ÿ• Updated 2026-09-13 ๐Ÿ“… Created 2017-09-27
66
robocurve/inspect-robots

Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.

robocurve
โญ 776 Python ๐Ÿ‘ค robocurve Organization ๐Ÿ• Updated 2026-11-11 ๐Ÿ“… Created 2026-06-26
67
angular/web-codegen-scorer

Web Codegen Scorer is a tool for evaluating the quality of web code generated by LLMs.

angular
โญ 774 TypeScript ๐Ÿ‘ค angular Organization ๐Ÿ• Updated 2026-05-05 ๐Ÿ“… Created 2025-09-04
68
IntelLabs/RAG-FiT

Framework for enhancing LLMs for RAG tasks using fine-tuning.

IntelLabs
โญ 766 Python ๐Ÿ‘ค IntelLabs Organization ๐Ÿ• Updated 2026-06-08 ๐Ÿ“… Created 2024-07-23
69
ModelTC/LightCompress

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

ModelTC
โญ 755 Python ๐Ÿ‘ค ModelTC Organization ๐Ÿ• Updated 2026-05-14 ๐Ÿ“… Created 2024-03-06
70
comet-ml/opik-openclaw

๐Ÿฆž Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.

comet-ml
โญ 723 TypeScript ๐Ÿ‘ค comet-ml Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-03-02
71
AgentSafeLabs/safelabs-eval

Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01โ€“ASI10 taxonomy

AgentSafeLabs
โญ 713 Python ๐Ÿ‘ค AgentSafeLabs Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2026-05-25
72
bochinski/iou-tracker

Python implementation of the IOU Tracker

bochinski
โญ 701 Python ๐Ÿ‘ค bochinski User ๐Ÿ• Updated 2026-06-14 ๐Ÿ“… Created 2017-07-25
73
google-deepmind/long-form-factuality

Benchmarking long-form factuality in large language models. Original code for our paper "Long-form factuality in large language models".

google-deepmind
โญ 698 Python ๐Ÿ‘ค google-deepmind Organization ๐Ÿ• Updated 2026-06-18 ๐Ÿ“… Created 2024-03-25
74
AmenRa/ranx

โšก๏ธA Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion ๐Ÿ

AmenRa
โญ 691 Python ๐Ÿ‘ค AmenRa User ๐Ÿ• Updated 2025-08-07 ๐Ÿ“… Created 2020-06-02
75
onejune2018/Awesome-LLM-Eval

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. A curated list of tools, benchmarks/datasets, demos, leaderboards, and large models, primarily focused on the evaluation of foundation large models, aiming to explore the technological boundaries of generative AI.

onejune2018
โญ 661 ๐Ÿ‘ค onejune2018 User ๐Ÿ• Updated 2025-11-24 ๐Ÿ“… Created 2023-04-26
76
run-llama/ParseBench

ParseBench - A Document Parsing Benchmark for AI Agents

run-llama
โญ 650 Python ๐Ÿ‘ค run-llama Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-10
77
ucinlp/autoprompt

AutoPrompt: Automatic Prompt Construction for Masked Language Models.

ucinlp
โญ 643 Python ๐Ÿ‘ค ucinlp Organization ๐Ÿ• Updated 2026-09-02 ๐Ÿ“… Created 2020-03-01
78
chrisliu298/awesome-llm-unlearning

A resource repository for machine unlearning in large language models

chrisliu298
โญ 640 ๐Ÿ‘ค chrisliu298 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-03-27
79
onyx-dot-app/EnterpriseRAG-Bench

Dataset and benchmark for RAG on company internal documents.

onyx-dot-app
โญ 639 ๐Ÿ‘ค onyx-dot-app Organization ๐Ÿ• Updated 2026-09-03 ๐Ÿ“… Created 2026-02-19
80
HowieHwong/TrustLLM

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

HowieHwong
โญ 635 Python ๐Ÿ‘ค HowieHwong User ๐Ÿ• Updated 2026-10-04 ๐Ÿ“… Created 2023-12-23
81
tecnickcom/tcexam

TCExam is a CBA (Computer-Based Assessment) system (e-exam, CBT - Computer Based Testing) for universities, schools and companies, that enables educators and trainers to author, schedule, deliver, and report on surveys, quizzes, tests and exams.

tecnickcom
โญ 634 PHP ๐Ÿ‘ค tecnickcom Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2013-04-15
82
codingseb/ExpressionEvaluator

A Simple Math and Pseudo C# Expression Evaluator in One C# File. Can also execute small C# like scripts

codingseb
โญ 630 C# ๐Ÿ‘ค codingseb User ๐Ÿ• Updated 2025-09-08 ๐Ÿ“… Created 2017-03-15
83
danthedeckie/simpleeval

Simple Safe Sandboxed Extensible Expression Evaluator for Python

danthedeckie
โญ 618 Python ๐Ÿ‘ค danthedeckie User ๐Ÿ• Updated 2026-10-25 ๐Ÿ“… Created 2013-12-03
84
DocAILab/XRAG

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

DocAILab
โญ 612 Python ๐Ÿ‘ค DocAILab Organization ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2024-09-07
85
MMMU-Benchmark/MMMU

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

MMMU-Benchmark
โญ 601 Python ๐Ÿ‘ค MMMU-Benchmark Organization ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2023-11-23
86
jkkummerfeld/text2sql-data

A collection of datasets that pair questions with SQL queries.

jkkummerfeld
โญ 587 Python ๐Ÿ‘ค jkkummerfeld User ๐Ÿ• Updated 2025-03-03 ๐Ÿ“… Created 2018-05-11
87
GrumpyZhou/image-matching-toolbox

This is a toolbox repository to help evaluate various methods that perform image matching from a pair of images.

GrumpyZhou
โญ 586 Jupyter Notebook ๐Ÿ‘ค GrumpyZhou User ๐Ÿ• Updated 2024-04-29 ๐Ÿ“… Created 2020-11-24
88
facebookresearch/meta-agents-research-environments

Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.

facebookresearch
โญ 581 Python ๐Ÿ‘ค facebookresearch Organization ๐Ÿ• Updated 2026-09-30 ๐Ÿ“… Created 2025-09-11
89
SeekingDream/Static-to-Dynamic-LLMEval

The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"

SeekingDream
โญ 500 ๐Ÿ‘ค SeekingDream User ๐Ÿ• Updated 2026-09-28 ๐Ÿ“… Created 2025-02-21