#evaluation Repositories
89 repositories total
Open memory and reliability layer for AI agents + independent judging/verification (NACRE judge, caliber-bench meta-benchmark, preregistered criteria) โ by Nautilus Platform
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
NCalc is a fast and lightweight expression evaluator library for .NET, designed for flexibility and high performance. It supports a wide range of mathematical and logical operations.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Trace-native CI/CD for AI agents โ production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards
A powerful model compression framework for LLMs and LVLMs, adapted for NVIDIA GPUs and Huawei Ascend NPUs.
Generate High-Quality Synthetics, Train, Measure, and Evaluate in a Single Pipeline
SemanticKITTI API for visualizing dataset, processing data, and evaluating results.
ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
Open source evals for physical AI. Run any LLM/VLA on any arm/humanoid against any real/sim benchmark.
Web Codegen Scorer is a tool for evaluating the quality of web code generated by LLMs.
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.
๐ฆ Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.
Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01โASI10 taxonomy
Benchmarking long-form factuality in large language models. Original code for our paper "Long-form factuality in large language models".
โก๏ธA Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion ๐
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. A curated list of tools, benchmarks/datasets, demos, leaderboards, and large models, primarily focused on the evaluation of foundation large models, aiming to explore the technological boundaries of generative AI.
A resource repository for machine unlearning in large language models
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
TCExam is a CBA (Computer-Based Assessment) system (e-exam, CBT - Computer Based Testing) for universities, schools and companies, that enables educators and trainers to author, schedule, deliver, and report on surveys, quizzes, tests and exams.
A Simple Math and Pseudo C# Expression Evaluator in One C# File. Can also execute small C# like scripts
XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation
This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"
This is a toolbox repository to help evaluate various methods that perform image matching from a pair of images.
Meta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.
The official GitHub repository of the paper "Recent advances in large language model benchmarks against data contamination: From static to dynamic evaluation"