#llm-evaluation Repositories
108 repositories total
A non-saturating, open-ended environment for evaluating LLMs in Factorio
[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification
A collection of notebooks/recipes showcasing usecases of open-source models with Together AI.
Weave is a toolkit for developing AI-powered applications, built by Weights & Biases.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Trace-native CI/CD for AI agents โ production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.
OrcaReplay โ Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
A curated, non-BS library of the best resources for building and evaluating AI agents โ papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
[ICLR 2025 Oral] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
The papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey.
GAOKAO-Bench is an evaluation framework that utilizes GAOKAO questions as a dataset to evaluate large language models.
Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corporaโwith retrieval and evaluation tooling included.
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. A curated list of tools, benchmarks/datasets, demos, leaderboards, and large models, primarily focused on the evaluation of foundation large models, aiming to explore the technological boundaries of generative AI.
Stanford CS329Z Engineering AI Agents Chinese tutorial: an AI agent course for Python developers, with 19 lectures and 4 hands-on projects, covering LLMs, RAG, tool calling, memory, multi-agent systems, evaluation, and safety. Complete Markdown, online and offline HTML. Chinese course notes.
Industrial-first evaluation benchmark for LLMs in the DevOps/AIOps domain.
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
An Open-Source Large-Scale Reinforcement Learning Project for Search Agents
MCP-Universe is a comprehensive framework designed for RL training, benchmarking, and developing AI agents for general tool-use.
The official repo for paper, LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods.
MCP gateway & OAuth MCP servers. LangGraph agent: memory, evaluation/tracking, PII protection, context trimming. Docker quick start for enterprise agents.
AI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.
This is the repository of HaluEval, a large-scale hallucination evaluation benchmark for Large Language Models.