Git Stars Git Stars

#llm-evaluation Repositories

108
Total Repositories
444,825
Total Stars
4,118
Avg Stars/Repo
51
JackHopkins/factorio-learning-environment

A non-saturating, open-ended environment for evaluating LLMs in Factorio

JackHopkins
โญ 1,224 Python ๐Ÿ‘ค JackHopkins User ๐Ÿ• Updated 2026-09-18 ๐Ÿ“… Created 2021-06-19
52
cvs-health/uqlm

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

cvs-health
โญ 1,215 Python ๐Ÿ‘ค cvs-health Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2025-04-17
โญ 1,208 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-01-10 ๐Ÿ“… Created 2025-01-17
54
togethercomputer/together-cookbook

A collection of notebooks/recipes showcasing usecases of open-source models with Together AI.

togethercomputer
โญ 1,191 Jupyter Notebook ๐Ÿ‘ค togethercomputer Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-10-07
55
wandb/weave

Weave is a toolkit for developing AI-powered applications, built by Weights & Biases.

wandb
โญ 1,151 Python ๐Ÿ‘ค wandb Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-06-07
56
xinxuxin/keystone-bench

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

xinxuxin
โญ 1,130 Python ๐Ÿ‘ค xinxuxin User ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2026-09-07
57
Jwuthri/Tracely-ai

Trace-native CI/CD for AI agents โ€” production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Jwuthri
โญ 1,122 Python ๐Ÿ‘ค Jwuthri User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-06-03
58
Arena-OSS/arena-hard-auto

Arena-Hard-Auto: An automatic LLM benchmark.

Arena-OSS
โญ 1,087 Python ๐Ÿ‘ค Arena-OSS Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-04
59
braintrustdata/autoevals

AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

braintrustdata
โญ 1,087 Python ๐Ÿ‘ค braintrustdata Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-07-11
60
Continuum-AI-Corp/OrcaReplay

OrcaReplay โ€” Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.

Continuum-AI-Corp
โญ 1,083 TypeScript ๐Ÿ‘ค Continuum-AI-Corp Organization ๐Ÿ• Updated 2026-10-01 ๐Ÿ“… Created 2026-08-29
โญ 1,083 Python ๐Ÿ‘ค bytedance Organization ๐Ÿ• Updated 2026-07-14 ๐Ÿ“… Created 2024-10-16
62
JudgmentLabs/judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

JudgmentLabs
โญ 1,073 Python ๐Ÿ‘ค JudgmentLabs Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-10-25
63
ray-project/llmperf

LLMPerf is a library for validating and benchmarking LLMs

ray-project
โญ 1,066 Python ๐Ÿ‘ค ray-project Organization ๐Ÿ• Updated 2024-12-09 ๐Ÿ“… Created 2023-10-10
64
benchflow-ai/awesome-evals

A curated, non-BS library of the best resources for building and evaluating AI agents โ€” papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.

benchflow-ai
โญ 1,049 ๐Ÿ‘ค benchflow-ai Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2026-06-24
65
amazon-science/RAGChecker

RAGChecker: A Fine-grained Framework For Diagnosing RAG

amazon-science
โญ 1,032 Python ๐Ÿ‘ค amazon-science Organization ๐Ÿ• Updated 2024-12-13 ๐Ÿ“… Created 2024-06-24
66
majacinka/crewai-experiments

Experiments with local as well as models available through an api

majacinka
โญ 1,006 Python ๐Ÿ‘ค majacinka User ๐Ÿ• Updated 2024-05-05 ๐Ÿ“… Created 2024-01-13
67
KCORES/kcores-llm-arena

LLM Arena by KCORES team

KCORES
โญ 960 HTML ๐Ÿ‘ค KCORES Organization ๐Ÿ• Updated 2025-04-29 ๐Ÿ“… Created 2025-02-12
68
superwesleyhys-ux/factcircuit

FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.

superwesleyhys-ux
โญ 926 Python ๐Ÿ‘ค superwesleyhys-ux User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2026-09-06
โญ 922 Python ๐Ÿ‘ค WeOpenML Organization ๐Ÿ• Updated 2024-05-22 ๐Ÿ“… Created 2023-04-30
70
hinthornw/promptimizer

Prompt optimization scratch

hinthornw
โญ 899 Python ๐Ÿ‘ค hinthornw User ๐Ÿ• Updated 2025-04-17 ๐Ÿ“… Created 2024-10-30
71
xlang-ai/Spider2

[ICLR 2025 Oral] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows

xlang-ai
โญ 890 HTML ๐Ÿ‘ค xlang-ai Organization ๐Ÿ• Updated 2026-08-12 ๐Ÿ“… Created 2024-08-22
72
yule-BUAA/MergeLM

Codebase for Merging Language Models (ICML 2024)

yule-BUAA
โญ 862 Python ๐Ÿ‘ค yule-BUAA User ๐Ÿ• Updated 2024-05-05 ๐Ÿ“… Created 2023-11-06
73
jwuthri-cloud/Tracely-ai

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

jwuthri-cloud
โญ 850 Python ๐Ÿ‘ค jwuthri-cloud User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-10-11
74
darkrishabh/agent-skills-eval

A test runner for agentskills.io-style AI agent skills

darkrishabh
โญ 819 TypeScript ๐Ÿ‘ค darkrishabh User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-05-06
75
tjunlp-lab/Awesome-LLMs-Evaluation-Papers

The papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey.

tjunlp-lab
โญ 791 ๐Ÿ‘ค tjunlp-lab User ๐Ÿ• Updated 2024-05-08 ๐Ÿ“… Created 2023-10-29
โญ 781 TypeScript ๐Ÿ‘ค langchain-ai Organization ๐Ÿ• Updated 2025-06-26 ๐Ÿ“… Created 2023-04-05
77
overmind-core/overmind

The platform for continuously improving AI agents.

overmind-core
โญ 771 Python ๐Ÿ‘ค overmind-core Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-27
79
OpenLMLab/GAOKAO-Bench

GAOKAO-Bench is an evaluation framework that utilizes GAOKAO questions as a dataset to evaluate large language models.

OpenLMLab
โญ 754 Python ๐Ÿ‘ค OpenLMLab Organization ๐Ÿ• Updated 2025-01-07 ๐Ÿ“… Created 2023-04-16
80
EverMind-AI/SkillCorpus

Open-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corporaโ€”with retrieval and evaluation tooling included.

EverMind-AI
โญ 706 Python ๐Ÿ‘ค EverMind-AI Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-08-11
81
Steven-Luo/MasteringRAG

Enterprise RAG Systems: From Novice to Expert

Steven-Luo
โญ 699 Jupyter Notebook ๐Ÿ‘ค Steven-Luo User ๐Ÿ• Updated 2025-06-25 ๐Ÿ“… Created 2024-07-16
82
mlabonne/llm-autoeval

Automatically evaluate your LLMs in Google Colab

mlabonne
โญ 678 Python ๐Ÿ‘ค mlabonne User ๐Ÿ• Updated 2024-05-07 ๐Ÿ“… Created 2024-01-01
83
onejune2018/Awesome-LLM-Eval

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. A curated list of tools, benchmarks/datasets, demos, leaderboards, and large models, primarily focused on the evaluation of foundation large models, aiming to explore the technological boundaries of generative AI.

onejune2018
โญ 661 ๐Ÿ‘ค onejune2018 User ๐Ÿ• Updated 2025-11-24 ๐Ÿ“… Created 2023-04-26
84
wanghao-SunSky/cs329z-chinese

Stanford CS329Z Engineering AI Agents Chinese tutorial: an AI agent course for Python developers, with 19 lectures and 4 hands-on projects, covering LLMs, RAG, tool calling, memory, multi-agent systems, evaluation, and safety. Complete Markdown, online and offline HTML. Chinese course notes.

wanghao-SunSky
โญ 658 HTML ๐Ÿ‘ค wanghao-SunSky User ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2026-10-28
85
ValueByte-AI/Awesome-LLM-in-Social-Science

Awesome papers involving LLMs in Social Science.

ValueByte-AI
โญ 652 ๐Ÿ‘ค ValueByte-AI Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2023-10-15
86
codefuse-ai/codefuse-devops-eval

Industrial-first evaluation benchmark for LLMs in the DevOps/AIOps domain.

codefuse-ai
โญ 646 Python ๐Ÿ‘ค codefuse-ai Organization ๐Ÿ• Updated 2024-07-10 ๐Ÿ“… Created 2023-09-28
87
onyx-dot-app/EnterpriseRAG-Bench

Dataset and benchmark for RAG on company internal documents.

onyx-dot-app
โญ 639 ๐Ÿ‘ค onyx-dot-app Organization ๐Ÿ• Updated 2026-09-03 ๐Ÿ“… Created 2026-02-19
88
HowieHwong/TrustLLM

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

HowieHwong
โญ 635 Python ๐Ÿ‘ค HowieHwong User ๐Ÿ• Updated 2026-10-04 ๐Ÿ“… Created 2023-12-23
89
inclusionAI/ASearcher

An Open-Source Large-Scale Reinforcement Learning Project for Search Agents

inclusionAI
โญ 617 Python ๐Ÿ‘ค inclusionAI Organization ๐Ÿ• Updated 2025-11-26 ๐Ÿ“… Created 2025-08-05
90
mlfoundations/evalchemy

Automatic evals for LLMs

mlfoundations
โญ 617 HTML ๐Ÿ‘ค mlfoundations Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2024-11-15
โญ 608 TypeScript ๐Ÿ‘ค T3-Content Organization ๐Ÿ• Updated 2026-02-20 ๐Ÿ“… Created 2025-05-29
92
SalesforceAIResearch/MCP-Universe

MCP-Universe is a comprehensive framework designed for RL training, benchmarking, and developing AI agents for general tool-use.

SalesforceAIResearch
โญ 606 Python ๐Ÿ‘ค SalesforceAIResearch Organization ๐Ÿ• Updated 2026-10-08 ๐Ÿ“… Created 2025-05-02
93
CSHaitao/Awesome-LLMs-as-Judges

The official repo for paper, LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods.

CSHaitao
โญ 600 ๐Ÿ‘ค CSHaitao User ๐Ÿ• Updated 2025-07-29 ๐Ÿ“… Created 2024-11-10
94
harish-gaggar/mcp-server-gateway

MCP gateway & OAuth MCP servers. LangGraph agent: memory, evaluation/tracking, PII protection, context trimming. Docker quick start for enterprise agents.

harish-gaggar
โญ 588 Python ๐Ÿ‘ค harish-gaggar User ๐Ÿ• Updated 2026-10-03 ๐Ÿ“… Created 2026-08-17
95
yfzhang114/Thyme

โœจโœจ [ICLR 2026] Think Beyond Images

yfzhang114
โญ 585 Python ๐Ÿ‘ค yfzhang114 User ๐Ÿ• Updated 2025-09-23 ๐Ÿ“… Created 2025-08-17
96
General-Analysis/GA

An encyclopedia of jailbreaking techniques to make AI models safer.

General-Analysis
โญ 583 Jupyter Notebook ๐Ÿ‘ค General-Analysis Organization ๐Ÿ• Updated 2025-05-26 ๐Ÿ“… Created 2025-01-18
โญ 583 ๐Ÿ‘ค llm-as-a-judge Organization ๐Ÿ• Updated 2026-05-21 ๐Ÿ“… Created 2024-11-24
98
juanjuandog/FinSight-AI

AI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.

juanjuandog
โญ 562 Java ๐Ÿ‘ค juanjuandog User ๐Ÿ• Updated 2026-09-02 ๐Ÿ“… Created 2026-05-11
โญ 556 Python ๐Ÿ‘ค Vahe1994 User ๐Ÿ• Updated 2026-02-08 ๐Ÿ“… Created 2023-06-05
100
RUCAIBox/HaluEval

This is the repository of HaluEval, a large-scale hallucination evaluation benchmark for Large Language Models.

RUCAIBox
โญ 540 Python ๐Ÿ‘ค RUCAIBox Organization ๐Ÿ• Updated 2024-02-12 ๐Ÿ“… Created 2023-05-18