Git Stars Git Stars

#evaluation-framework Repositories

Discover 13 active open source GitHub repositories tagged with #evaluation-framework, accumulating 100,331 total stars (averaging 7,717 stars per project). Leading this topic is promptfoo/promptfoo with 27,516 stars.

13
Total Repositories
100,331
Total Stars
7,717
Avg Stars/Repo
1
promptfoo/promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

promptfoo
โญ 27,516 TypeScript ๐Ÿ‘ค promptfoo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-04-28
2
openai/evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

openai
โญ 19,776 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-04-14 ๐Ÿ“… Created 2023-01-23
3
confident-ai/deepeval

The LLM Evaluation Framework

confident-ai
โญ 19,585 Python ๐Ÿ‘ค confident-ai Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-08-10
4
EleutherAI/lm-evaluation-harness

A framework for few-shot evaluation of language models.

EleutherAI
โญ 14,515 Python ๐Ÿ‘ค EleutherAI Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2020-08-28
5
Kiln-AI/Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Kiln-AI
โญ 5,286 Python ๐Ÿ‘ค Kiln-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-07-23
6
Nicolepcx/ai-agents-the-definitive-guide

Repo for AI Agents The Definitive Guide

Nicolepcx
โญ 3,020 Jupyter Notebook ๐Ÿ‘ค Nicolepcx User ๐Ÿ• Updated 2026-08-01 ๐Ÿ“… Created 2025-08-22
7
huggingface/lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

huggingface
โญ 2,587 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2024-01-26
8
future-agi/future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing ยท Evals ยท Simulations ยท Datasets ยท Gateway ยท Guardrails. Self-hostable. Apache 2.0.

future-agi
โญ 2,284 Python ๐Ÿ‘ค future-agi Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-23
9
Cloud-CV/EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI

Cloud-CV
โญ 2,049 Python ๐Ÿ‘ค Cloud-CV Organization ๐Ÿ• Updated 2026-11-14 ๐Ÿ“… Created 2016-10-21
10
tsale/EDR-Telemetry

This project aims to compare and evaluate the telemetry of various EDR products.

tsale
โญ 2,004 Python ๐Ÿ‘ค tsale User ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2023-04-01
11
ServiceNow/AgentLab

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

ServiceNow
โญ 668 Python ๐Ÿ‘ค ServiceNow Organization ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-05-21
12
lartpang/SAMs-CDConcepts-Eval

Inspiring the Next Generation of Segment Anything Models: Comprehensively Evaluate SAM and SAM 2 with Diverse Prompts Towards Context-Dependent Concepts under Different Scenes

lartpang
โญ 575 Python ๐Ÿ‘ค lartpang User ๐Ÿ• Updated 2026-08-30 ๐Ÿ“… Created 2024-11-26
13
relari-ai/continuous-eval

Data-Driven Evaluation for LLM-Powered Applications

relari-ai
โญ 519 Python ๐Ÿ‘ค relari-ai Organization ๐Ÿ• Updated 2026-08-10 ๐Ÿ“… Created 2023-12-08

Frequently Asked Questions

1

What is the most popular #evaluation-framework repository on GitHub?

The top ranked open source #evaluation-framework project on GitHub is promptfoo/promptfoo with 27,516 stars. Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

2

Which programming languages are most used in #evaluation-framework repositories?

Among the top ranked #evaluation-framework projects on GitHub, the most common programming languages are Python, TypeScript, Jupyter Notebook.

3

How is the #evaluation-framework repository ranking calculated?

We track 13 active open source repositories tagged with #evaluation-framework (500+ stars), totaling 100,331 stars with an average of 7,717 stars per repository. Rankings update daily from the GitHub API.