Git Stars Git Stars

#llm-evaluation Repositories

Discover 108 active open source GitHub repositories tagged with #llm-evaluation, accumulating 444,825 total stars (averaging 4,118 stars per project). Leading this topic is ifixai-ai/iFixAi with 46,592 stars.

108
Total Repositories
444,825
Total Stars
4,118
Avg Stars/Repo
1
ifixai-ai/iFixAi

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

ifixai-ai
โญ 46,592 Python ๐Ÿ‘ค ifixai-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-27
2
langfuse/langfuse

๐Ÿชข Open-source AI engineering platform: AI Gateway, agent observability, evaluations, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.

langfuse
โญ 37,696 TypeScript ๐Ÿ‘ค langfuse Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-18
3
mlflow/mlflow

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

mlflow
โญ 28,920 Python ๐Ÿ‘ค mlflow Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2018-06-05
4
promptfoo/promptfoo

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

promptfoo
โญ 27,516 TypeScript ๐Ÿ‘ค promptfoo Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-04-28
5
huggingface/open-r1

Fully open reproduction of DeepSeek-R1

huggingface
โญ 26,483 Python ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2025-01-24
6
comet-ml/opik

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

comet-ml
โญ 23,479 Java ๐Ÿ‘ค comet-ml Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-05-10
7
openai/evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

openai
โญ 19,776 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-04-14 ๐Ÿ“… Created 2023-01-23
8
confident-ai/deepeval

The LLM Evaluation Framework

confident-ai
โญ 19,585 Python ๐Ÿ‘ค confident-ai Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-08-10
9
Arize-ai/phoenix

AI Observability & Evaluation

Arize-ai
โญ 12,368 Python ๐Ÿ‘ค Arize-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-11-09
10
NVIDIA/garak

the LLM vulnerability scanner

NVIDIA
โญ 10,122 Python ๐Ÿ‘ค NVIDIA Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2023-05-10
11
oumi-ai/oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

oumi-ai
โญ 9,441 Python ๐Ÿ‘ค oumi-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-07
12
HKUDS/ClawWork

"ClawWork: OpenClaw as Your AI Coworker - ๐Ÿ’ฐ $15K earned in 11 Hours"

HKUDS
โญ 8,574 Python ๐Ÿ‘ค HKUDS Organization ๐Ÿ• Updated 2026-03-03 ๐Ÿ“… Created 2026-02-15
13
Tencent/AI-Infra-Guard

A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP Scan, AI Infra Scan, and LLM jailbreak evaluation.

Tencent
โญ 7,604 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-12-25
14
gepa-ai/gepa

Optimize prompts, code, and more with AI-powered Reflective Optimization

gepa-ai
โญ 7,523 Jupyter Notebook ๐Ÿ‘ค gepa-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-05
15
gkamradt/langchain-tutorials

Overview and tutorial of the LangChain Library

gkamradt
โญ 7,350 Jupyter Notebook ๐Ÿ‘ค gkamradt User ๐Ÿ• Updated 2024-08-05 ๐Ÿ“… Created 2023-02-13
16
simplescaling/s1

s1: Simple test-time scaling

simplescaling
โญ 6,656 Python ๐Ÿ‘ค simplescaling Organization ๐Ÿ• Updated 2025-06-25 ๐Ÿ“… Created 2025-02-01
17
jeinlee1991/chinese-llm-benchmark

NoneLinear - ReLE Evaluation: Evaluation of Chinese AI Large Model capabilities (continuously updated): Currently includes 374 models, covering commercial models such as ChatGPT, GPT-5.4, Google Gemini-3.1-pro, Claude-4.6, Ernie-X1.1, Ernie-5.0, Qwen3.6-max, Qwen3.6-plus, Baichuan, iFlytek Spark, SenseChat, as well as open-source models like Step3.5-flash, Kimi-k2.6, Ernie4.5, MiniMax-M2.7, DeepSeek-v4, Qwen3.6, Llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, Gemma4, Mistral, and more. It provides not only a ranking but also a database of over 2 million model defects, facilitating research, analysis, and improvement of large models for the community.

jeinlee1991
โญ 6,522 ๐Ÿ‘ค jeinlee1991 User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-06-04
18
Q00/ouroboros

Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.

Q00
โญ 6,385 Python ๐Ÿ‘ค Q00 User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-01-14
19
Helicone/helicone

๐ŸงŠ Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ๐Ÿ“

Helicone
โญ 6,292 TypeScript ๐Ÿ‘ค Helicone Organization ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2023-01-31
20
Giskard-AI/giskard-oss

๐Ÿข Open-Source Evaluation & Testing library for LLM Agents

Giskard-AI
โญ 5,965 Python ๐Ÿ‘ค Giskard-AI Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-03-06
21
PacktPublishing/LLM-Engineers-Handbook

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

PacktPublishing
โญ 5,438 Python ๐Ÿ‘ค PacktPublishing Organization ๐Ÿ• Updated 2026-04-22 ๐Ÿ“… Created 2024-04-09
22
going-doer/Paper2Code

Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

going-doer
โญ 5,044 Python ๐Ÿ‘ค going-doer User ๐Ÿ• Updated 2026-03-25 ๐Ÿ“… Created 2025-04-25
23
EvolvingLMMs-Lab/lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

EvolvingLMMs-Lab
โญ 4,501 Python ๐Ÿ‘ค EvolvingLMMs-Lab Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-03-07
24
hemansnation/AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

hemansnation
โญ 3,782 Jupyter Notebook ๐Ÿ‘ค hemansnation User ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2022-03-29
25
truera/trulens

Evaluation and Tracking for LLM Experiments and AI Agents

truera
โญ 3,665 Python ๐Ÿ‘ค truera Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-11-02
26
lmnr-ai/lmnr

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

lmnr-ai
โญ 3,516 TypeScript ๐Ÿ‘ค lmnr-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-08-29
27
openai/human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"

openai
โญ 3,431 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-10-16 ๐Ÿ“… Created 2021-07-06
28
Sumanth077/ai-engineering-toolkit

A curated list of 100+ libraries and frameworks for AI engineers building with LLMs

Sumanth077
โญ 3,402 ๐Ÿ‘ค Sumanth077 User ๐Ÿ• Updated 2026-05-11 ๐Ÿ“… Created 2025-08-06
29
UKGovernmentBEIS/inspect_ai

Inspect: A framework for large language model evaluations

UKGovernmentBEIS
โญ 3,302 Python ๐Ÿ‘ค UKGovernmentBEIS Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-14
30
FranxYao/chain-of-thought-hub

Benchmarking large language models' complex reasoning ability with chain-of-thought prompting

FranxYao
โญ 2,762 Jupyter Notebook ๐Ÿ‘ค FranxYao User ๐Ÿ• Updated 2024-08-04 ๐Ÿ“… Created 2023-03-10
31
genieincodebottle/generative-ai

Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.

genieincodebottle
โญ 2,675 Jupyter Notebook ๐Ÿ‘ค genieincodebottle User ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2024-01-09
32
gkamradt/needle-in-a-haystack

Doing simple retrieval from LLM models at various context lengths to measure accuracy

gkamradt
โญ 2,408 Jupyter Notebook ๐Ÿ‘ค gkamradt User ๐Ÿ• Updated 2026-06-08 ๐Ÿ“… Created 2023-11-11
33
future-agi/future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing ยท Evals ยท Simulations ยท Datasets ยท Gateway ยท Guardrails. Self-hostable. Apache 2.0.

future-agi
โญ 2,284 Python ๐Ÿ‘ค future-agi Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-23
34
break-into-data/ai-engineer-toolkit

Projects & Resources to help you become a better AI Developer.

break-into-data
โญ 2,189 TypeScript ๐Ÿ‘ค break-into-data Organization ๐Ÿ• Updated 2025-09-15 ๐Ÿ“… Created 2025-02-24
36
msoedov/agentic_security

Agentic LLM Vulnerability Scanner / AI red teaming kit ๐Ÿงช

msoedov
โญ 2,048 Python ๐Ÿ‘ค msoedov User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-04-11
37
ai-that-works/ai-that-works

๐Ÿฆ„ ai that works - every tuesday 10 AM PST

ai-that-works
โญ 2,016 TypeScript ๐Ÿ‘ค ai-that-works Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2025-04-02
39
centerforaisafety/hle

Humanity's Last Exam

centerforaisafety
โญ 1,749 Python ๐Ÿ‘ค centerforaisafety Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2025-01-23
โญ 1,703 Python ๐Ÿ‘ค leochlon User ๐Ÿ• Updated 2026-06-17 ๐Ÿ“… Created 2025-09-01
41
lizhe2004/Awesome-LLM-RAG-Application

the resources about the application based on LLM with RAG pattern

lizhe2004
โญ 1,662 ๐Ÿ‘ค lizhe2004 User ๐Ÿ• Updated 2026-10-12 ๐Ÿ“… Created 2023-12-27
42
huggingface/aisheets

Build, enrich, and transform datasets using AI models with no code

huggingface
โญ 1,656 TypeScript ๐Ÿ‘ค huggingface Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2025-01-16
43
cyberark/FuzzyAI

A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.

cyberark
โญ 1,611 Jupyter Notebook ๐Ÿ‘ค cyberark Organization ๐Ÿ• Updated 2026-02-06 ๐Ÿ“… Created 2024-12-03
44
sierra-research/tau-bench

Code and Data for Tau-Bench

sierra-research
โญ 1,514 Python ๐Ÿ‘ค sierra-research Organization ๐Ÿ• Updated 2026-03-18 ๐Ÿ“… Created 2024-06-06
45
safety-research/bloom

bloom - evaluate any behavior immediately ๐ŸŒธ๐ŸŒฑ

safety-research
โญ 1,429 Python ๐Ÿ‘ค safety-research Organization ๐Ÿ• Updated 2026-05-07 ๐Ÿ“… Created 2025-06-24
โญ 1,423 Jupyter Notebook ๐Ÿ‘ค ARENA-education Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-08
47
meridianlabs-ai/inspect_petri

An alignment auditing agent capable of quickly exploring alignment hypothesis

meridianlabs-ai
โญ 1,421 Python ๐Ÿ‘ค meridianlabs-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-08-19
48
TIGER-AI-Lab/ClawBench

Open-source benchmark for browser AI agents on daily tasks.

TIGER-AI-Lab
โญ 1,325 Python ๐Ÿ‘ค TIGER-AI-Lab Organization ๐Ÿ• Updated 2026-11-05 ๐Ÿ“… Created 2026-04-10
49
microsoft/prompty

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.

microsoft
โญ 1,288 Rust ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-09 ๐Ÿ“… Created 2024-04-22
50
langchain-ai/openevals

Readymade evaluators for your LLM apps

langchain-ai
โญ 1,259 Python ๐Ÿ‘ค langchain-ai Organization ๐Ÿ• Updated 2026-11-18 ๐Ÿ“… Created 2025-02-08

Frequently Asked Questions

1

What is the most popular #llm-evaluation repository on GitHub?

The top ranked open source #llm-evaluation project on GitHub is ifixai-ai/iFixAi with 46,592 stars. Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

2

Which programming languages are most used in #llm-evaluation repositories?

Among the top ranked #llm-evaluation projects on GitHub, the most common programming languages are Python, Jupyter Notebook, TypeScript, Java.

3

How is the #llm-evaluation repository ranking calculated?

We track 108 active open source repositories tagged with #llm-evaluation (500+ stars), totaling 444,825 stars with an average of 4,118 stars per repository. Rankings update daily from the GitHub API.