Git Stars Git Stars

#llm-as-a-judge Repositories

Discover 10 active open source GitHub repositories tagged with #llm-as-a-judge, accumulating 16,286 total stars (averaging 1,628 stars per project). Leading this topic is oumi-ai/oumi with 9,441 stars.

10
Total Repositories
16,286
Total Stars
1,628
Avg Stars/Repo
Programming Languages: Python HTML JavaScript
1
oumi-ai/oumi

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

oumi-ai
โญ 9,441 Python ๐Ÿ‘ค oumi-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-05-07
2
prometheus-eval/prometheus-eval

Evaluate your LLM's response with Prometheus and GPT4 ๐Ÿ’ฏ

prometheus-eval
โญ 1,087 Python ๐Ÿ‘ค prometheus-eval Organization ๐Ÿ• Updated 2025-04-25 ๐Ÿ“… Created 2024-04-18
3
braintrustdata/autoevals

AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

braintrustdata
โญ 1,087 Python ๐Ÿ‘ค braintrustdata Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2023-07-11
4
metauto-ai/agent-as-a-judge

๐Ÿ‘ฉโ€โš–๏ธ Agent-as-a-Judge: The Magic for Open-Endedness

metauto-ai
โญ 852 HTML ๐Ÿ‘ค metauto-ai Organization ๐Ÿ• Updated 2026-03-28 ๐Ÿ“… Created 2024-10-16
5
MigoXLab/dingo

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

MigoXLab
โญ 771 Python ๐Ÿ‘ค MigoXLab Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2024-12-24
6
OpenLMLab/GAOKAO-Bench

GAOKAO-Bench is an evaluation framework that utilizes GAOKAO questions as a dataset to evaluate large language models.

OpenLMLab
โญ 754 Python ๐Ÿ‘ค OpenLMLab Organization ๐Ÿ• Updated 2025-01-07 ๐Ÿ“… Created 2023-04-16
โญ 583 ๐Ÿ‘ค llm-as-a-judge Organization ๐Ÿ• Updated 2026-05-21 ๐Ÿ“… Created 2024-11-24
9
OmniJev/Flide

๐Ÿ”ฅ๐Ÿ”ฅ The extremely fast inference engine for decision models

OmniJev
โญ 550 Python ๐Ÿ‘ค OmniJev Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-24
10
PAIR-code/llm-comparator

LLM Comparator is an interactive data visualization tool for evaluating and analyzing LLM responses side-by-side, developed by the PAIR team.

PAIR-code
โญ 519 JavaScript ๐Ÿ‘ค PAIR-code Organization ๐Ÿ• Updated 2025-02-11 ๐Ÿ“… Created 2024-05-07

Frequently Asked Questions

1

What is the most popular #llm-as-a-judge repository on GitHub?

The top ranked open source #llm-as-a-judge project on GitHub is oumi-ai/oumi with 9,441 stars. Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

2

Which programming languages are most used in #llm-as-a-judge repositories?

Among the top ranked #llm-as-a-judge projects on GitHub, the most common programming languages are Python, HTML, JavaScript.

3

How is the #llm-as-a-judge repository ranking calculated?

We track 10 active open source repositories tagged with #llm-as-a-judge (500+ stars), totaling 16,286 stars with an average of 1,628 stars per repository. Rankings update daily from the GitHub API.