Git Stars Git Stars

#evals Repositories

Discover 25 active open source GitHub repositories tagged with #evals, accumulating 108,810 total stars (averaging 4,352 stars per project). Leading this topic is mastra-ai/mastra with 29,719 stars.

25
Total Repositories
108,810
Total Stars
4,352
Avg Stars/Repo
1
mastra-ai/mastra

Mastra is the modern TypeScript framework for AI-powered applications and agents.

mastra-ai
โญ 29,719 TypeScript ๐Ÿ‘ค mastra-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-08-06
2
Arize-ai/phoenix

AI Observability & Evaluation

Arize-ai
โญ 12,368 Python ๐Ÿ‘ค Arize-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2022-11-09
3
FailproofAI/failproofai

Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.

FailproofAI
โญ 9,904 TypeScript ๐Ÿ‘ค FailproofAI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-06
4
harbor-framework/harbor

Framework for evaluating and improving agents

harbor-framework
โญ 7,313 Python ๐Ÿ‘ค harbor-framework Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-08-04
5
AgentOps-AI/agentops

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

AgentOps-AI
โญ 5,971 Python ๐Ÿ‘ค AgentOps-AI Organization ๐Ÿ• Updated 2026-06-25 ๐Ÿ“… Created 2023-08-15
6
Kiln-AI/Kiln

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Kiln-AI
โญ 5,286 Python ๐Ÿ‘ค Kiln-AI Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-07-23
7
pydantic/logfire

AI observability platform for production LLM and agent systems.

pydantic
โญ 4,564 Python ๐Ÿ‘ค pydantic Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2024-04-23
8
ombharatiya/ai-system-design-guide

AI system design guide for engineers building production AI systems and evals.

ombharatiya
โญ 4,426 ๐Ÿ‘ค ombharatiya User ๐Ÿ• Updated 2026-11-10 ๐Ÿ“… Created 2025-12-16
9
truera/trulens

Evaluation and Tracking for LLM Experiments and AI Agents

truera
โญ 3,665 Python ๐Ÿ‘ค truera Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-11-02
10
lmnr-ai/lmnr

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

lmnr-ai
โญ 3,516 TypeScript ๐Ÿ‘ค lmnr-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-08-29
11
FrancyJGLisboa/agent-skills-platform

Build tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.

FrancyJGLisboa
โญ 2,417 Python ๐Ÿ‘ค FrancyJGLisboa User ๐Ÿ• Updated 2026-09-15 ๐Ÿ“… Created 2025-10-18
12
MCPJam/inspector

Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.

MCPJam
โญ 2,331 TypeScript ๐Ÿ‘ค MCPJam Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-05-23
13
future-agi/future-agi

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing ยท Evals ยท Simulations ยท Datasets ยท Gateway ยท Guardrails. Self-hostable. Apache 2.0.

future-agi
โญ 2,284 Python ๐Ÿ‘ค future-agi Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-04-23
14
ShenSeanChen/waku-agent

Waku Agent is an open-source AI agent harness you own: loop, memory, evals, and an always-on layer (event inbox, timers, approvals, outbox), in code that stays readable.

ShenSeanChen
โญ 2,206 Python ๐Ÿ‘ค ShenSeanChen User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-07-10
15
heymrun/heym

Build agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure.

heymrun
โญ 1,821 Python ๐Ÿ‘ค heymrun Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-28
16
mattpocock/evalite

Evaluate your LLM-powered apps with TypeScript

mattpocock
โญ 1,740 TypeScript ๐Ÿ‘ค mattpocock User ๐Ÿ• Updated 2026-04-28 ๐Ÿ“… Created 2024-11-12
17
GitHamza0206/simba

OpenSource Production ready Customer service with built in Evals and monitoring

GitHamza0206
โญ 1,545 TypeScript ๐Ÿ‘ค GitHamza0206 User ๐Ÿ• Updated 2026-06-18 ๐Ÿ“… Created 2024-12-20
18
LilMGenius/paperthin

The skillbook for recursive self-improvement (RSI). NVIDIA's latest proof, unpacked with timeless engineering. Distilled from the OpenAI harness engineering hackathon winner's zero-touch method.

LilMGenius
โญ 1,469 JavaScript ๐Ÿ‘ค LilMGenius User ๐Ÿ• Updated 2026-11-15 ๐Ÿ“… Created 2026-06-19
19
superlinear-ai/raglite

๐Ÿฅค RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

superlinear-ai
โญ 1,215 Python ๐Ÿ‘ค superlinear-ai Organization ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2024-06-10
20
Jwuthri/Tracely-ai

Trace-native CI/CD for AI agents โ€” production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

Jwuthri
โญ 1,122 Python ๐Ÿ‘ค Jwuthri User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-06-03
21
benchflow-ai/awesome-evals

A curated, non-BS library of the best resources for building and evaluating AI agents โ€” papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.

benchflow-ai
โญ 1,049 ๐Ÿ‘ค benchflow-ai Organization ๐Ÿ• Updated 2026-11-04 ๐Ÿ“… Created 2026-06-24
22
tripleyak/SkillForge

A skill creator that proves its skills work. Evidence-driven skill creation for Claude Code and Codex: baseline-tested generation, per-skill regression evals, ecosystem doctor, cross-runtime compile, and an opt-in proactive advisor.

tripleyak
โญ 918 Python ๐Ÿ‘ค tripleyak User ๐Ÿ• Updated 2026-07-29 ๐Ÿ“… Created 2025-12-30
23
overmind-core/overmind

The platform for continuously improving AI agents.

overmind-core
โญ 771 Python ๐Ÿ‘ค overmind-core Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-27
24
calmrocks/ai-engineer-notebooks

Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set โ€” model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.

calmrocks
โญ 756 Jupyter Notebook ๐Ÿ‘ค calmrocks User ๐Ÿ• Updated 2026-11-07 ๐Ÿ“… Created 2026-08-11
25
ratel-ai/ratel

The self-improvement layer for AI agents. Ratel reads your agent's traces, finds what fails, recommends fixes, and runs evaluations on real sessions before you ship. Self-hosted: one app, one Postgres. Includes the TypeScript and Python SDKs.

ratel-ai
โญ 566 TypeScript ๐Ÿ‘ค ratel-ai Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2025-11-12

Frequently Asked Questions

1

What is the most popular #evals repository on GitHub?

The top ranked open source #evals project on GitHub is mastra-ai/mastra with 29,719 stars. Mastra is the modern TypeScript framework for AI-powered applications and agents.

2

Which programming languages are most used in #evals repositories?

Among the top ranked #evals projects on GitHub, the most common programming languages are Python, TypeScript, JavaScript, Jupyter Notebook.

3

How is the #evals repository ranking calculated?

We track 25 active open source repositories tagged with #evals (500+ stars), totaling 108,810 stars with an average of 4,352 stars per repository. Rankings update daily from the GitHub API.