#llm-evaluation Repositories
Discover 108 active open source GitHub repositories tagged with #llm-evaluation, accumulating 444,825 total stars (averaging 4,118 stars per project). Leading this topic is ifixai-ai/iFixAi with 46,592 stars.
108 repositories total
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
๐ชข Open-source AI engineering platform: AI Gateway, agent observability, evaluations, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP Scan, AI Infra Scan, and LLM jailbreak evaluation.
NoneLinear - ReLE Evaluation: Evaluation of Chinese AI Large Model capabilities (continuously updated): Currently includes 374 models, covering commercial models such as ChatGPT, GPT-5.4, Google Gemini-3.1-pro, Claude-4.6, Ernie-X1.1, Ernie-5.0, Qwen3.6-max, Qwen3.6-plus, Baichuan, iFlytek Spark, SenseChat, as well as open-source models like Step3.5-flash, Kimi-k2.6, Ernie4.5, MiniMax-M2.7, DeepSeek-v4, Qwen3.6, Llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, Gemma4, Mistral, and more. It provides not only a ranking but also a database of over 2 million model defects, facilitating research, analysis, and improvement of large models for the community.
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
๐ง Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 ๐
The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
A curated list of 100+ libraries and frameworks for AI engineers building with LLMs
Benchmarking large language models' complex reasoning ability with chain-of-thought prompting
Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.
Doing simple retrieval from LLM models at various context lengths to measure accuracy
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing ยท Evals ยท Simulations ยท Datasets ยท Gateway ยท Guardrails. Self-hostable. Apache 2.0.
Projects & Resources to help you become a better AI Developer.
the resources about the application based on LLM with RAG pattern
A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.
An alignment auditing agent capable of quickly exploring alignment hypothesis
Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.
Frequently Asked Questions
What is the most popular #llm-evaluation repository on GitHub?
The top ranked open source #llm-evaluation project on GitHub is ifixai-ai/iFixAi with 46,592 stars. Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Which programming languages are most used in #llm-evaluation repositories?
Among the top ranked #llm-evaluation projects on GitHub, the most common programming languages are Python, Jupyter Notebook, TypeScript, Java.
How is the #llm-evaluation repository ranking calculated?
We track 108 active open source repositories tagged with #llm-evaluation (500+ stars), totaling 444,825 stars with an average of 4,118 stars per repository. Rankings update daily from the GitHub API.