Git Stars Git Stars

#ai-safety Repositories

Discover 53 active open source GitHub repositories tagged with #ai-safety, accumulating 169,967 total stars (averaging 3,206 stars per project). Leading this topic is ifixai-ai/iFixAi with 46,592 stars.

53
Total Repositories
169,967
Total Stars
3,206
Avg Stars/Repo
1
ifixai-ai/iFixAi

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

ifixai-ai
โญ 46,592 Python ๐Ÿ‘ค ifixai-ai Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-04-27
2
meta-llama/llama3

The official Meta Llama 3 GitHub site

meta-llama
โญ 29,268 Python ๐Ÿ‘ค meta-llama Organization ๐Ÿ• Updated 2025-01-26 ๐Ÿ“… Created 2024-03-15
3
Tencent/AI-Infra-Guard

A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP Scan, AI Infra Scan, and LLM jailbreak evaluation.

Tencent
โญ 7,604 Python ๐Ÿ‘ค Tencent Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2024-12-25
4
openai/openai-cs-agents-demo

Demo of a customer service use case implemented with the OpenAI Agents SDK

openai
โญ 6,618 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2025-12-18 ๐Ÿ“… Created 2025-06-14
5
microsoft/agent-governance-toolkit

AI Agent Governance Toolkit โ€” Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.

microsoft
โญ 6,586 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-03-02
6
dontriskit/awesome-ai-system-prompts

๐Ÿง  Curated collection of system prompts for top AI tools. Perfect for AI agent builders and prompt engineers. Incuding: ChatGPT, Claude, Perplexity, Manus, Claude-Code, Loveable, v0, Grok, same new, windsurf, notion, and MetaAI.

dontriskit
โญ 6,302 TypeScript ๐Ÿ‘ค dontriskit User ๐Ÿ• Updated 2026-09-29 ๐Ÿ“… Created 2025-03-05
7
microsoft/PyRIT

Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

microsoft
โญ 5,003 Python ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-12-12
8
ogx-ai/llama-stack-apps

Agentic components of the Llama Stack APIs

ogx-ai
โญ 4,302 ๐Ÿ‘ค ogx-ai Organization ๐Ÿ• Updated 2025-08-05 ๐Ÿ“… Created 2024-06-25
โญ 4,123 Python ๐Ÿ‘ค openai Organization ๐Ÿ• Updated 2026-04-15 ๐Ÿ“… Created 2024-03-11
10
jphall663/awesome-machine-learning-interpretability

A curated list of awesome responsible machine learning resources.

jphall663
โญ 4,088 ๐Ÿ‘ค jphall663 User ๐Ÿ• Updated 2026-06-03 ๐Ÿ“… Created 2018-06-21
11
UKGovernmentBEIS/inspect_ai

Inspect: A framework for large language model evaluations

UKGovernmentBEIS
โญ 3,302 Python ๐Ÿ‘ค UKGovernmentBEIS Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-14
13
sahibzada-allahyar/Formal-Verification-RSI

Formal verification for AI: a Lean 4 + Dafny proof checker that catches fake AI proofs (reward hacking), and verified recursive self-improvement (RSI) for coding agents. AI for math, physics and software engineering.

sahibzada-allahyar
โญ 2,910 Python ๐Ÿ‘ค sahibzada-allahyar User ๐Ÿ• Updated 2026-10-22 ๐Ÿ“… Created 2025-02-22
14
wuyoscar/Internal-Safety-Collapse

[NeurIPS 2026] Safety failures in coding agents and frontier LLMs. Reproducible workspaces for studying when AI agents generate unsafe data during task completion.

wuyoscar
โญ 2,623 Python ๐Ÿ‘ค wuyoscar User ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2026-03-01
15
kenryu42/cc-safety-net

A pre-execution guard for AI coding agents. It blocks destructive Git and file system commands, plus common attempts to access sensitive files, before a tool call runs. Supports Amp Code, Antigravity CLI, Claude Code, Codex, Cursor, DeepSeek Harness, Devin CLI, GitHub Copilot CLI, Grok Build, Hermes Agent, Kimi Code, OpenClaw, OpenCode, and Pi.

kenryu42
โญ 2,086 TypeScript ๐Ÿ‘ค kenryu42 User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-12-25
16
wuyoscar/AISafetyHot

AI Safety HOT: daily AI safety news, papers, and incidents in one place โ€” jailbreaks, prompt injection, alignment, evaluations, and governance. Website, daily digest, RSS, and MCP.

wuyoscar
โญ 2,079 TeX ๐Ÿ‘ค wuyoscar User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-01
17
centerforaisafety/hle

Humanity's Last Exam

centerforaisafety
โญ 1,749 Python ๐Ÿ‘ค centerforaisafety Organization ๐Ÿ• Updated 2026-09-23 ๐Ÿ“… Created 2025-01-23
โญ 1,628 Python ๐Ÿ‘ค deepseek-ai Organization ๐Ÿ• Updated 2025-12-01 ๐Ÿ“… Created 2025-11-27
19
PKU-Alignment/safe-rlhf

Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback

PKU-Alignment
โญ 1,624 Python ๐Ÿ‘ค PKU-Alignment Organization ๐Ÿ• Updated 2025-11-24 ๐Ÿ“… Created 2023-05-15
20
decoderesearch/SAELens

Training Sparse Autoencoders on Language Models

decoderesearch
โญ 1,588 Python ๐Ÿ‘ค decoderesearch Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-11-29
21
valqore/valqore

Safety-first guardrails for AI-driven cloud and Kubernetes operations

valqore
โญ 1,568 Python ๐Ÿ‘ค valqore Organization ๐Ÿ• Updated 2026-09-05 ๐Ÿ“… Created 2026-03-04
โญ 1,423 Jupyter Notebook ๐Ÿ‘ค ARENA-education Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-08
23
OpenLMLab/MOSS-RLHF

Secrets of RLHF in Large Language Models Part I: PPO

OpenLMLab
โญ 1,408 Python ๐Ÿ‘ค OpenLMLab Organization ๐Ÿ• Updated 2024-03-03 ๐Ÿ“… Created 2023-07-05
โญ 1,376 Python ๐Ÿ‘ค harishsg993010 User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2025-04-16
25
cvs-health/uqlm

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

cvs-health
โญ 1,215 Python ๐Ÿ‘ค cvs-health Organization ๐Ÿ• Updated 2026-11-08 ๐Ÿ“… Created 2025-04-17
26
xinxuxin/keystone-bench

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

xinxuxin
โญ 1,130 Python ๐Ÿ‘ค xinxuxin User ๐Ÿ• Updated 2026-10-19 ๐Ÿ“… Created 2026-09-07
27
lobehub/awesome-rsi

A curated research map of Recursive Self-Improvement (RSI): models, agents, harnesses, embodied systems, automated AI R&D, benchmarks, and safety.

lobehub
โญ 1,073 ๐Ÿ‘ค lobehub Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-08-11
28
showlab/Awesome-MLLM-Hallucination

๐Ÿ“– A curated list of resources dedicated to hallucination of multimodal large language models (MLLM).

showlab
โญ 1,044 ๐Ÿ‘ค showlab Organization ๐Ÿ• Updated 2025-09-27 ๐Ÿ“… Created 2023-12-10
29
ZhangJinHaHaHa/AgentLens

Agentlens is a trusted agent trading platform. Here, you can quickly find the Agent that meets your needs, and you can also publish your own Agent to turn it into your digital asset. We encourage everyone to transform their areas of expertise into Agents and turn them into digital assets, allowing others to see your unique strengths.

ZhangJinHaHaHa
โญ 1,013 TypeScript ๐Ÿ‘ค ZhangJinHaHaHa User ๐Ÿ• Updated 2026-08-24 ๐Ÿ“… Created 2026-04-15
30
h5i-dev/h5i

An Axum-based Rust web framework with Lean 4-verifiable application kernels that run on servers, bare-metal embedded devices, and GPUs, plus integrated security testing.

h5i-dev
โญ 936 Rust ๐Ÿ‘ค h5i-dev Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-03-11
31
tg12/gpt_jailbreak_status

This is a repository that aims to provide updates on the status of jailbreaking the OpenAI GPT language model.

tg12
โญ 934 HTML ๐Ÿ‘ค tg12 User ๐Ÿ• Updated 2026-05-16 ๐Ÿ“… Created 2023-03-01
32
andyzoujm/representation-engineering

Representation Engineering: A Top-Down Approach to AI Transparency

andyzoujm
โญ 931 Jupyter Notebook ๐Ÿ‘ค andyzoujm User ๐Ÿ• Updated 2024-08-14 ๐Ÿ“… Created 2023-09-26
33
superwesleyhys-ux/factcircuit

FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.

superwesleyhys-ux
โญ 926 Python ๐Ÿ‘ค superwesleyhys-ux User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2026-09-06
34
tjunlp-lab/Awesome-LLMs-Evaluation-Papers

The papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey.

tjunlp-lab
โญ 791 ๐Ÿ‘ค tjunlp-lab User ๐Ÿ• Updated 2024-05-08 ๐Ÿ“… Created 2023-10-29
35
Cygnux-Labs/Tracekit

Tamper-evident tracing, policy gates and cross-checks for AI agents: see what the agent did, and why.

Cygnux-Labs
โญ 759 Python ๐Ÿ‘ค Cygnux-Labs Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-09-26
โญ 749 Python ๐Ÿ‘ค patrickrchao User ๐Ÿ• Updated 2025-07-02 ๐Ÿ“… Created 2023-10-13
37
l0gicx/ai-model-bypass

Exploit prompts and roleplay techniques for bypassing AI model restrictions.

l0gicx
โญ 722 ๐Ÿ‘ค l0gicx User ๐Ÿ• Updated 2025-10-04 ๐Ÿ“… Created 2025-05-21
39
AgentSafeLabs/safelabs-eval

Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01โ€“ASI10 taxonomy

AgentSafeLabs
โญ 713 Python ๐Ÿ‘ค AgentSafeLabs Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2026-05-25
40
lalipa2003-arch/JailbreakHub

A hub/research area for jailbreaking LLMs and breaking through their safety filters.

lalipa2003-arch
โญ 687 Roff ๐Ÿ‘ค lalipa2003-arch User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2026-09-27
41
chrisliu298/awesome-llm-unlearning

A resource repository for machine unlearning in large language models

chrisliu298
โญ 640 ๐Ÿ‘ค chrisliu298 User ๐Ÿ• Updated 2026-10-06 ๐Ÿ“… Created 2024-03-27
42
aisa-group/PostTrainBench

Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours

aisa-group
โญ 627 Python ๐Ÿ‘ค aisa-group Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2025-11-28
43
PrismorSec/prismor

Self-hosted runtime control plane for AI agents. Observe, approve with HITL, or block rogue tool calls before they executeโ€”preventing secret leaks, prompt injection, supply-chain attacks, and moreโ€”from a local dashboard. Agent-agnostic (Claude, Codex, LangChain, etc.).

PrismorSec
โญ 585 Python ๐Ÿ‘ค PrismorSec Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-02-20
44
strands-agents/box

Run AI agents in a sandbox that restricts what they can execute, read, write, and access on the network. Box combines OS isolation with default-deny Dogwood policies and credential injection that keeps secrets outside the agent. Written in Rust. Supports macOS on Apple silicon and Linux.

strands-agents
โญ 561 Rust ๐Ÿ‘ค strands-agents Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2026-10-02
45
PacificAI/langtest

Deliver safe & effective language models

PacificAI
โญ 560 Python ๐Ÿ‘ค PacificAI Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2022-11-18
46
Alejandros17487/ai-jailbreaks

AI jailbreaks: 35 paste-ready AI jailbreak prompts for 25+ frontier and open-weight models โ€” system prompts, AGENTS.md specs, and Ollama Modelfiles. Built for AI security research and red-team testing.

Alejandros17487
โญ 559 ๐Ÿ‘ค Alejandros17487 User ๐Ÿ• Updated 2026-10-21 ๐Ÿ“… Created 2026-10-21
47
xop01/ai_goodpractice

A complete guide to help you integrate AI into your workflow, for both beginners and experienced users

xop01
โญ 556 ๐Ÿ‘ค xop01 User ๐Ÿ• Updated 2026-10-02 ๐Ÿ“… Created 2026-10-02
48
opendilab/LLMRiddles

Open-Source Reproduction/Demo of the LLM Riddles Game

opendilab
โญ 549 Python ๐Ÿ‘ค opendilab Organization ๐Ÿ• Updated 2024-07-30 ๐Ÿ“… Created 2023-11-07
49
agencyenterprise/PromptInject

PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. ๐Ÿ† Best Paper Awards @ NeurIPS ML Safety Workshop 2022

agencyenterprise
โญ 529 Python ๐Ÿ‘ค agencyenterprise Organization ๐Ÿ• Updated 2026-04-27 ๐Ÿ“… Created 2022-10-25
50
Justin0504/Aegis

Runtime policy enforcement for AI agents. Cryptographic audit trail, human-in-the-loop approvals, kill switch. Zero code changes.

Justin0504
โญ 524 TypeScript ๐Ÿ‘ค Justin0504 User ๐Ÿ• Updated 2026-10-31 ๐Ÿ“… Created 2026-03-04

Frequently Asked Questions

1

What is the most popular #ai-safety repository on GitHub?

The top ranked open source #ai-safety project on GitHub is ifixai-ai/iFixAi with 46,592 stars. Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

2

Which programming languages are most used in #ai-safety repositories?

Among the top ranked #ai-safety projects on GitHub, the most common programming languages are Python, TypeScript, Jupyter Notebook, Rust.

3

How is the #ai-safety repository ranking calculated?

We track 53 active open source repositories tagged with #ai-safety (500+ stars), totaling 169,967 stars with an average of 3,206 stars per repository. Rankings update daily from the GitHub API.