#ai-safety Repositories
Discover 53 active open source GitHub repositories tagged with #ai-safety, accumulating 169,967 total stars (averaging 3,206 stars per project). Leading this topic is ifixai-ai/iFixAi with 46,592 stars.
53 repositories total
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP Scan, AI Infra Scan, and LLM jailbreak evaluation.
Demo of a customer service use case implemented with the OpenAI Agents SDK
AI Agent Governance Toolkit โ Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.
๐ง Curated collection of system prompts for top AI tools. Perfect for AI agent builders and prompt engineers. Incuding: ChatGPT, Claude, Perplexity, Manus, Claude-Code, Loveable, v0, Grok, same new, windsurf, notion, and MetaAI.
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
A curated list of awesome responsible machine learning resources.
Formal verification for AI: a Lean 4 + Dafny proof checker that catches fake AI proofs (reward hacking), and verified recursive self-improvement (RSI) for coding agents. AI for math, physics and software engineering.
[NeurIPS 2026] Safety failures in coding agents and frontier LLMs. Reproducible workspaces for studying when AI agents generate unsafe data during task completion.
A pre-execution guard for AI coding agents. It blocks destructive Git and file system commands, plus common attempts to access sensitive files, before a tool call runs. Supports Amp Code, Antigravity CLI, Claude Code, Codex, Cursor, DeepSeek Harness, Devin CLI, GitHub Copilot CLI, Grok Build, Hermes Agent, Kimi Code, OpenClaw, OpenCode, and Pi.
AI Safety HOT: daily AI safety news, papers, and incidents in one place โ jailbreaks, prompt injection, alignment, evaluations, and governance. Website, daily digest, RSS, and MCP.
Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback
[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
A curated research map of Recursive Self-Improvement (RSI): models, agents, harnesses, embodied systems, automated AI R&D, benchmarks, and safety.
๐ A curated list of resources dedicated to hallucination of multimodal large language models (MLLM).
Agentlens is a trusted agent trading platform. Here, you can quickly find the Agent that meets your needs, and you can also publish your own Agent to turn it into your digital asset. We encourage everyone to transform their areas of expertise into Agents and turn them into digital assets, allowing others to see your unique strengths.
An Axum-based Rust web framework with Lean 4-verifiable application kernels that run on servers, bare-metal embedded devices, and GPUs, plus integrated security testing.
This is a repository that aims to provide updates on the status of jailbreaking the OpenAI GPT language model.
Representation Engineering: A Top-Down Approach to AI Transparency
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
The papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey.
Tamper-evident tracing, policy gates and cross-checks for AI agents: see what the agent did, and why.
Exploit prompts and roleplay techniques for bypassing AI model restrictions.
Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01โASI10 taxonomy
A hub/research area for jailbreaking LLMs and breaking through their safety filters.
A resource repository for machine unlearning in large language models
Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours
Self-hosted runtime control plane for AI agents. Observe, approve with HITL, or block rogue tool calls before they executeโpreventing secret leaks, prompt injection, supply-chain attacks, and moreโfrom a local dashboard. Agent-agnostic (Claude, Codex, LangChain, etc.).
Run AI agents in a sandbox that restricts what they can execute, read, write, and access on the network. Box combines OS isolation with default-deny Dogwood policies and credential injection that keeps secrets outside the agent. Written in Rust. Supports macOS on Apple silicon and Linux.
AI jailbreaks: 35 paste-ready AI jailbreak prompts for 25+ frontier and open-weight models โ system prompts, AGENTS.md specs, and Ollama Modelfiles. Built for AI security research and red-team testing.
A complete guide to help you integrate AI into your workflow, for both beginners and experienced users
PromptInject is a framework that assembles prompts in a modular fashion to provide a quantitative analysis of the robustness of LLMs to adversarial prompt attacks. ๐ Best Paper Awards @ NeurIPS ML Safety Workshop 2022
Runtime policy enforcement for AI agents. Cryptographic audit trail, human-in-the-loop approvals, kill switch. Zero code changes.
Frequently Asked Questions
What is the most popular #ai-safety repository on GitHub?
The top ranked open source #ai-safety project on GitHub is ifixai-ai/iFixAi with 46,592 stars. Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
Which programming languages are most used in #ai-safety repositories?
Among the top ranked #ai-safety projects on GitHub, the most common programming languages are Python, TypeScript, Jupyter Notebook, Rust.
How is the #ai-safety repository ranking calculated?
We track 53 active open source repositories tagged with #ai-safety (500+ stars), totaling 169,967 stars with an average of 3,206 stars per repository. Rankings update daily from the GitHub API.