Git Stars Git Stars

#interpretability Repositories

Discover 39 active open source GitHub repositories tagged with #interpretability, accumulating 134,415 total stars (averaging 3,446 stars per project). Leading this topic is shap/shap with 25,883 stars.

39
Total Repositories
134,415
Total Stars
3,446
Avg Stars/Repo
1
shap/shap

A game theoretic approach to explain the output of any machine learning model

shap
โญ 25,883 Jupyter Notebook ๐Ÿ‘ค shap Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2016-11-22
2
EthicalML/awesome-production-machine-learning

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

EthicalML
โญ 21,121 ๐Ÿ‘ค EthicalML Organization ๐Ÿ• Updated 2026-11-12 ๐Ÿ“… Created 2018-08-15
3
KindXiaoming/pykan

Kolmogorov Arnold Networks

KindXiaoming
โญ 16,291 Jupyter Notebook ๐Ÿ‘ค KindXiaoming User ๐Ÿ• Updated 2025-01-19 ๐Ÿ“… Created 2024-04-27
4
jacobgil/pytorch-grad-cam

Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.

jacobgil
โญ 13,014 Python ๐Ÿ‘ค jacobgil User ๐Ÿ• Updated 2026-11-01 ๐Ÿ“… Created 2017-05-31
5
interpretml/interpret

Fit interpretable models. Explain blackbox machine learning.

interpretml
โญ 6,986 C++ ๐Ÿ‘ค interpretml Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2019-05-03
6
meta-pytorch/captum

Model interpretability and understanding for PyTorch

meta-pytorch
โญ 5,723 Python ๐Ÿ‘ค meta-pytorch Organization ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2019-08-27
7
jphall663/awesome-machine-learning-interpretability

A curated list of awesome responsible machine learning resources.

jphall663
โญ 4,088 ๐Ÿ‘ค jphall663 User ๐Ÿ• Updated 2026-06-03 ๐Ÿ“… Created 2018-06-21
8
MAIF/shapash

๐Ÿ”… Shapash: User-friendly Explainability and Interpretability to Develop Reliable and Transparent Machine Learning Models

MAIF
โญ 3,297 Jupyter Notebook ๐Ÿ‘ค MAIF Organization ๐Ÿ• Updated 2026-11-16 ๐Ÿ“… Created 2020-04-29
9
stellargraph/stellargraph

StellarGraph - Machine Learning on Graphs

stellargraph
โญ 3,044 Python ๐Ÿ‘ค stellargraph Organization ๐Ÿ• Updated 2024-04-10 ๐Ÿ“… Created 2018-04-13
10
SeldonIO/alibi

Algorithms for explaining machine learning models

SeldonIO
โญ 2,648 Python ๐Ÿ‘ค SeldonIO Organization ๐Ÿ• Updated 2025-10-17 ๐Ÿ“… Created 2019-02-26
11
frgfm/torch-cam

Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM, Finer-CAM, LeGrad, RefineCAM)

frgfm
โญ 2,302 Python ๐Ÿ‘ค frgfm User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2020-03-23
12
lyramakesmusic/neuron-explorer

Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results

lyramakesmusic
โญ 2,045 Python ๐Ÿ‘ค lyramakesmusic User ๐Ÿ• Updated 2026-11-02 ๐Ÿ“… Created 2026-10-21
13
google-deepmind/penzai

A JAX research toolkit for building, editing, and visualizing neural networks.

google-deepmind
โญ 1,894 Python ๐Ÿ‘ค google-deepmind Organization ๐Ÿ• Updated 2025-06-22 ๐Ÿ“… Created 2024-04-04
14
microsoft/responsible-ai-toolbox

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

microsoft
โญ 1,858 TypeScript ๐Ÿ‘ค microsoft Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2020-07-06
15
wangyongjie-ntu/Awesome-explainable-AI

A collection of research materials on explainable AI/ML

wangyongjie-ntu
โญ 1,660 Markdown ๐Ÿ‘ค wangyongjie-ntu User ๐Ÿ• Updated 2026-08-19 ๐Ÿ“… Created 2020-02-16
16
csinva/imodels

Interpretable ML package ๐Ÿ” for concise, transparent, and accurate predictive modeling (sklearn-compatible).

csinva
โญ 1,631 Jupyter Notebook ๐Ÿ‘ค csinva User ๐Ÿ• Updated 2026-11-13 ๐Ÿ“… Created 2019-07-04
17
stanfordnlp/pyreft

Stanford NLP Python library for Representation Finetuning (ReFT)

stanfordnlp
โญ 1,592 Python ๐Ÿ‘ค stanfordnlp Organization ๐Ÿ• Updated 2026-03-05 ๐Ÿ“… Created 2024-02-17
18
ModelOriented/DALEX

Model Agnostic Language for Exploration and Explanation

ModelOriented
โญ 1,489 Python ๐Ÿ‘ค ModelOriented Organization ๐Ÿ• Updated 2026-07-15 ๐Ÿ“… Created 2018-02-18
19
albermax/innvestigate

A toolbox to iNNvestigate neural networks' predictions!

albermax
โญ 1,305 Python ๐Ÿ‘ค albermax User ๐Ÿ• Updated 2025-04-11 ๐Ÿ“… Created 2017-12-13
20
EthicalML/xai

XAI - An eXplainability toolbox for machine learning

EthicalML
โญ 1,270 Python ๐Ÿ‘ค EthicalML Organization ๐Ÿ• Updated 2025-11-29 ๐Ÿ“… Created 2019-01-11
21
hijohnnylin/neuronpedia

open source interpretability platform ๐Ÿง 

hijohnnylin
โญ 1,225 TypeScript ๐Ÿ‘ค hijohnnylin User ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-06-21
22
ndif-team/nnsight

The nnsight package enables interpreting and manipulating the internals of deep learned models.

ndif-team
โญ 1,152 Python ๐Ÿ‘ค ndif-team Organization ๐Ÿ• Updated 2026-11-21 ๐Ÿ“… Created 2023-10-20
23
sicara/tf-explain

Interpretability Methods for tf.keras models with Tensorflow 2.x

sicara
โญ 1,036 Python ๐Ÿ‘ค sicara Organization ๐Ÿ• Updated 2024-06-03 ๐Ÿ“… Created 2019-07-15
โญ 1,019 Python ๐Ÿ‘ค RexYing User ๐Ÿ• Updated 2024-08-30 ๐Ÿ“… Created 2018-12-22
25
stanfordnlp/pyvene

Stanford NLP Python library for understanding and improving PyTorch models via interventions

stanfordnlp
โญ 919 Python ๐Ÿ‘ค stanfordnlp Organization ๐Ÿ• Updated 2026-03-06 ๐Ÿ“… Created 2023-02-06
26
shubhomoydas/ad_examples

A collection of anomaly detection methods (iid/point-based, graph and time series) including active learning for anomaly detection/discovery, bayesian rule-mining, description for diversity/explanation/interpretability. Analysis of incorporating label feedback with ensemble and tree-based detectors. Includes adversarial attacks with Graph Convolutional Network.

shubhomoydas
โญ 870 Python ๐Ÿ‘ค shubhomoydas User ๐Ÿ• Updated 2024-05-22 ๐Ÿ“… Created 2017-11-03
27
divagr18/qwenputer

A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.

divagr18
โญ 840 Python ๐Ÿ‘ค divagr18 User ๐Ÿ• Updated 2026-11-19 ๐Ÿ“… Created 2026-10-19
28
mmschlk/shapiq

Shapley Values and Shapley Interactions for Machine Learning

mmschlk
โญ 797 Python ๐Ÿ‘ค mmschlk User ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2023-10-17
29
oneTaken/awesome_deep_learning_interpretability

Recent high-citation/top conference papers on neural network model interpretability in deep learning (with code)

oneTaken
โญ 762 ๐Ÿ‘ค oneTaken User ๐Ÿ• Updated 2024-04-08 ๐Ÿ“… Created 2020-01-01
30
deel-ai/xplique

๐Ÿ‘‹ Xplique is a Neural Networks Explainability Toolbox

deel-ai
โญ 761 Python ๐Ÿ‘ค deel-ai Organization ๐Ÿ• Updated 2026-11-06 ๐Ÿ“… Created 2020-04-05
31
johnmarktaylor91/torchlens

Inspect, visualize, and intervene in any operation in any neural network. Capture every activation and gradient, the full computational graph, and 800+ metadata fields without changing model code. Supports PyTorch, JAX, TensorFlow, MLX, PaddlePaddle, and tinygrad. Validated on 6,400+ models. Built for researchers and their AI agents.

johnmarktaylor91
โญ 714 Python ๐Ÿ‘ค johnmarktaylor91 User ๐Ÿ• Updated 2026-11-17 ๐Ÿ“… Created 2022-10-14
32
kmeng01/rome

Locating and editing factual associations in GPT (NeurIPS 2022)

kmeng01
โญ 713 Python ๐Ÿ‘ค kmeng01 User ๐Ÿ• Updated 2024-04-20 ๐Ÿ“… Created 2022-02-11
33
tensorflow/decision-forests

A collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models in Keras.

tensorflow
โญ 693 Python ๐Ÿ‘ค tensorflow Organization ๐Ÿ• Updated 2026-05-19 ๐Ÿ“… Created 2021-04-22
34
understandable-machine-intelligence-lab/Quantus

[JMLR 2023] Quantus is an eXplainable AI toolkit for responsible evaluation of neural network explanations

understandable-machine-intelligence-lab
โญ 682 Jupyter Notebook ๐Ÿ‘ค understandable-machine-intelligence-lab Organization ๐Ÿ• Updated 2026-10-26 ๐Ÿ“… Created 2021-03-18
35
google/yggdrasil-decision-forests

A library to train, evaluate, interpret, and productionize decision forest models such as Random Forest and Gradient Boosted Decision Trees.

google
โญ 680 C++ ๐Ÿ‘ค google Organization ๐Ÿ• Updated 2026-11-20 ๐Ÿ“… Created 2021-04-22
36
jphall663/interpretable_machine_learning_with_python

Examples of techniques for training interpretable ML models, explaining ML models, and debugging ML models for accuracy, discrimination, and security.

jphall663
โญ 679 Jupyter Notebook ๐Ÿ‘ค jphall663 User ๐Ÿ• Updated 2024-06-17 ๐Ÿ“… Created 2018-03-14
37
tensorflow/tcav

Code for the TCAV ML interpretability project

tensorflow
โญ 653 Jupyter Notebook ๐Ÿ‘ค tensorflow Organization ๐Ÿ• Updated 2026-07-22 ๐Ÿ“… Created 2018-07-03
38
ScalaConsultants/Aspect-Based-Sentiment-Analysis

๐Ÿ’ญ Aspect-Based-Sentiment-Analysis: Transformer & Explainable ML (TensorFlow)

ScalaConsultants
โญ 585 Python ๐Ÿ‘ค ScalaConsultants Organization ๐Ÿ• Updated 2026-09-07 ๐Ÿ“… Created 2020-01-08
39
BCG-X-Official/facet

Human-explainable AI.

BCG-X-Official
โญ 533 Jupyter Notebook ๐Ÿ‘ค BCG-X-Official Organization ๐Ÿ• Updated 2026-10-28 ๐Ÿ“… Created 2020-08-05

Frequently Asked Questions

1

What is the most popular #interpretability repository on GitHub?

The top ranked open source #interpretability project on GitHub is shap/shap with 25,883 stars. A game theoretic approach to explain the output of any machine learning model

2

Which programming languages are most used in #interpretability repositories?

Among the top ranked #interpretability projects on GitHub, the most common programming languages are Python, Jupyter Notebook, C++, TypeScript.

3

How is the #interpretability repository ranking calculated?

We track 39 active open source repositories tagged with #interpretability (500+ stars), totaling 134,415 stars with an average of 3,446 stars per repository. Rankings update daily from the GitHub API.