tutorials/explainers/KernelSHAP/kernelshap_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<img width=\"150\" alt=\"Logo_ER10\" src=\"https://user-images.githubusercontent.com/3244249/151994514-b584b984-a148-4ade-80ee-0f88b0aefa45.png\">\n",
    "\n",
    "### Model Interpretation for Pretrained Binary MNIST Model using KernelSHAP\n",
    "\n",
    "This notebook demonstrates how to apply KernelSHAP method on pretrained binary MNIST model using a hand-written digit image. It visualizes the relevance attributions for each pixel/super-pixel by displaying them on the image. <br>\n",
    "\n",
    "SHapley Additive exPlanations, in short, SHAP, is a model-agnostic explainable AI approach which is used to decrypt the black-box models through estimating the Shapley values.<br>\n",
    "\n",
    "KernelSHAP is a variant of SHAP. It is a method that uses the LIME framework to compute Shapley Values.<br>\n",
    "\n",
    "More details about this method can be found in the paper https://arxiv.org/abs/1705.07874."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Colab Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "running_in_colab = 'google.colab' in str(get_ipython())\n",
    "if running_in_colab:\n",
    "    # install dianna\n",
    "    !python3 -m pip install dianna[notebooks]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0 - Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-19 11:29:25.222033: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore') # disable warnings relateds to versions of tf\n",
    "import numpy as np\n",
    "import onnx\n",
    "from onnx_tf.backend import prepare\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import dianna\n",
    "from dianna.utils.downloader import download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 1 - Loading the model and the dataset\n",
    "Loads pretrained binary MNIST model and the image to be explained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Load saved binary MNIST data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# load dataset\n",
    "data = np.load(download('binary-mnist.npz', 'data'))\n",
    "# load testing data and the related labels\n",
    "X_test = data['X_test'].astype(np.float32).reshape([-1, 28, 28, 1]) / 255\n",
    "y_test = data['y_test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Load the pretrained binary MNIST model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Load saved onnx model\n",
    "onnx_model_path = download('mnist_model_tf.onnx', 'model')\n",
    "onnx_model = onnx.load(onnx_model_path)\n",
    "# get the output node\n",
    "output_node = prepare(onnx_model, gen_tensor_dict=True).outputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Print class and image of a single instance in the test data for preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The predicted class is: digit 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x13bdf01f0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAANrUlEQVR4nO3df4gU9xnH8c+jbf+x/UPrVcyPaluDQQqNxZhCg0lTWjQQvP6RRgnBksKZYKKBQisKqaEUQtKm/0SUCwm9ljalYNIeIq2pSG1ASs6QH+aubX6gVrmcMUIakRCjT//YMZx6853LzszOns/7BcfuzrM7+2SST2Z2vzvzNXcXgMvftKYbANAZhB0IgrADQRB2IAjCDgTxqU6+mZnx1T9QM3e3iZaX2rOb2XIz+7eZvWFmG8usC0C9rN1xdjObLuk/kr4j6aikFyStdvfhxGvYswM1q2PPvlTSG+7+lrt/KOkPklaWWB+AGpUJ+5WS/jvu8dFs2QXMrM/MhsxsqMR7ASip9i/o3L1fUr/EYTzQpDJ79mOSrh73+KpsGYAuVCbsL0i6xsy+ZGafkbRK0mA1bQGoWtuH8e7+kZndJ+mvkqZLesrdX6usMwCVanvora034zM7ULtaflQDYOog7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiCIjk7ZjM6bMWNGsv7oo48m62vXrk3WDxw4kKzffvvtubXDhw8nX4tqsWcHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYxfUyt2DBgmR9ZGSk1PqnTUvvL9avX59b27p1a6n3xsTyZnEt9aMaMzsk6X1JZyV95O5LyqwPQH2q+AXdt9z9RAXrAVAjPrMDQZQNu0vabWYHzKxvoieYWZ+ZDZnZUMn3AlBC2cP4G939mJl9QdJzZvYvd983/gnu3i+pX+ILOqBJpfbs7n4suz0u6VlJS6toCkD12g67mc0ws8+dvy/pu5IOVtUYgGqVOYyfI+lZMzu/nt+7+18q6QqfSE9PT25tYGCgg52gm7Uddnd/S9LXKuwFQI0YegOCIOxAEIQdCIKwA0EQdiAILiU9BaROE5Wk3t7e3NrSpc3+zmnZsmW5taLTY19++eVkfd++fck6LsSeHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeC4FLSU8DZs2eT9XPnznWok0sVjZWX6a1oSuc77rgjWS+aTvpylXcpafbsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAE4+xdYNeuXcn6ihUrkvUmx9nffffdZP3UqVO5tXnz5lXdzgWmT59e6/q7FePsQHCEHQiCsANBEHYgCMIOBEHYgSAIOxAE143vgJtuuilZX7hwYbJeNI5e5zj79u3bk/Xdu3cn6++9915u7ZZbbkm+dvPmzcl6kXvvvTe3tm3btlLrnooK9+xm9pSZHTezg+OWzTKz58zs9ex2Zr1tAihrMofxv5a0/KJlGyXtcfdrJO3JHgPoYoVhd/d9kk5etHilpIHs/oCk3mrbAlC1dj+zz3H30ez+25Lm5D3RzPok9bX5PgAqUvoLOnf31Aku7t4vqV/iRBigSe0OvY2Z2VxJym6PV9cSgDq0G/ZBSWuy+2sk/bmadgDUpfB8djN7WtLNkmZLGpP0U0l/kvRHSV+UdFjS99394i/xJlrXZXkYP3/+/GR9//79yfrs2bOT9TLXZi+69vqOHTuS9YceeihZP336dLKeUnQ+e9F26+npSdY/+OCD3NqDDz6YfO3jjz+erJ85cyZZb1Le+eyFn9ndfXVO6dulOgLQUfxcFgiCsANBEHYgCMIOBEHYgSC4lHQFFixYkKyPjIyUWn/R0NvevXtza6tWrUq+9sSJE2311An3339/sv7YY48l66ntVnRa8LXXXpusv/nmm8l6k7iUNBAcYQeCIOxAEIQdCIKwA0EQdiAIwg4EwaWkp4ChoaFk/e67786tdfM4epHBwcFk/c4770zWr7/++irbmfLYswNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIyzd0DR+ehFbrjhhoo6mVrMJjwt+2NF27XMdt+yZUuyftddd7W97qawZweCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIBhnr8A999yTrBddoxwTu+2225L1xYsXJ+up7V7076RonH0qKtyzm9lTZnbczA6OW7bFzI6Z2UvZ3631tgmgrMkcxv9a0vIJlv/K3a/L/nZV2xaAqhWG3d33STrZgV4A1KjMF3T3mdkr2WH+zLwnmVmfmQ2ZWfpCagBq1W7Yt0n6iqTrJI1K+mXeE929392XuPuSNt8LQAXaCru7j7n7WXc/J+kJSUurbQtA1doKu5nNHffwe5IO5j0XQHcoHGc3s6cl3SxptpkdlfRTSTeb2XWSXNIhSWvra7H7FY0HR9bT05NbW7RoUfK1mzZtqrqdj73zzjvJ+pkzZ2p776YUht3dV0+w+MkaegFQI34uCwRB2IEgCDsQBGEHgiDsQBCc4opabd68Obe2bt26Wt/70KFDubU1a9YkX3vkyJGKu2kee3YgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIJxdpSya1f6WqMLFy7sUCeXGh4ezq09//zzHeykO7BnB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGevgJkl69Omlft/6ooVK9p+bX9/f7J+xRVXtL1uqfifrcnpqrnE94XYswNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIyzV2Dbtm3J+iOPPFJq/Tt37kzWy4xl1z0OXuf6t2/fXtu6L0eFe3Yzu9rM9prZsJm9ZmYbsuWzzOw5M3s9u51Zf7sA2jWZw/iPJP3I3RdJ+oakdWa2SNJGSXvc/RpJe7LHALpUYdjdfdTdX8zuvy9pRNKVklZKGsieNiCpt6YeAVTgE31mN7P5khZL+qekOe4+mpXeljQn5zV9kvpK9AigApP+Nt7MPitph6QH3P1/42vu7pJ8ote5e7+7L3H3JaU6BVDKpMJuZp9WK+i/c/dnssVjZjY3q8+VdLyeFgFUwVo75cQTWudvDkg66e4PjFv+qKR33f1hM9soaZa7/7hgXek3m6LmzZuXrO/fvz9Z7+npSda7+TTSot7GxsZyayMjI8nX9vWlP/2Njo4m66dPn07WL1fuPuE515P5zP5NSXdJetXMXsqWbZL0sKQ/mtkPJR2W9P0K+gRQk8Kwu/vzkvKuzvDtatsBUBd+LgsEQdiBIAg7EARhB4Ig7EAQhePslb7ZZTrOXmTZsmXJem9vb7K+YcOGZL2bx9nXr1+fW9u6dWvV7UD54+zs2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMbZp4Dly5cn66nzvoumLR4cHEzWi6Z8Lpquenh4OLd25MiR5GvRHsbZgeAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtmBywzj7EBwhB0IgrADQRB2IAjCDgRB2IEgCDsQRGHYzexqM9trZsNm9pqZbciWbzGzY2b2UvZ3a/3tAmhX4Y9qzGyupLnu/qKZfU7SAUm9as3HfsrdfzHpN+NHNUDt8n5UM5n52UcljWb33zezEUlXVtsegLp9os/sZjZf0mJJ/8wW3Wdmr5jZU2Y2M+c1fWY2ZGZD5VoFUMakfxtvZp+V9HdJP3f3Z8xsjqQTklzSz9Q61L+7YB0cxgM1yzuMn1TYzezTknZK+qu7PzZBfb6kne7+1YL1EHagZm2fCGOty4c+KWlkfNCzL+7O+56kg2WbBFCfyXwbf6Okf0h6VdL5uYE3SVot6Tq1DuMPSVqbfZmXWhd7dqBmpQ7jq0LYgfpxPjsQHGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiCIwgtOVuyEpMPjHs/OlnWjbu2tW/uS6K1dVfY2L6/Q0fPZL3lzsyF3X9JYAwnd2lu39iXRW7s61RuH8UAQhB0Ioumw9zf8/ind2lu39iXRW7s60lujn9kBdE7Te3YAHULYgSAaCbuZLTezf5vZG2a2sYke8pjZITN7NZuGutH56bI59I6b2cFxy2aZ2XNm9np2O+Ecew311hXTeCemGW902zU9/XnHP7Ob2XRJ/5H0HUlHJb0gabW7D3e0kRxmdkjSEndv/AcYZrZM0ilJvzk/tZaZPSLppLs/nP2Pcqa7/6RLetuiTziNd0295U0z/gM1uO2qnP68HU3s2ZdKesPd33L3DyX9QdLKBvroeu6+T9LJixavlDSQ3R9Q6z+WjsvprSu4+6i7v5jdf1/S+WnGG912ib46oomwXynpv+MeH1V3zffuknab2QEz62u6mQnMGTfN1tuS5jTZzAQKp/HupIumGe+abdfO9Odl8QXdpW50969LWiFpXXa42pW89Rmsm8ZOt0n6ilpzAI5K+mWTzWTTjO+Q9IC7/298rcltN0FfHdluTYT9mKSrxz2+KlvWFdz9WHZ7XNKzan3s6CZj52fQzW6PN9zPx9x9zN3Puvs5SU+owW2XTTO+Q9Lv3P2ZbHHj226ivjq13ZoI+wuSrjGzL5nZZyStkjTYQB+XMLMZ2RcnMrMZkr6r7puKelDSmuz+Gkl/brCXC3TLNN5504yr4W3X+PTn7t7xP0m3qvWN/JuSNjfRQ05fX5b0cvb3WtO9SXparcO6M2p9t/FDSZ+XtEfS65L+JmlWF/X2W7Wm9n5FrWDNbai3G9U6RH9F0kvZ361Nb7tEXx3ZbvxcFgiCL+iAIAg7EARhB4Ig7EAQhB0IgrADQRB2IIj/A8nhboC3dEL1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# class name\n",
    "class_name = ['digit 0', 'digit 1']\n",
    "# instance index\n",
    "i_instance = 1\n",
    "# select instance for testing\n",
    "test_sample = X_test[i_instance].copy().astype(np.float32)\n",
    "# model predictions with added batch axis to test sample\n",
    "predictions = prepare(onnx_model).run(test_sample[None, ...])[f'{output_node}']\n",
    "pred_class = class_name[np.argmax(predictions)]\n",
    "print(\"The predicted class is:\", pred_class)\n",
    "plt.imshow(X_test[i_instance][:,:,0], cmap='gray')  # 0 for channel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 2 - Compute Shapley values and visualize the relevance attributions\n",
    "Approximate Shapley values using KernelSHAP and visualize the relevance attributions on the image. <br>\n",
    "\n",
    "KernelSHAP approximate Shapley values in the LIME framework.\n",
    "The user need to specified the number of times to re-evaluate the model when explaining each prediction (`nsamples`). A binary mask need to be applied to the image to represent if an image region is hidden. It requires the background color for the masked image, which can be specified by `background`.<br>\n",
    "\n",
    "Performing KernelSHAP on each pixel is inefficient. It is always a good practice to segment the input image and perform computations on the obtained superpixels. This requires the user to specify some keyword arguments related to the segmentation, like the (approximate) number of labels in the segmented output image (`n_segments`), and width of Gaussian smoothing kernel for pre-processing for each dimension of the image (`sigma`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6572db3492df40a0b3e06945cc130768",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 5 calls to <function BackendTFModule.__call__ at 0x147465040> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n",
      "WARNING:tensorflow:6 out of the last 6 calls to <function BackendTFModule.__call__ at 0x147465ca0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    }
   ],
   "source": [
    "# use KernelSHAP to explain the network's predictions\n",
    "shap_values = dianna.explain_image(onnx_model_path, test_sample, labels=[0, 1],\n",
    "                                  method=\"KernelSHAP\", nsamples=1000,\n",
    "                                  background=0, n_segments=200, sigma=0,\n",
    "                                  axis_labels=('height','width','channels'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Visualize Shapley scores on the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAD/CAYAAAD17AypAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUZElEQVR4nO3df5AUdXrH8c+zuwoCAfTYMyBXLEZDBa+Mv/XqEjDo+SuQ44xGjKdwVkQ8ObXqYsoTK4UXjYpVakUIhMofR12lFFQ0lBKDXlRiEg/hFD3gRCCsyhkBf0AABZb95o9pzg3TC9/e2Z6eeeb9qpra2e880/NMz/byobe7vxZCEAAAgGdNRTcAAACQNwIPAABwj8ADAADcI/AAAAD3CDwAAMA9Ag8AAHCPwAMAANwj8ACoGjM7zsyeNrPdZtZuZn9edE9AkcxsupmtNLO9ZvaTovvxrKXoBgA0lDmS9kk6XtJpkp4zs9UhhDWFdgUU59eS7pF0saRjCu7FNeNKywCqwcz6S/pU0tdDCOuTsZ9K2hJCuKPQ5oCCmdk9koaHEKYU3YtX/EkLQLX8rqSOg2EnsVrSKQX1A6CBEHgAVMsASTsPGdsh6bcK6AVAgyHwAKiWXZIGHjI2UNL/FtALgAZD4AFQLesltZjZyV3Gfl8SBywDyB2BB0BVhBB2S1os6cdm1t/Mvinp25J+WmxnQHHMrMXM+kpqltRsZn3NjDOoc0DgAVBN31fp1Nutkh6TdBOnpKPB3SXpc0l3SPpucv+uQjtyitPSAQCAe+zhAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADu9ehcfzPj1C7kKoRgRfeQxZAhQ0JbW1vRbTS8LCedWh39hG3evFnbt2+vo47ZJmqG042iJ9sEFzcCekFbW5tWvv560W00vL374n//9Tm6fv7fdtbZZxfdQmZsEzVi37742qOPzq+PXtaTbYI/aQEAAPcIPAAAwD0CDwAAcI/AAwAA3CPwAAAA9wg8AADAPQIPAABwj8ADAADc48KDQLXt2RNf269ffn04VE8XE8SXdu2Ov2DkgP58xpnU0cUE88YeHgAA4B6BBwAAuEfgAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAegQcAALjH1BIV6N+/f9nYgw8+WDZ24403pj5/1apVZWNXXnllam17e3vG7lCzmC5C6uiILn3imaOia6+8gmkH6hHTRUj7O+Kn1zjqmSfiF3zFFT3oxif28AAAAPcIPAAAwD0CDwAAcI/AAwAA3OOg5QoMHTq0bOyGG24oG+vs7Ex9/plnnlk2Nn78+NTaOXPmZOwOAAAcxB4eAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAeZ2lFaG1tTR1fsGBBlTsBatzWrXF1y5dHL/LKWrg0/mefxdcOHpxXF6hDH22NmzIiwyZRE9vEp5/FT4Vx7ODamDqEPTwAAMA9Ag8AAHCPwAMAANwj8AAAAPc4aPkQt9xyS9nYxIkTU2vPOeecXn/9MWPGpI43NZVn09WrV6fWLs9y9BsAAA2APTwAAMA9Ag8AAHCPwAMAANwj8AAAAPcIPAAAwD3O0jrEww8/XDbW2dlZtde//PLLo8fb29tTa6+66qqysVWrVlXWGGrfF1/E1/btG1/7/PPxtUuWxNXNnBm9yEdnx1/C/gfTM1zCfsOG+NqTToqvRc34/Iv4n51j+sb/7PzL8/HLzWGTkGbPjq+dPj269N0N8e/r5JNqY7qILNjDAwAA3CPwAAAA9wg8AADAPQIPAABwr2EPWl66dGnqeNoUDnn5+OOPy8Z27dqVWjtixIiysZEjR6bWrlixomysubk5Y3cAAPjBHh4AAOAegQcAALhH4AEAAO4ReAAAgHsEHgAA4F5DnKU1duzYsrFRo0al1qZNI1Hp1BLz5s1LHV+2bFnZ2I4dO1Jrx40bVzY2Y8aM6B5uuumm1PG5c+dGLwOHceCAtHNnXO3AgfHL7eiIr80yXcT27fG1ixfH1w4eHFX26KLjoxeZ26wOr70WX5tlGopLLsnei0MHDkg7dsZNVTBoYPw0Bfs78pkuYtv2+OXmsEno+EWPxi80p40i2yYRv74uvaQ2pqFgDw8AAHCPwAMAANwj8AAAAPcIPAAAwD1XBy23tbWljj/++ONlY0OGDKn49drb28vGnnrqqbKxu+++O/X5e/bsqei1pk6dmlrb2tpaNjZr1qzU2r4pB7rOnj07tXb//v2HaxEAgJrFHh4AAOAegQcAALhH4AEAAO4ReAAAgHsEHgAA4J6rs7RaWtLfTqVnZL3yyiup45MmTSob257lkv0ZpJ2ldd9996XWPvTQQ2Vj/fr1S61NO3tryZIlqbUbN248XIuNrbk525QRsbr5ma5Ylmkoli+Pr73ttqiyH0yrgUvNDxsWX3vhhfG1H3yQTw91prk525QRsY5qyednp+BNQpo2PX6hOclrk3j//fhpKIafkN/vBvbwAAAA9wg8AADAPQIPAABwj8ADAADcc3XQcm9YuXJl2dj111+fWpvXAcqxuju4+JprrikbO/vss/NuBwCAmsUeHgAA4B6BBwAAuEfgAQAA7hF4AACAew1x0HJTU3yuO/fcc3PspHeZpV+9Mu39ZlkHM2fOTB2/9tpro5eBAuzbF1+7fn187f798bXTpsXXFm3Pnvja00+Pr127Nr7W8ZWWa8HeffFX+M2ySXR0xNfeVAtXFY+UZZM444z42nXr4muHnxBfmxV7eAAAgHsEHgAA4B6BBwAAuEfgAQAA7hF4AACAe67O0prWzRkinZ2dVe6kOiZMmJA6fnrKGSXdrYO08e7O0gIAoF6xhwcAALhH4AEAAO4ReAAAgHsEHgAA4J6rg5a7O4i3nrS2tqaOjx49umzszjvvrPj1tm3bVja2P8tUAig5cEDauTOuduDAfFpo6RNd27xoUfyCHWxXqVasiK/tZrtMNW5c9l4cOnBA2rEzbmqHQQPzmX6hT8uB6NpFi5qja8eP70k3tS/LJjFkSHxtrWwS7OEBAADuEXgAAIB7BB4AAOAegQcAALhH4AEAAO65OkvLgxkzZqSO33zzzRUtd/PmzanjkydPLht77733KnotAABqDXt4AACAewQeAADgHoEHAAC4R+ABAADucdBygZYuXVo2NmrUqFxea+3atanjr776ai6v13Cam3ObMiK6haYMl+cfMCC+9q23sjdzBI/OjptyQJL27Ytf7g87HogvXrYsvvaRR+Jrm/h/pFTaJPKaMiJahs+i4E1Cmj07vjbDRvFAxw+ja/PaJDL9bsoRWyYAAHCPwAMAANwj8AAAAPcIPAAAwD0CDwAAcM/VWVpm6Wd+NGU4Uv/SSy+Nrp0/f37Z2LBhw6Kfn9ZXZ2dn9POzmDBhQi7LBQCgHrCHBwAAuEfgAQAA7hF4AACAewQeAADgnquDlufOnZs6PmvWrOhlPPvss2VjWQ4krvSg4944aHnevHkVLwN1aM+e+Nrx4+NrX3wxey9HkOUy/t+bkuGy9Pdn2H4uuyy+9rzz4mtRM3btjp/CpOBNIttGMWVKdGnn/fGLzbJJfOO82pguIgv28AAAAPcIPAAAwD0CDwAAcI/AAwAA3CPwAAAA91ydpbV48eLU8dtvv71srLW1Ne92emTbtm2p4+vWrSsbmzp1amrthx9+2Ks9AQBQ79jDAwAA3CPwAAAA9wg8AADAPQIPAABwz9VBy+3t7anjkyZNKhubOHFiau2tt97amy1ldu+996aOz5kzp8qdoO706xdfe+qp8bVLlmTv5QhOPDFD8aZN8bXXXRdfO2xYhiZQjwb0j5/+4NRT46ehyGGTyLRRbNwU3+u118a3MPyE+psuIgv28AAAAPcIPAAAwD0CDwAAcI/AAwAA3CPwAAAA91ydpdWd5cuXR41J0rJly8rGupvCYcKECWVjS1IO358/f37q883Kj7Rfu3Ztai0AAOg59vAAAAD3CDwAAMA9Ag8AAHCPwAMAANxriIOWs3j++eejxoAe27kzvnbAgPjapgz/f8lSO3BgfG2ksWOyXMI+yzwUdaajI64u+L7k/46d8VMlZNkkmpvi11uWTWLQwBw+jzFjokt/R35/HvZ3xP0s9GSTYA8PAABwj8ADAADcI/AAAAD3CDwAAMA9Ag8AAHCPwAMAANwj8AAAAPcIPAAAwD0CDwAAcI/AAwAA3GNqCaC3dHbG1eUwVYOk+GkKJKmFTb8mxH4OFj/1Qi050BnXdy5TNSh+mgJJOqrF73QN9ST2c+jJJsEeHgAA4B6BBwAAuEfgAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAegQcAALhnIXA5baBSZrZNUnvRfcCtESGE1qKbyIJtAjnLvE0QeAAAgHv8SQsAALhH4AEAAO4ReAAAgHsEHgAA4B6BBwAAuEfgAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAegQcAALhH4AEAAO4ReAAAgHsEHgAA4B6BBwAAuEfgAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAegQcAALhH4AEAAO4ReAAAgHsEHgAA4B6BBwAAuEfgAQAA7hF4AACAey2He9BMQZKausSig/djv+ZVW4vLr8We6n35tdgTy6/8OcmvFqmz88vig/djv+ZVW4vLr8We6n35tdgTy6/8OSGYusEeHgAA4B6BBwAAuEfgAQAA7hF4AACAewQeAADgHoEHAAC4R+ABAADuEXgAAIB7FkIouoeGZGZTQwjzi+6jUbH+i8X6Lxbrv3h8BtXHHp7iTC26gQbH+i8W679YrP/i8RlUGYEHAAC4R+ABAADuEXiKw99ui8X6Lxbrv1is/+LxGVQZBy0DAAD32MMDAADcI/DkyMyOM7MXzOzd5Oux3dRNTmreNbPJyVg/M3vOzH5lZmvM7P7qdl//Kln/yfi9Zva+me2qXtf1z8wuMbN3zGyDmd2R8ngfM1uYPP5zM2vr8tiPkvF3zOziqjbuRE/Xv5l9xcxeMrNdZja76o07UcH6/5aZrTKzt5Ov46revHchBG453STNknRHcv8OSQ+k1BwnaVPy9djk/rGS+kn6o6TmaEn/LunSot9TPd0qWf/JY+dJGippV9HvpV5ukpolbZR0YvJzu1rS6ENqvi9pXnJ/kqSFyf3RSX0fSSOT5TQX/Z7q6Vbh+u8v6Q8kTZM0u+j3Uo+3Ctf/6ZKGJfe/LmlL0e/H2409PPn6tqQFyf0Fkiam1Fws6YUQwichhE8lvSDpkhDCnhDCS5IUQtgn6ReShuffsis9Xv+SFEJ4LYTwYTUadeQcSRtCCJuSn9vHVfocuur6uTwp6QIzs2T88RDC3hDCf0vakCwP8Xq8/kMIu0MIr0r6onrtulPJ+n8jhPDrZHyNpGPMrE9Vum4QBJ58Hd/lH8z/kXR8Ss0Jkt7v8v0HydhvmNlgSRMk/SyHHj3rlfWPTGLW529qQggdknZI+krkc3F4lax/VK631v+fSvpFCGFvTn02pJaiG6h3ZvaipN9OeWhG129CCMHMMp8SZ2Ytkh6T9HchhE0969KvvNc/AFSTmZ0i6QFJFxXdizcEngqFEC7s7jEz+8jMhoYQPjSzoZK2ppRtkXR+l++HS3q5y/fzJb0bQnik8m79qcL6RzZbJH2ty/fDk7G0mg+SQD9I0seRz8XhVbL+UbmK1r+ZDZf0tKTrQggb82+3sfAnrXwtkXTwrJ/Jkv45peZfJV1kZscmZxFdlIzJzO5RaWO4Lf9WXapo/aNHXpd0spmNNLOjVTooc8khNV0/lysk/VsoHam5RNKk5CyWkZJOlrSiSn17Ucn6R+V6vP6TQxeeU+lEi/+oVsMNpeijpj3fVPq77M8kvSvpRUnHJeNnSfrHLnXXq3SA5gZJ30vGhksKktZJejO5/UXR76mebpWs/2R8lkp/g+9Mvs4s+j3Vw03SZZLWq3S2yoxk7MeS/iS531fSE8n6XiHpxC7PnZE87x1xVmIR63+zpE8k7Up+5kdXu/96v/V0/Uu6S9LuLr/v35T01aLfj6cbV1oGAADu8SctAADgHoEHAAC4R+ABAADuEXgAAIB7BB4AAOAegQeAJMnMZpjZGjN7y8zeNLNzk/GXzeysLnVtZvbLQ577iJltMbOmLmNTzGxbsqy1ZnZDL/R4vpk9W+lyADQerrQMQGb2DUnjJZ0RQthrZkNUmu055rlNkr6j0vxAYyW91OXhhSGE6Wb2VUlrzGxJCOGjXm4fAI6IPTwAJGmopO0hmawwhLA9fDlz85Gcr9LsznMlXZ1WEELYqtKF2EZ0HTez15K5gw5+/7KZnWVm55jZf5nZG2b2n2Y26tBlmtlMM/vLLt//0szakvvfNbMVyd6lfzCz5sj3AsApAg8ASVom6Wtmtt7M/t7Mxh7y+D8l4eFNSUsPeexqlSa4fVrSH5vZUYcu3MxOlHSiSleX7WqhpD9LaoZKGhpCWCnpV5L+MIRwuqS/lvS3sW/EzH5P0lWSvhlCOE3SAUnXxD4fgE8EHgAKIeySdKakqZK2SVpoZlO6lFwTQjgtCRCXHRxM5gu6TNIzIYSdkn4u6eIuz7sqCUmPSboxhPDJIS+9SKX5hKRS8HkyuT9I0hPJsUIPSzpF8S5I3svryWtfoFLYAtDAOIYHgCQphHBApZniXzazt1Wa4PAnR3jaxZIGS3rbzCSpn6TPJR08sHhhCGH6YV5zi5l9bGanqrRXZlry0N9IeimE8J3kz1Qvpzy9Q///P219k68maUEI4UdH6B1AA2EPDwCZ2SgzO7nL0GmS2iOeerVKk9q2hRDaJI2U9C0z65fh5RdK+itJg0IIbyVjgyRtSe5P6eZ5myWdkfR/RvLaUmnC2CuSA6VlZseZ2YjUJQBoGAQeAJI0QNKC5PTxtySNljTzcE9IQs0lkp47OBZC2C3pVUkTMrz2k5ImqfTnrYNmSbrPzN5Q93uin5J0nJmtkTRdpRmqFUJYq9LM08uS9/KCSgdlA2hgzJYOAADcYw8PAABwj8ADAADcI/AAAAD3CDwAAMA9Ag8AAHCPwAMAANwj8AAAAPcIPAAAwL3/A5sZHwB7PfppAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get the index of predictions\n",
    "top_preds = np.argsort(-predictions)\n",
    "inds = top_preds[0]\n",
    "# Visualize the explanations\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10,4))\n",
    "axes[0].imshow(test_sample, cmap='gray')\n",
    "axes[0].axis('off')\n",
    "# get the range for color bar\n",
    "max_val = np.max([np.max(np.abs(shap_values[i][:,:-1])) for i in range(len(shap_values))])\n",
    "# plot the test image and the attributions on the image for each class\n",
    "for i in range(2):\n",
    "    m = shap_values[inds[i]]\n",
    "    axes[i+1].set_title(str(inds[i]))\n",
    "    axes[i+1].imshow(test_sample, alpha=0.15)\n",
    "    im = axes[i+1].imshow(m, vmin=-max_val, vmax=max_val, cmap='bwr')\n",
    "    #axes[i+1].axis('off')\n",
    "    axes[i+1].set_xticks([])\n",
    "    axes[i+1].set_yticks([])\n",
    "cb = fig.colorbar(im, ax=axes.ravel().tolist(), label=\"SHAP value\", orientation=\"horizontal\", aspect=60)\n",
    "cb.outline.set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 3 - Conclusions\n",
    "The Shapley scores are estimated using KernelSHAP for models used to categorize the binary MNIST. The example here shows that the KernelSHAP method evaluates the importance of each segmentation/super pixel to the classification and the results are reasonable compared to the human visual preception of the chosen testing hand-written digit image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e7604e8ec5f09e490e10161e37a4725039efd3ab703d81b1b8a1e00d6741866c"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}