tutorials/explainers/KernelSHAP/kernelshap_geometric_shapes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<img width=\"150\" alt=\"Logo_ER10\" src=\"https://user-images.githubusercontent.com/3244249/151994514-b584b984-a148-4ade-80ee-0f88b0aefa45.png\">\n",
    "\n",
    "### Model Interpretation for Pretrained Model to Recognize Geometric Shapes using KernelSHAP\n",
    "\n",
    "This notebook demonstrates how to apply the KernelSHAP explainability method on a pretrained model used to classify geometric shapes. The relevance attributions for each pixel/super-pixel are visualized by displaying them on the image. <br>\n",
    "\n",
    "SHapley Additive exPlanations, in short, SHAP, is a model-agnostic explainable AI approach which is used to decrypt the black-box models through estimating the Shapley values.<br>\n",
    "\n",
    "KernelSHAP is a variant of SHAP. It is a method that uses the LIME framework to compute Shapley Values.<br>\n",
    "\n",
    "More details about this method can be found in the paper https://arxiv.org/abs/1705.07874."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Colab Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "running_in_colab = 'google.colab' in str(get_ipython())\n",
    "if running_in_colab:\n",
    "    # install dianna\n",
    "    !python3 -m pip install dianna[notebooks]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 0 - Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore') # disable warnings relateds to versions of tf\n",
    "import numpy as np\n",
    "import onnx\n",
    "from onnx_tf.backend import prepare\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import dianna\n",
    "from dianna.utils.downloader import download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 1 - Loading the model and the dataset\n",
    "Loads pretrained model and the image to be explained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Load saved geometric shapes data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# load dataset\n",
    "data = np.load(download('shapes.npz', 'data'))\n",
    "# load testing data and the related labels\n",
    "X_test = data['X_test'].astype(np.float32).reshape([-1, 1, 64, 64])\n",
    "y_test = data['y_test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Load the pretrained binary MNIST model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading data from 'doi:10.5281/zenodo.5907058/geometric_shapes_model.onnx' to file '/home/sarah/.cache/dianna/geometric_shapes_model.onnx'.\n"
     ]
    }
   ],
   "source": [
    "# Load saved onnx model\n",
    "onnx_model_path = download('geometric_shapes_model.onnx', 'model')\n",
    "onnx_model = onnx.load(onnx_model_path)\n",
    "# get the output node\n",
    "output_node = prepare(onnx_model, gen_tensor_dict=True).outputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Print class and image of a single instance in the test data for preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The predicted class is: triangle\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-25 15:43:59.064190: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f3ff2eb1510>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGfCAYAAAAZGgYhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAh4klEQVR4nO3dfXBU1f3H8U9CkiU8ZMPjJhFCY0UDIojhaQf9tYVoyjgUhXGooyMKoyMNyJMtxg6gbTWoVRSLQSkFOwKp2EFFBpGJEkYbEKKMIG0EpSYVNgia3YAkQXJ+f3TcuuRG3bDJyW7er5kzQ7735uZ7DOzHkz25N84YYwQAQBuLt90AAKBjIoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYktNaFV6xYoccee0w+n0/Dhg3T008/rVGjRn3v5zU2Nuro0aPq3r274uLiWqs9AEArMcaotrZWGRkZio//jnWOaQXFxcUmKSnJ/OUvfzEffvihufPOO01qaqqprq7+3s+tqqoykhgMBoMR5aOqquo7X+9bJYBGjRpl8vPzgx+fO3fOZGRkmMLCwu/93JqaGuv/0RgMBoNx4aOmpuY7X+8j/h5QQ0ODysvLlZubG6zFx8crNzdXZWVlTc6vr69XIBAIjtra2ki3BACw4PveRol4AJ04cULnzp2Tx+MJqXs8Hvl8vibnFxYWyu12B0f//v0j3RIAoB2yvguuoKBAfr8/OKqqqmy3BABoAxHfBde7d2916tRJ1dXVIfXq6mqlpaU1Od/lcsnlckW6DQBAOxfxFVBSUpJycnJUUlISrDU2NqqkpERerzfSXw4AEKVa5feA5s+fr2nTpmnEiBEaNWqUnnzySZ0+fVp33HFHa3w5AEAUapUAmjp1qj7//HMtXrxYPp9PV155pV5//fUmGxMAAB1XnDHG2G7i2wKBgNxut+02AAAXyO/3KyUlpdnj1nfBAQA6JgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArwg6gnTt3auLEicrIyFBcXJxefvnlkOPGGC1evFjp6elKTk5Wbm6uDh06FKl+AQAxIuwAOn36tIYNG6YVK1Y4Hn/00Ue1fPlyrVy5Urt371bXrl2Vl5enurq6C24WABBDzAWQZDZt2hT8uLGx0aSlpZnHHnssWKupqTEul8ts2LDB8Rp1dXXG7/cHR1VVlZHEYDAYjCgffr//OzMkou8BHTlyRD6fT7m5ucGa2+3W6NGjVVZW5vg5hYWFcrvdwdG/f/9ItgQAaKciGkA+n0+S5PF4Quoejyd47HwFBQXy+/3BUVVVFcmWAADtVILtBlwul1wul+02AABtLKIBlJaWJkmqrq5Wenp6sF5dXa0rr7wykl8KQAdx/k9UvvHFF1841s+ePdua7SCCIvojuKysLKWlpamkpCRYCwQC2r17t7xebyS/FAAgyoW9Ajp16pQOHz4c/PjIkSPat2+fevbsqczMTM2dO1d/+MMfNHDgQGVlZWnRokXKyMjQDTfcEMm+AQDRLtyt12+99Zbjdrtp06YFt2IvWrTIeDwe43K5zPjx401FRcUPvr7f77e+dZDBYLSf4fF4HEdiYqLjsN0v43/j+7ZhxxljjNqRQCAgt9ttuw0A7QTvAUUvv9+vlJSUZo9b3wUHoOPp27dvk9rTTz/teO5NN93kWO/SpYtjnQCKHtyMFABgBQEEALCCAAIAWEEAAQCsIIAAAFawCw7ABevTp49jffny5Y71qVOnNqnFxcWF9TXr6+vDOh/tDysgAIAVBBAAwAoCCABgBQEEALCCAAIAWMEuOABN9OrVy7He3K62m2++2bEe7s42Jw0NDY71dnYfZbQAKyAAgBUEEADACgIIAGAFAQQAsIIAAgBYwS44oIPo2bOnY/3JJ59sUrv11lsdz43ErrZw1dXVtfnXRNtgBQQAsIIAAgBYQQABAKwggAAAVrAJAYhSPXr0cKwvW7bMsX7bbbc51m1sLAjHV199ZbsFtBJWQAAAKwggAIAVBBAAwAoCCABgBQEEALCCXXBAO5KamupYf/zxx5vUbr/9dsdz4+Nj6/8rA4GA7RbQSmLrbyoAIGoQQAAAKwggAIAVBBAAwAoCCABgBbvggFbkdrsd63/84x8d69OnT3esx9rOtnCwCy52ddy/1QAAqwggAIAVBBAAwAoCCABgBQEEALCCXXBAmFJSUprUHn30Ucdz77zzTsd6R97VFq4vvvjCdgtoJfwrAABYQQABAKwggAAAVhBAAAArwgqgwsJCjRw5Ut27d1ffvn11ww03qKKiIuScuro65efnq1evXurWrZumTJmi6urqiDYNAIh+Ye2CKy0tVX5+vkaOHKmvv/5a999/v6677jodPHhQXbt2lSTNmzdPW7Zs0caNG+V2uzVr1ixNnjxZ77zzTqtMALhQ3bt3d6w/8sgjjvW77rqrSa1Tp04R7Qn/c/LkSdstoJWEFUCvv/56yMdr165V3759VV5erv/7v/+T3+/X6tWrtX79eo0bN06StGbNGg0aNEi7du3SmDFjItc5ACCqXdB7QH6/X5LUs2dPSVJ5ebnOnj2r3Nzc4DnZ2dnKzMxUWVmZ4zXq6+sVCARCBgAg9rU4gBobGzV37lyNHTtWQ4YMkST5fD4lJSUpNTU15FyPxyOfz+d4ncLCQrnd7uDo379/S1sCAESRFgdQfn6+Dhw4oOLi4gtqoKCgQH6/Pziqqqou6HoAgOjQolvxzJo1S6+99pp27typfv36BetpaWlqaGhQTU1NyCqourpaaWlpjtdyuVxyuVwtaQNoVrdu3ZrUCgsLHc+dOXOmY52NBe3DiRMnbLeAVhLWCsgYo1mzZmnTpk168803lZWVFXI8JydHiYmJKikpCdYqKipUWVkpr9cbmY4BADEhrBVQfn6+1q9fr1deeUXdu3cPvq/jdruVnJwst9utGTNmaP78+erZs6dSUlI0e/Zseb1edsABAEKEFUBFRUWSpJ/+9Kch9TVr1uj222+XJC1btkzx8fGaMmWK6uvrlZeXp2eeeSYizQIAYkdYAWSM+d5zOnfurBUrVmjFihUtbgoAEPu4FxwAwAoeSIeo8M2tns738MMPO9bz8/Ob1NjVFp0+//xz2y2glbACAgBYQQABAKwggAAAVhBAAAArCCAAgBXsgoMVze1q+/3vf+9Ynz17tmM9IYG/wrGOXXCxixUQAMAKAggAYAUBBACwggACAFhBAAEArGALESKmS5cujvUHH3ywSW3u3LmO57KrDefjiaixixUQAMAKAggAYAUBBACwggACAFhBAAEArGDLEZqVnJzsWH/ggQcc6/PmzXOsJyYmRqoldEAnT5603QJaCSsgAIAVBBAAwAoCCABgBQEEALCCTQgdTOfOnZvUFi9e7Hjuvffe61hnUwHa0pdffmm7BbQSVkAAACsIIACAFQQQAMAKAggAYAUBBACwgl1wUc5pV5sk/fa3v3Ws/+Y3v2lSS0pKimhPQCQFAgHbLaCVsAICAFhBAAEArCCAAABWEEAAACsIIACAFeyCa4dcLleT2v333+947n333edYZ2cbYkVtba3tFtBKWAEBAKwggAAAVhBAAAArCCAAgBUEEADACnbBtQGnXW2StHDhQse604635q4BxLozZ87YbgGthBUQAMAKAggAYAUBBACwggACAFgR1iaEoqIiFRUV6d///rck6fLLL9fixYs1YcIESVJdXZ0WLFig4uJi1dfXKy8vT88884w8Hk/EG7epudvcOD3sTWr+4XDNPUwOwP/U1dXZbgGtJKwVUL9+/bR06VKVl5dr7969GjdunCZNmqQPP/xQkjRv3jxt3rxZGzduVGlpqY4eParJkye3SuMAgOgW1gpo4sSJIR8/9NBDKioq0q5du9SvXz+tXr1a69ev17hx4yRJa9as0aBBg7Rr1y6NGTMmcl0DAKJei98DOnfunIqLi3X69Gl5vV6Vl5fr7Nmzys3NDZ6TnZ2tzMxMlZWVNXud+vp6BQKBkAEAiH1hB9D+/fvVrVs3uVwu3X333dq0aZMGDx4sn8+npKQkpaamhpzv8Xjk8/mavV5hYaHcbndw9O/fP+xJAACiT9gBdNlll2nfvn3avXu3Zs6cqWnTpungwYMtbqCgoEB+vz84qqqqWnwtAED0CPtWPElJSbrkkkskSTk5OdqzZ4+eeuopTZ06VQ0NDaqpqQlZBVVXVystLa3Z67lcrqi7zUxCgvN/tpEjRzrW2e0GfL/GxkbH+tdff93GnaCtXPDvATU2Nqq+vl45OTlKTExUSUlJ8FhFRYUqKyvl9Xov9MsAAGJMWCuggoICTZgwQZmZmaqtrdX69eu1Y8cObdu2TW63WzNmzND8+fPVs2dPpaSkaPbs2fJ6veyAAwA0EVYAHT9+XLfddpuOHTsmt9utoUOHatu2bbr22mslScuWLVN8fLymTJkS8ouoAACcL84YY2w38W2BQEBut9t2G9+pS5cujvUNGzY41n/xi1+0ZjtATGjuPaBOnTq1cSeIFL/fr5SUlGaPcy84AIAVrIDawDc/ojzf3//+d8d69+7dW7MdoF366quvHOtdu3Zt404QKayAAADtEgEEALCCAAIAWEEAAQCsIIAAAFaEfS84hG/79u2O9T59+jjW165d26T2y1/+MpItAe0OTz7teFgBAQCsIIAAAFYQQAAAKwggAIAVBBAAwAruBRclrr76asf6q6++6ljv0aNHa7YDRFxlZaVjfcCAAW3cCSKFe8EBANolAggAYAUBBACwggACAFjBrXiixNtvv+1Y93g8jvWVK1c61qdPnx6xnoBICgQCtltAG2MFBACwggACAFhBAAEArCCAAABWEEAAACvYBRflzp4961ifMWOGY72oqKhJbcuWLY7n9u3bt+WNAWGqqamx3QLaGCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWMEuuA5m7969TWoXXXSR47nLli1zrM+aNSuiPQGS9MUXX9huAW2MFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsYBcc9PXXXzvWZ8+e7Vh/9tlnHetbt251rPfr169ljaFDOXHihO0W0MZYAQEArCCAAABWEEAAACsIIACAFWxCQNgOHDjgWB8wYIBjvbCwsEnt17/+teO5cXFxLW8MUe3zzz+33QLaGCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWHFBu+CWLl2qgoICzZkzR08++aQkqa6uTgsWLFBxcbHq6+uVl5enZ555Rh6PJxL9oh1rbGx0rC9cuLBJbdWqVY7nvvHGG471rKysljeGqMAuuI6nxSugPXv26Nlnn9XQoUND6vPmzdPmzZu1ceNGlZaW6ujRo5o8efIFNwoAiC0tCqBTp07plltu0apVq9SjR49g3e/3a/Xq1XriiSc0btw45eTkaM2aNfrHP/6hXbt2RaxpAED0a1EA5efn6/rrr1dubm5Ivby8XGfPng2pZ2dnKzMzU2VlZY7Xqq+vVyAQCBkAgNgX9ntAxcXFeu+997Rnz54mx3w+n5KSkpSamhpS93g88vl8jtcrLCzUgw8+GG4bAIAoF9YKqKqqSnPmzNG6devUuXPniDRQUFAgv98fHFVVVRG5LgCgfQtrBVReXq7jx4/rqquuCtbOnTunnTt36k9/+pO2bdumhoYG1dTUhKyCqqurlZaW5nhNl8sll8vVsu4RtQ4fPuxY//GPf+xYX7RokWN9yZIljvX4eH7DINqwC67jCSuAxo8fr/3794fU7rjjDmVnZ2vhwoXq37+/EhMTVVJSoilTpkiSKioqVFlZKa/XG7muAQBRL6wA6t69u4YMGRJS69q1q3r16hWsz5gxQ/Pnz1fPnj2VkpKi2bNny+v1asyYMZHrGgAQ9SL+OIZly5YpPj5eU6ZMCflFVAAAvu2CA2jHjh0hH3fu3FkrVqzQihUrLvTSAIAYxju1AAAreCIq2hVjjGP9d7/7nWP9+eefd6xv27atSe2yyy5reWNodSdPnrTdAtoYKyAAgBUEEADACgIIAGAFAQQAsIIAAgBYwS44RLVPP/3UsT5o0KAmtfnz5zue+8gjjzjWO3Xq1PLGEDZ2wXU8rIAAAFYQQAAAKwggAIAVBBAAwIo409y9TywJBAJyu92220AHkp6e7ljfunWrY33YsGGt2U6HdfnllzvWDx482MadIFL8fr9SUlKaPc4KCABgBQEEALCCAAIAWEEAAQCsIIAAAFZwKx50eMeOHXOsX3nllY71mTNnNqktX77c8dyEBP6J/VCBQMB2C2hjrIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBfeCAyKgd+/ejvUtW7Y41keNGtWa7USlHj16ONZramrathFEDPeCAwC0SwQQAMAKAggAYAUBBACwggACAFjBLjjAgmnTpjnWn3vuOcd6UlJSa7bTLnTp0sWxfubMmTbuBJHCLjgAQLtEAAEArCCAAABWEEAAACvYhAC0I6mpqY71l19+uUntJz/5Ses208bi453/f7idvUQhDGxCAAC0SwQQAMAKAggAYAUBBACwggACAFjBLjggSt10002O9b/+9a+O9c6dO7dmO2FpaGhoUnO5XBY6QWtiFxwAoF0igAAAVhBAAAArCCAAgBUEEADAirB2wT3wwAN68MEHQ2qXXXaZ/vWvf0mS6urqtGDBAhUXF6u+vl55eXl65pln5PF4fnBD7IIDLky3bt0c6y+99JJjPS8vrzXbcVRbW9uk9l27pRCdIr4L7vLLL9exY8eC4+233w4emzdvnjZv3qyNGzeqtLRUR48e1eTJk1vWOQAgpiWE/QkJCUpLS2tS9/v9Wr16tdavX69x48ZJktasWaNBgwZp165dGjNmjOP16uvrVV9fH/w4EAiE2xIAIAqFvQI6dOiQMjIydPHFF+uWW25RZWWlJKm8vFxnz55Vbm5u8Nzs7GxlZmaqrKys2esVFhbK7XYHR//+/VswDQBAtAkrgEaPHq21a9fq9ddfV1FRkY4cOaJrrrlGtbW18vl8SkpKavI8E4/HI5/P1+w1CwoK5Pf7g6OqqqpFEwEARJewfgQ3YcKE4J+HDh2q0aNHa8CAAXrxxReVnJzcogZcLhe34ACADijs94C+LTU1VZdeeqkOHz6sa6+9Vg0NDaqpqQlZBVVXVzu+ZwSgdZw6dcqx/vOf/9yxfv311zepFRcXO57b3A67cH311VcRuQ6i2wX9HtCpU6f08ccfKz09XTk5OUpMTFRJSUnweEVFhSorK+X1ei+4UQBAbAlrBXTvvfdq4sSJGjBggI4ePaolS5aoU6dOuvnmm+V2uzVjxgzNnz9fPXv2VEpKimbPni2v19vsDjgAQMcVVgD95z//0c0336yTJ0+qT58+uvrqq7Vr1y716dNHkrRs2TLFx8drypQpIb+ICgDA+XgeENDB2XgPqLq6ukmN94pjD88DAgC0Sxe0Cw5A9NuyZUuTWt++fR3PXbdunWP9xhtvDOtrcscTSKyAAACWEEAAACsIIACAFQQQAMAKNiEAaOLMmTOO9eae7/Wzn/3Msb5p0ybHutMD6dDxsAICAFhBAAEArCCAAABWEEAAACsIIACAFeyCA3DB3nrrLce6x+NxrE+fPr0120GUYAUEALCCAAIAWEEAAQCsIIAAAFYQQAAAK3gkNwCgVfBIbgBAu0QAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFaEHUCfffaZbr31VvXq1UvJycm64oortHfv3uBxY4wWL16s9PR0JScnKzc3V4cOHYpo0wCA6BdWAH355ZcaO3asEhMTtXXrVh08eFCPP/64evToETzn0Ucf1fLly7Vy5Urt3r1bXbt2VV5enurq6iLePAAgipkwLFy40Fx99dXNHm9sbDRpaWnmscceC9ZqamqMy+UyGzZs+EFfw+/3G0kMBoPBiPLh9/u/8/U+rBXQq6++qhEjRuimm25S3759NXz4cK1atSp4/MiRI/L5fMrNzQ3W3G63Ro8erbKyMsdr1tfXKxAIhAwAQOwLK4A++eQTFRUVaeDAgdq2bZtmzpype+65R88//7wkyefzSZI8Hk/I53k8nuCx8xUWFsrtdgdH//79WzIPAECUCSuAGhsbddVVV+nhhx/W8OHDddddd+nOO+/UypUrW9xAQUGB/H5/cFRVVbX4WgCA6BFWAKWnp2vw4MEhtUGDBqmyslKSlJaWJkmqrq4OOae6ujp47Hwul0spKSkhAwAQ+8IKoLFjx6qioiKk9tFHH2nAgAGSpKysLKWlpamkpCR4PBAIaPfu3fJ6vRFoFwAQM37Y/rf/evfdd01CQoJ56KGHzKFDh8y6detMly5dzAsvvBA8Z+nSpSY1NdW88sor5oMPPjCTJk0yWVlZ5syZM+yCYzAYjA40vm8XXFgBZIwxmzdvNkOGDDEul8tkZ2eb5557LuR4Y2OjWbRokfF4PMblcpnx48ebioqKH3x9AojBYDBiY3xfAMUZY4zakUAgILfbbbsNAMAF8vv93/m+PveCAwBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAAr2l0AtbN7owIAWuj7Xs/bXQDV1tbabgEAEAHf93re7h7H0NjYqKNHj6p79+6qra1V//79VVVVFdOP6g4EAswzRnSEOUrMM9ZEep7GGNXW1iojI0Px8c2vcxIu+CtFWHx8vPr16ydJiouLkySlpKTE9Df/G8wzdnSEOUrMM9ZEcp4/5Llu7e5HcACAjoEAAgBY0a4DyOVyacmSJXK5XLZbaVXMM3Z0hDlKzDPW2Jpnu9uEAADoGNr1CggAELsIIACAFQQQAMAKAggAYAUBBACwol0H0IoVK/SjH/1InTt31ujRo/Xuu+/abumC7Ny5UxMnTlRGRobi4uL08ssvhxw3xmjx4sVKT09XcnKycnNzdejQITvNtlBhYaFGjhyp7t27q2/fvrrhhhtUUVERck5dXZ3y8/PVq1cvdevWTVOmTFF1dbWljlumqKhIQ4cODf7muNfr1datW4PHY2GO51u6dKni4uI0d+7cYC0W5vnAAw8oLi4uZGRnZwePx8Icv/HZZ5/p1ltvVa9evZScnKwrrrhCe/fuDR5v69egdhtAf/vb3zR//nwtWbJE7733noYNG6a8vDwdP37cdmstdvr0aQ0bNkwrVqxwPP7oo49q+fLlWrlypXbv3q2uXbsqLy9PdXV1bdxpy5WWlio/P1+7du3S9u3bdfbsWV133XU6ffp08Jx58+Zp8+bN2rhxo0pLS3X06FFNnjzZYtfh69evn5YuXary8nLt3btX48aN06RJk/Thhx9Kio05ftuePXv07LPPaujQoSH1WJnn5ZdfrmPHjgXH22+/HTwWK3P88ssvNXbsWCUmJmrr1q06ePCgHn/8cfXo0SN4Tpu/Bpl2atSoUSY/Pz/48blz50xGRoYpLCy02FXkSDKbNm0KftzY2GjS0tLMY489FqzV1NQYl8tlNmzYYKHDyDh+/LiRZEpLS40x/51TYmKi2bhxY/Ccf/7zn0aSKSsrs9VmRPTo0cP8+c9/jrk51tbWmoEDB5rt27ebn/zkJ2bOnDnGmNj5Xi5ZssQMGzbM8ViszNEYYxYuXGiuvvrqZo/beA1qlyughoYGlZeXKzc3N1iLj49Xbm6uysrKLHbWeo4cOSKfzxcyZ7fbrdGjR0f1nP1+vySpZ8+ekqTy8nKdPXs2ZJ7Z2dnKzMyM2nmeO3dOxcXFOn36tLxeb8zNMT8/X9dff33IfKTY+l4eOnRIGRkZuvjii3XLLbeosrJSUmzN8dVXX9WIESN00003qW/fvho+fLhWrVoVPG7jNahdBtCJEyd07tw5eTyekLrH45HP57PUVev6Zl6xNOfGxkbNnTtXY8eO1ZAhQyT9d55JSUlKTU0NOTca57l//35169ZNLpdLd999tzZt2qTBgwfH1ByLi4v13nvvqbCwsMmxWJnn6NGjtXbtWr3++usqKirSkSNHdM0116i2tjZm5ihJn3zyiYqKijRw4EBt27ZNM2fO1D333KPnn39ekp3XoHb3OAbEjvz8fB04cCDk5+mx5LLLLtO+ffvk9/v10ksvadq0aSotLbXdVsRUVVVpzpw52r59uzp37my7nVYzYcKE4J+HDh2q0aNHa8CAAXrxxReVnJxssbPIamxs1IgRI/Twww9LkoYPH64DBw5o5cqVmjZtmpWe2uUKqHfv3urUqVOTnSbV1dVKS0uz1FXr+mZesTLnWbNm6bXXXtNbb70VfL6T9N95NjQ0qKamJuT8aJxnUlKSLrnkEuXk5KiwsFDDhg3TU089FTNzLC8v1/Hjx3XVVVcpISFBCQkJKi0t1fLly5WQkCCPxxMT8zxfamqqLr30Uh0+fDhmvpeSlJ6ersGDB4fUBg0aFPxxo43XoHYZQElJScrJyVFJSUmw1tjYqJKSEnm9XoudtZ6srCylpaWFzDkQCGj37t1RNWdjjGbNmqVNmzbpzTffVFZWVsjxnJwcJSYmhsyzoqJClZWVUTVPJ42Njaqvr4+ZOY4fP1779+/Xvn37gmPEiBG65ZZbgn+OhXme79SpU/r444+Vnp4eM99LSRo7dmyTX4n46KOPNGDAAEmWXoNaZWtDBBQXFxuXy2XWrl1rDh48aO666y6TmppqfD6f7dZarLa21rz//vvm/fffN5LME088Yd5//33z6aefGmOMWbp0qUlNTTWvvPKK+eCDD8ykSZNMVlaWOXPmjOXOf7iZM2cat9ttduzYYY4dOxYcX331VfCcu+++22RmZpo333zT7N2713i9XuP1ei12Hb777rvPlJaWmiNHjpgPPvjA3HfffSYuLs688cYbxpjYmKOTb++CMyY25rlgwQKzY8cOc+TIEfPOO++Y3Nxc07t3b3P8+HFjTGzM0Rhj3n33XZOQkGAeeughc+jQIbNu3TrTpUsX88ILLwTPaevXoHYbQMYY8/TTT5vMzEyTlJRkRo0aZXbt2mW7pQvy1ltvGUlNxrRp04wx/90GuWjRIuPxeIzL5TLjx483FRUVdpsOk9P8JJk1a9YEzzlz5oz51a9+ZXr06GG6dOlibrzxRnPs2DF7TbfA9OnTzYABA0xSUpLp06ePGT9+fDB8jImNOTo5P4BiYZ5Tp0416enpJikpyVx00UVm6tSp5vDhw8HjsTDHb2zevNkMGTLEuFwuk52dbZ577rmQ4239GsTzgAAAVrTL94AAALGPAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs+H9FhHJc4OWIhAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# class name\n",
    "class_name = ['circle', 'triangle']\n",
    "# instance index\n",
    "i_instance = 4\n",
    "# select instance for testing\n",
    "test_sample = X_test[i_instance].copy().astype(np.float32)\n",
    "# model predictions with added batch axis to test sample\n",
    "predictions = prepare(onnx_model).run(test_sample[None, ...])[f'{output_node}']\n",
    "pred_class = class_name[np.argmax(predictions)]\n",
    "print(\"The predicted class is:\", pred_class)\n",
    "plt.imshow(X_test[i_instance][0,:,:], cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 2 - Compute Shapley values and visualize the relevance attributions\n",
    "Approximate Shapley values using KernelSHAP and visualize the relevance attributions on the image. <br>\n",
    "\n",
    "KernelSHAP approximate Shapley values in the LIME framework.\n",
    "The user need to specified the number of times to re-evaluate the model when explaining each prediction (`nsamples`). A binary mask need to be applied to the image to represent if an image region is hidden. It requires the background color for the masked image, which can be specified by `background`.<br>\n",
    "\n",
    "Performing KernelSHAP on each pixel is inefficient. It is always a good practice to segment the input image and perform computations on the obtained superpixels. This requires the user to specify some keyword arguments related to the segmentation, like the (approximate) number of labels in the segmented output image (`n_segments`), and width of Gaussian smoothing kernel for pre-processing for each dimension of the image (`sigma`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-25 15:44:06.381574: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b86a64f288864e3eb7067245a411842c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-25 15:44:06.613572: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n",
      "2024-06-25 15:44:17.294571: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 65536000 exceeds 10% of free system memory.\n",
      "2024-06-25 15:44:17.501726: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n",
      "2024-06-25 15:44:17.512529: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 73984000 exceeds 10% of free system memory.\n",
      "2024-06-25 15:44:17.529496: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 524288000 exceeds 10% of free system memory.\n",
      "2024-06-25 15:44:17.941527: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 524288000 exceeds 10% of free system memory.\n",
      "2024-06-25 15:44:18.219229: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 524288000 exceeds 10% of free system memory.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 5 calls to <function BackendTFModule.__call__ at 0x7f3f2d90f600> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-06-25 15:44:20.327126: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n",
      "2024-06-25 15:44:31.096407: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:933] Skipping loop optimization for Merge node with control input: assert_equal_1/Assert/AssertGuard/branch_executed/_9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:6 out of the last 6 calls to <function BackendTFModule.__call__ at 0x7f4099106a20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    }
   ],
   "source": [
    "# use KernelSHAP to explain the network's predictions\n",
    "shap_values = dianna.explain_image(onnx_model_path, test_sample,\n",
    "                                  method=\"KernelSHAP\", labels=[0, 1], nsamples=2000,\n",
    "                                  n_segments=200, sigma=0,\n",
    "                                  axis_labels=('channels','height','width'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Visualize Shapley scores on the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# used only for plotting\n",
    "def normalize(data):\n",
    "    \"\"\"Squash all values into [-1,1] range.\"\"\"\n",
    "    zero_to_one = (data - np.min(data)) / (np.max(data) - np.min(data))\n",
    "    return 2*zero_to_one -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz8AAAFoCAYAAAB5QUUlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxLklEQVR4nO3deXgUVb7/8U8WkhCyACYQ2XeUTdzIsAioKKACLtcxCALCwIjCuCIiItuMLC644+iFgHdAlEVkLssoAncEGVQEZdERkGVQQBIMTczCkvP7w19KigSSTiXpJOf9ep48z+mqU9/+dp2u6v5yqpogY4wRAAAAAFRwwYFOAAAAAABKA8UPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAKDIgoKCNGHChECnIUlat26dgoKCtG7dukCnAsCjkjiey9L5CoFD8QMAkCR9+umnmjBhgtLS0gKdCgAAJSI00AkAAMqGTz/9VBMnTtSgQYNUtWrVQm2TmZmp0FA+SgAUr86dOyszM1NhYWGBTgUVDDM/AAC/5OTkKCsrS5IUERFB8QOg2AUHBysiIkLBwRf+qpqRkVFKGaGioPgBAGjChAkaNWqUJKlhw4YKCgpSUFCQ9u3bp6CgII0YMULz5s1Ty5YtFR4erlWrVknKew39/v37df/996t58+aqXLmyLrroIt15553at2+f6/nmzJmjoKAgbdiwQY888oji4+NVpUoV3XbbbTp69Kirb05OjiZMmKBatWopMjJS1157rXbu3KkGDRpo0KBBBb62TZs2qUePHoqNjVVkZKS6dOmiDRs2eNpfALz74YcfNGTIENWqVUvh4eFq2LChhg8frpMnT+Z7z0/Xrl3VqlUrbd68WZ07d1ZkZKSefPJJSVJWVpYmTJigZs2aKSIiQhdffLFuv/127dmzp8AcBg8erJo1ayo8PFwtW7bU7NmzS/JlI8D45zoAgG6//XZ99913eueddzRjxgzFxcVJkuLj4yVJa9as0XvvvacRI0YoLi5ODRo0yDfO559/rk8//VRJSUmqU6eO9u3bp5kzZ6pr167auXOnIiMjXf1HjhypatWqafz48dq3b59efPFFjRgxQu+++67TZ8yYMZo+fbp69eql7t2766uvvlL37t2d2acLWbNmjXr27Kkrr7xS48ePV3BwsJKTk3Xdddfpk08+Ubt27Yq4xwB48eOPP6pdu3ZKS0vTsGHDdMkll+iHH37QokWLLjibk5qaqp49eyopKUn9+/dXzZo1debMGd1yyy36+OOPlZSUpAcffFAnTpzQRx99pO3bt6tx48b5xjpy5Ih+97vfOf/AEx8fr5UrV2rIkCHy+Xx66KGHSujVI6AMAADGmGeffdZIMnv37nUtl2SCg4PNjh078mwjyYwfP955nJGRkafPxo0bjSTz9ttvO8uSk5ONJNOtWzeTk5PjLH/44YdNSEiISUtLM8YYc/jwYRMaGmpuvfVWV8wJEyYYSWbgwIHOsrVr1xpJZu3atcYYY3JyckzTpk1N9+7dXc+RkZFhGjZsaG644YYC9wmAkjFgwAATHBxsPv/88zzrcnJy8hzPxhjTpUsXI8m88cYbrv6zZ882kswLL7yQb6xc556vhgwZYi6++GKTkpLi2iYpKcnExsbmez5D+cdlbwCAAnXp0kUtWrQosF/lypWd9qlTp5SamqomTZqoatWq+vLLL/P0HzZsmIKCgpzH11xzjc6cOaP9+/dLkj7++GOdPn1a999/v2u7kSNHFpjL1q1btWvXLt19991KTU1VSkqKUlJS9Msvv+j666/XP//5T+Xk5BQYB0DxysnJ0dKlS9WrVy9dddVVedaffU44V3h4uO69917XssWLFysuLi7f88L5YhljtHjxYvXq1UvGGOf8kJKSou7du+v48eP5nrNQ/nHZGwCgQA0bNixUv8zMTE2ZMkXJycn64YcfZIxx1h0/fjxP/3r16rkeV6tWTZL0888/S5JTBDVp0sTVr3r16k7f89m1a5ckaeDAgeftc/z48QLjACheR48elc/nU6tWrfzetnbt2nl+AW7Pnj1q3ry5Xz++cvToUaWlpenNN9/Um2++mW+fn376ye/8UPZR/AAACnT2jM6FjBw5UsnJyXrooYfUvn17xcbGKigoSElJSfnOsoSEhOQb5+yiqahyn+/ZZ59V27Zt8+0TFRXl+XkAlJ7CnosKknt+6N+//3n/gaRNmzbF8lwoWyh+AACSLnypSWEtWrRIAwcO1PPPP+8sy8rKKvJ/nFq/fn1J0u7du12zT6mpqc7s0Pnk3uQcExOjbt26Fen5ARS/+Ph4xcTEaPv27cUSr3Hjxtq0aZNOnTqlSpUqFTqH6OhonTlzhvODZbjnBwAgSapSpYokFblQkX6dyTl31uaVV17RmTNnihTv+uuvV2hoqGbOnOla/uqrrxa47ZVXXqnGjRvrueeeU3p6ep715/6kNoDSERwcrFtvvVV///vf9cUXX+RZ7+/M7x133KGUlJR8zwvnixUSEqI77rhDixcvzrcI4/xQcTHzAwCQ9GuxIEljx45VUlKSKlWqpF69evkV45ZbbtH//M//KDY2Vi1atNDGjRu1evVqXXTRRUXKqWbNmnrwwQf1/PPPq3fv3urRo4e++uorrVy5UnFxcRecrQoODtZ///d/q2fPnmrZsqXuvfde1a5dWz/88IPWrl2rmJgY/f3vfy9SXgC8eeaZZ/Thhx+qS5cuGjZsmC699FIdOnRICxcu1Pr16/2KNWDAAL399tt65JFH9Nlnn+maa67RL7/8otWrV+v+++9Xnz598t1u6tSpWrt2rRITEzV06FC1aNFCx44d05dffqnVq1fr2LFjxfFSUcZQ/AAAJElXX321Jk+erDfeeEOrVq1STk6O9u7d61eMl156SSEhIZo3b56ysrLUsWNHrV69Wt27dy9yXtOmTVNkZKTeeustrV69Wu3bt9eHH36oTp06KSIi4oLbdu3aVRs3btTkyZP16quvKj09XQkJCUpMTNQf//jHIucEwJvatWtr06ZNGjdunObNmyefz6fatWurZ8+eef4/sIKEhIRoxYoV+stf/qL58+dr8eLFuuiii9SpUye1bt36vNvVrFlTn332mSZNmqQlS5bo9ddf10UXXaSWLVtq2rRpXl8iyqggUxx3lQIAUIrS0tJUrVo1/fnPf9bYsWMDnQ4AoJzgnh8AQJmWmZmZZ9mLL74o6deZHQAACovL3gAAZdq7776rOXPm6KabblJUVJTWr1+vd955RzfeeKM6duwY6PQAAOUIxQ8AoExr06aNQkNDNX36dPl8PudHEP785z8HOjUAQDnDPT8AAAAArMA9PwAAAACsQPEDAAAAwArc8wOgwsnJydGPP/6o6OjoC/4nmAD8Y4zRiRMnVKtWLQUHV9x/P+UcApScQJ9HymTxw4kGKDk23Ob3448/qm7duoFOA6iw/vOf/6hOnTqBTqPEcA4BSl6gziNlsvgBAC+io6MlSS+99B9VrhxT5Dh3jarnOZfsXQc8xwhfs9JbDtf19J7D5+s9x1D16p5DLP6mhecYP/3kOYTefdd7jHXXTvQe5M47PW2e3di//XnihE+NG9d1jrGKKvf1/efJJxUTERHgbICKxZeVpbrPPBOw8wjFD4AKJ3f2uHLlGE/FT0wxzEJnxxT9+XOFR0YGPocqVTzHUFSU5xCRkd5fS3F8lw0J8R4jJjzcexCP+7So742KfoVG7uuLiYig+AFKSKDOIxX3gl0AAAAAOAvFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArhAY6AQAoKXc12ayYqKiiB/jiC885fPaZ5xD68vtenra/z3sKyv5dF88xwtd/7DnGXQmpnmPo5AHPIe5bf4/nGNl6xnOM8AO7PG0/Z45//TMzPT0dgFw//+w9RrVq3mNYiJkfAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABgBYofAAAAAFag+AEAAABghdBAJwAAJSW71ZXKjokJaA7TH/Ie4+8tRnsLkPGE9ySqV/ce47nnvMcIC/McYlbkSM8x+nuOUDyGTm/qafvLLvOv/6lTnp4OCLz0dO8xoqK8x6hWzXuM4nDsmLfti+OzoZQx8wMAAADAChQ/AAAAAKzAZW+wTs2aNZ32sXOme09xTQcAAECFxcwPAAAAACtQ/AAAAACwApe9oUKqUaOG037llVdc6+68806nHRkZ6VrHZW8AAAAVFzM/AAAAAKxA8QMAAADAChQ/AAAAAKzAPT8ot+Lj412PX375Zad91113Oe2goKDzxsjOzi7+xAAAAFAmMfMDAAAAwAoUPwAAAACswGVvKPMuuugip332pW19+/Z19bvQ5W1nO3nypNM2xnjMDgAAAOUFMz8AAAAArEDxAwAAAMAKFD8AAAAArMA9PygTqlev7rRffPFF17r+/fs77cLe13MhWVlZnmOgfAjfs1PhUVFF3n7jiVaecxg82HMIZd80zdP2S5d6z+GuQYO8B9mxw3uMfv08h+jfyXsaEyd6j/HRR95jrF/vPYY/fD7p8cdL9zmBYuXhM8HRoIH3GPv2eY9RHM76/mULZn4AAAAAWIHiBwAAAIAVuOwNpaZatWpOe8aMGa51AwYMcNrFcWnbhWRkZJRofAAAAJRNzPwAAAAAsALFDwAAAAArcNkbilXVqlVdj59//nmnPeisX4sKDg5c3e3z+QL23AAAAAgcZn4AAAAAWIHiBwAAAIAVKH4AAAAAWIF7flAksbGxTvu5555z2oPP+e/sA3lvz/lwzw8AAICdyt43UwAAAAAoARQ/AAAAAKzAZW84r5iYGKc9ffp017qhQ4c67bJ4aduFHDt2LNApAAAAIADK17dWAAAAACgiih8AAAAAVuCyN8tFR0c77WnTprnWDRs2zGmHhISUWk4lLTU1NdApoJS8traFIiJiCu54Hg+fftZ7Etdd5z3GvihPm991eJX3HAYM8B7jzTe9x1i0yHuMTtd7DnHlld7TGD/eewyvwle871//jIwSygQV3s6d3mPMmeM9RlaW9xinT3uP8frr3mPs3u09RpMm3mOUM8z8AAAAALACxQ8AAAAAK1D8AAAAALAC9/xYICrKfb/AlClTnPbw4cOddkW6r+dCUlJSAp0CAAAAAoCZHwAAAABWoPgBAAAAYAUue6tAqlSp4rSfeeYZp/3AAw+4+tlyedv5HD16NNApAAAAIACY+QEAAABgBYofAAAAAFag+AEAAABgBe75KWfOvq9n8uTJrnUjR4502qGhDO35cM8PAACAnZj5AQAAAGAFih8AAAAAVuDaqDIoMjLSaU+cONG17qGHHnLaXNpWNCkpKYFOAQAAAAHAzA8AAAAAK1D8AAAAALAC100FSOXKlV2PJ0yY4LQffvhhp12pUqXSSskaqampgU4BpeSBPgcVEx1d5O1/jhrlOQefz3MI1T+8yVuARo28JxFcDP9W1rGj9xibN3sOEX7skOcYd/y4yHOMbI0suFMBpk/3tv3jj9/mV//s4nhDlycJCdI5n9d+OeT9vVZhHD/uPcbw4d5jPPaY9xi//733GL/7nfcYBw96j2EhZn4AAAAAWIHiBwAAAIAVKH4AAAAAWIF7fkpYRESE03766aed9mPnXHPKvT2l5+effw50CgAAAAgAZn4AAAAAWIHiBwAAAIAVuOytGJx9advYsWNd6x5//HGnHRYWVmo54fx8tv1UKwAAACQx8wMAAADAEhQ/AAAAAKzAZW+FFB4e7rSffPJJ17onnnjCaXNpW9l34sSJQKcAAACAAGDmBwAAAIAVKH4AAAAAWIHiBwAAAIAVuOfnLGff1yNJo0ePdtpn3+dzbj+UL5mZmYFOAQAAAAHAzA8AAAAAK1D8AAAAALCClZe9nf1z1I8//rjTHjt2rKtfREREqeWE0pOVlRXoFFBadu+WqlQp8ubTV9f1nMLrr3sOIZ8v0dP2xfGWD3/zFe9BiiORatW8x2jUyHuM++7zHqMYnPURViTbt/vXPz3d2/OVO4cPS16+CxTHe/7gQe8xdu3ytn3Llt5zMMZ7jBtu8B5j927vMf72N+8xiuMcUqeO9xgWYuYHAAAAgBUofgAAAABYgeIHAAAAgBWsvOcnNPS3l3311Vc7be7xqZhycnJcj0+fPh2gTAAAABBIzPwAAAAAsALFDwAAAAArWHnZW0ZGhtPu06eP077hnJ9QXLx4sdOOjo4u+cRQIvhpawAAAEjM/AAAAACwBMUPAAAAACtYednb+Xz00Ueux/Hx8U57zpw5TjspKam0UkIx4LI3AAAASMz8AAAAALAExQ8AAAAAK1D8AAAAALAC9/xcQHZ2ttPu27ev037ttddc/ZYtW+a0q1WrVvKJwS/p6emBTgEAAABlADM/AAAAAKxA8QMAAADAClz2VgTr1693Pa5Zs6bTfuONN1zrBg8eXCo54fx8Pl+gU0CAPL6iq8LDY4q8/Uu/e8dzDp0X9C24UwkL/9sszzE+/91IzzGuvv9qzzGy13/uOYamvOA5RPiEMZ5jbN/uOYRatfK2/YED/vXPyPD2fNaJiPAeo0mTwMfYu9d7Dt9+6z3GmjXeY9Sr5z1GcYzJ6tXeY6BImPkBAAAAYAWKHwAAAABW4LK3YnDq1CmnPWTIENe6mTNnOu3ly5c77Ro1apR8YpAkpaWlBToFAAAAlAHM/AAAAACwAsUPAAAAACtQ/AAAAACwAvf8lLAvvvjCadeuXdtpz5gxw9VvxIgRpZaTbY4dOxboFAAAAFAGMPMDAAAAwAoUPwAAAACswGVvpej06dNOe+RI9/+W/te//tVpr1y50mnXqVOn5BOr4FJSUgKdAgAAAMoAZn4AAAAAWIHiBwAAAIAVKH4AAAAAWIF7fsqI7du3O+369es77SlTprj6jRo1ymkHBQWVfGIVwNGjRwOdAgAAAMoAZn4AAAAAWIHiBwAAAIAVuOytDMrJyXHao0ePdq176623nPaHH37oWtewYcOSTayc4rI3e02eLMXEeAiwq7XnHK5r6jmEd7NPeg7xv//rPY026z/3HCO8d3fviVxyifcYHTp4DnGlNnuOka0rPW2/cKF//U+d8vR0KK+K4/tFccQ453tPkfz0k/cYzzzjPcahQ95jbNzoPYaFmPkBAAAAYAWKHwAAAABW4LK3cmb37t1Ou3Hjxq5148aNc9rjx4932sHBdte4XPYGAAAAiZkfAAAAAJag+AEAAABgBYofAAAAAFbgnp9yzBjjejxp0iSnPXfuXKf9j3/8w9WvefPmJZtYGZOamhroFAAAAFAGMPMDAAAAwAoUPwAAAACswGVvFdT+/fud9qWXXupa98gjjzjtadOmOe2QkJCSTywAuOwNAAAAEjM/AAAAACxB8QMAAADAClz2ZoFzfxXu+eefd9rz58932itXrnT1u+yyy0o2sVLi8/kCnQIAAADKAGZ+AAAAAFiB4gcAAACAFSh+AAAAAFiBe34sd+jQIafdtm1b17rhw4c77Zdfftm1LjS0/Lx1uOfHXgcOSFFRRd9+565WnnO4LXyX5xhBzZp62j4ra3jBnQrQ40vPIRR+wPu+0L593mN8+KH3GN995zlE9tKVBXcqwMMPe9s+Odm//j6ftGiRt+cEiuzGGwOdgSTpr6sbe45x8KD3GJNjNnqOYSNmfgAAAABYgeIHAAAAgBXKz7VLKHUzZ8502gsXLnStW758udNu165dqeVUFOnp6YFOAQAAAGUAMz8AAAAArEDxAwAAAMAKFD8AAAAArMA9PyiUlJQU1+PExESnPXDgQKf95ptvuvqFhYWVbGKFkJ2dHegUAAAAUAYw8wMAAADAChQ/AAAAAKzAZW/wbO7cuU77gw8+cK1bunSp0+7SpUtppeSSlZUVkOcFAABA2cLMDwAAAAArUPwAAAAAsAKXvaFYpaWluR537drVad95551O++2333b1i4iIKNY8Tp486bSNMcUaGwAAAOUTMz8AAAAArEDxAwAAAMAKFD8AAAAArMA9Pyg1CxcudNorV650rVu0aJHT7t69u+fnys7O9hwD5V/TI+sVk16lyNs3TwjznEN2vfaeY0yf7m378H9+5DmHKzrf4DmG3ljhPUbv3t5jDB7sPcakSZ5DhP/vYs8xEhPv8LT9R36+NTIyPD0dEHht23oOcfOl3tOoW9d7jMkePxskSSkp3raPiyuGJEoXMz8AAAAArEDxAwAAAMAKXPaGgEhPT3c97tGjh9O++eabXesWLFjgtKOiogoVP4NrMwAAAHAOZn4AAAAAWIHiBwAAAIAVuOwNZc7y5ctdj2vUqOG0582b57Rvu+2288bw+XzFnxgAAADKNWZ+AAAAAFiB4gcAAACAFSh+AAAAAFiBe35Q5mVmZjrt22+/3Wlfe+21rn7vv/++0z5x4kTJJwYAAIByhZkfAAAAAFag+AEAAABgBS57Q7m1du1a1+OaNWs67cGDB5d2OgAAACjjmPkBAAAAYAWKHwAAAABWoPgBAAAAYAXu+UGFkZ2d7bRnzpwZwExQVtw8pZNCQ2OKvH2nTt5zmBy1zXOMP/2ptaftP//6Bs85tPEcQVJM0cfC8Yc/eI/h83mP0a+f5xCtx9ziOcYXX3jb/oUX/OufleXt+YCAO3zYc4g6W5/1HMNM9xyieMTFBTqDUsfMDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArhAY6AQAoKcsHvKuYypWLvP2Wlv0957DldGvPMS4/vN/T9ldXPek5h3eWNPUc4/a77/UcIzw91XOM7AbNvefx53GeY7RqdYvnGN9/7237Xbv863/S+1upfKlWTfJwDtGxY8WXC4rH4cOBzgABxswPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwQmigEwCA4maMkST5MjM9xUlP9xVHOp75Kp3wFuDUKc85ZGR43xe+Ytid4b943BeSsnMqec8jO9tzjFOnvO+Q9HRv25886V//3Jxzj7GKqrjOIcrKKoZsgIrF9/+Pi0CdR4JMRT+DAbDOwYMHVbdu3UCnAVRY//nPf1SnTp1Ap1FiOIcAJS9Q5xGKHwAVTk5Ojn788UdFR0crKCgo0OkAFYYxRidOnFCtWrUUHFxxr5znHAKUnECfRyh+AAAAAFih4v6zDQAAAACcheIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYgeIHAAAAgBUofgAAAABYITSQTx4U9Fs7ODjw7bKSB+3Ct8tKHrSL1i4redAuWrs4YwXJ/PYgJ4d2SbXLSh60i9YuK3nQLny7rORRXtrmrM+CEsLMDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArUPwAAAAAsEKQMcYE4omzs7M1ZcoUjRkzRuHh4YFIwXqMQeAxBoHHGAQeYxB4jEFgsf8DjzEIvNIag4AVPz6fT7GxsTp+/LhiYmICkYL1GIPAYwwCjzEIPMYg8BiDwGL/Bx5jEHilNQZc9gYAAADAChQ/AAAAAKxA8QMAAADACgErfsLDwzV+/HhuKgsgxiDwGIPAYwwCjzEIPMYgsNj/gccYBF5pjUHAfvAAAAAAAEoTl70BAAAAsALFDwAAAAArUPwAAAAAsALFDwAAAAArlFjx85e//EUdOnRQZGSkqlatWqhtjDF6+umndfHFF6ty5crq1q2bdu3a5epz7Ngx9evXTzExMapataqGDBmi9PT0EngF5Z+/+2rfvn0KCgrK92/hwoVOv/zWL1iwoDReUrlTlPdr165d8+zf++67z9XnwIEDuvnmmxUZGakaNWpo1KhROn36dEm+lHLL3zE4duyYRo4cqebNm6ty5cqqV6+e/vSnP+n48eOufhwH5/faa6+pQYMGioiIUGJioj777LML9l+4cKEuueQSRUREqHXr1lqxYoVrfWE+G+Dmzxi89dZbuuaaa1StWjVVq1ZN3bp1y9N/0KBBed7vPXr0KOmXUa75MwZz5szJs38jIiJcfTgO/OfPGOT32RsUFKSbb77Z6cNxUHj//Oc/1atXL9WqVUtBQUFaunRpgdusW7dOV1xxhcLDw9WkSRPNmTMnTx9/P1/yZUrI008/bV544QXzyCOPmNjY2EJtM3XqVBMbG2uWLl1qvvrqK9O7d2/TsGFDk5mZ6fTp0aOHueyyy8y//vUv88knn5gmTZqYvn37ltCrKN/83VenT582hw4dcv1NnDjRREVFmRMnTjj9JJnk5GRXv7PHCL8pyvu1S5cuZujQoa79e/z4cWf96dOnTatWrUy3bt3Mli1bzIoVK0xcXJwZM2ZMSb+ccsnfMdi2bZu5/fbbzbJly8zu3bvNxx9/bJo2bWruuOMOVz+Og/wtWLDAhIWFmdmzZ5sdO3aYoUOHmqpVq5ojR47k23/Dhg0mJCTETJ8+3ezcudM89dRTplKlSmbbtm1On8J8NuA3/o7B3XffbV577TWzZcsW880335hBgwaZ2NhYc/DgQafPwIEDTY8ePVzv92PHjpXWSyp3/B2D5ORkExMT49q/hw8fdvXhOPCPv2OQmprq2v/bt283ISEhJjk52enDcVB4K1asMGPHjjVLliwxksz7779/wf7ff/+9iYyMNI888ojZuXOneeWVV0xISIhZtWqV08ffMT2fEit+ciUnJxeq+MnJyTEJCQnm2WefdZalpaWZ8PBw88477xhjjNm5c6eRZD7//HOnz8qVK01QUJD54Ycfij338qy49lXbtm3N4MGDXcsK8yZG0cegS5cu5sEHHzzv+hUrVpjg4GDXB+PMmTNNTEyMyc7OLpbcK4riOg7ee+89ExYWZk6dOuUs4zjIX7t27cwDDzzgPD5z5oypVauWmTJlSr79f//735ubb77ZtSwxMdH88Y9/NMYU7rMBbv6OwblOnz5toqOjzdy5c51lAwcONH369CnuVCssf8egoO9KHAf+83oczJgxw0RHR5v09HRnGcdB0RTm8/Lxxx83LVu2dC276667TPfu3Z3HXsc0V5m552fv3r06fPiwunXr5iyLjY1VYmKiNm7cKEnauHGjqlatqquuusrp061bNwUHB2vTpk2lnnNZVhz7avPmzdq6dauGDBmSZ90DDzyguLg4tWvXTrNnz5bhv4vKw8sYzJs3T3FxcWrVqpXGjBmjjIwMV9zWrVurZs2azrLu3bvL5/Npx44dxf9CyrHiOmccP35cMTExCg0NdS3nOHA7efKkNm/e7DqPBwcHq1u3bs55/FwbN2509Zd+fT/n9i/MZwN+U5QxOFdGRoZOnTql6tWru5avW7dONWrUUPPmzTV8+HClpqYWa+4VRVHHID09XfXr11fdunXVp08f1/mc48A/xXEczJo1S0lJSapSpYprOcdBySjos6A4xjRXaMFdSsfhw4clyfWFLvdx7rrDhw+rRo0arvWhoaGqXr260we/Ko59NWvWLF166aXq0KGDa/mkSZN03XXXKTIyUh9++KHuv/9+paen609/+lOx5V8RFHUM7r77btWvX1+1atXS119/rdGjR+vf//63lixZ4sTN7zjJXYffFMdxkJKSosmTJ2vYsGGu5RwHeaWkpOjMmTP5vj+//fbbfLc53/v57PN+7rLz9cFvijIG5xo9erRq1arl+pLRo0cP3X777WrYsKH27NmjJ598Uj179tTGjRsVEhJSrK+hvCvKGDRv3lyzZ89WmzZtdPz4cT333HPq0KGDduzYoTp16nAc+MnrcfDZZ59p+/btmjVrlms5x0HJOd9ngc/nU2Zmpn7++WfP57ZcfhU/TzzxhKZNm3bBPt98840uueQSv5JA4RV2DLzKzMzU/PnzNW7cuDzrzl52+eWX65dfftGzzz5rzZe+kh6Ds79kt27dWhdffLGuv/567dmzR40bNy5y3IqktI4Dn8+nm2++WS1atNCECRNc62w/DlAxTZ06VQsWLNC6detcN9wnJSU57datW6tNmzZq3Lix1q1bp+uvvz4QqVYo7du3V/v27Z3HHTp00KWXXqq//vWvmjx5cgAzs9OsWbPUunVrtWvXzrWc46Bi8Kv4efTRRzVo0KAL9mnUqFGREklISJAkHTlyRBdffLGz/MiRI2rbtq3T56effnJtd/r0aR07dszZvqIr7Bh43VeLFi1SRkaGBgwYUGDfxMRETZ48WdnZ2QoPDy+wf3lXWmOQKzExUZK0e/duNW7cWAkJCXl+3eTIkSOSxHFwFq9jcOLECfXo0UPR0dF6//33ValSpQv2t+04yE9cXJxCQkKc92OuI0eOnHd/JyQkXLB/YT4b8JuijEGu5557TlOnTtXq1avVpk2bC/Zt1KiR4uLitHv3br70ncPLGOSqVKmSLr/8cu3evVsSx4G/vIzBL7/8ogULFmjSpEkFPg/HQfE532dBTEyMKleurJCQEM/HVS6/7vmJj4/XJZdccsG/sLAwvxLI1bBhQyUkJOjjjz92lvl8Pm3atMn515D27dsrLS1NmzdvdvqsWbNGOTk5zhfEiq6wY+B1X82aNUu9e/dWfHx8gX23bt2qatWqWfOFr7TGINfWrVslyfnAa9++vbZt2+b6Uv/RRx8pJiZGLVq0KJ4XWcaV9Bj4fD7deOONCgsL07Jly/L85Gx+bDsO8hMWFqYrr7zSdR7PycnRxx9/7PpX7bO1b9/e1V/69f2c278wnw34TVHGQJKmT5+uyZMna9WqVa575M7n4MGDSk1NdX0Rx6+KOgZnO3PmjLZt2+bsX44D/3gZg4ULFyo7O1v9+/cv8Hk4DopPQZ8FxXFcOfz6eQQ/7N+/32zZssX5qeQtW7aYLVu2uH4yuXnz5mbJkiXO46lTp5qqVauaDz74wHz99demT58++f7U9eWXX242bdpk1q9fb5o2bcpPXZ9HQfvq4MGDpnnz5mbTpk2u7Xbt2mWCgoLMypUr88RctmyZeeutt8y2bdvMrl27zOuvv24iIyPN008/XeKvpzzydwx2795tJk2aZL744guzd+9e88EHH5hGjRqZzp07O9vk/tT1jTfeaLZu3WpWrVpl4uPj+anr8/B3DI4fP24SExNN69atze7du10/aXr69GljDMfBhSxYsMCEh4ebOXPmmJ07d5phw4aZqlWrOr9OeM8995gnnnjC6b9hwwYTGhpqnnvuOfPNN9+Y8ePH5/tT1wV9NuA3/o7B1KlTTVhYmFm0aJHr/Z77eX3ixAnz2GOPmY0bN5q9e/ea1atXmyuuuMI0bdrUZGVlBeQ1lnX+jsHEiRPNP/7xD7Nnzx6zefNmk5SUZCIiIsyOHTucPhwH/vF3DHJ16tTJ3HXXXXmWcxz458SJE853f0nmhRdeMFu2bDH79+83xhjzxBNPmHvuucfpn/tT16NGjTLffPONee211/L9qesLjWlhlVjxM3DgQCMpz9/atWt/e/L///9k5MrJyTHjxo0zNWvWNOHh4eb66683//73v11xU1NTTd++fU1UVJSJiYkx9957r6ugwm8K2ld79+7NMybGGDNmzBhTt25dc+bMmTwxV65cadq2bWuioqJMlSpVzGWXXWbeeOONfPvC/zE4cOCA6dy5s6levboJDw83TZo0MaNGjXL9Pz/GGLNv3z7Ts2dPU7lyZRMXF2ceffRR188w4zf+jsHatWvzPXdJMnv37jXGcBwU5JVXXjH16tUzYWFhpl27duZf//qXs65Lly5m4MCBrv7vvfeeadasmQkLCzMtW7Y0y5cvd60vzGcD3PwZg/r16+f7fh8/frwxxpiMjAxz4403mvj4eFOpUiVTv359M3ToUL+/cNjGnzF46KGHnL41a9Y0N910k/nyyy9d8TgO/Ofvuejbb781ksyHH36YJxbHgX/O91mau88HDhxounTpkmebtm3bmrCwMNOoUSNXjZDrQmNaWEHGWP7brAAAAACsUGb+nx8AAAAAKEkUPwAAAACsQPEDAAAAwAoUPwAAAACsQPEDAAAAwAoUPwAAAACsQPEDAAAAwAoUPwCACmXChAlq27ZtoNMAAJRBFD8AYImjR49q+PDhqlevnsLDw5WQkKDu3btrw4YNTp8GDRroxRdfzLPt+QqKgwcPKiwsTK1atcr3OYOCgpy/2NhYdezYUWvWrCmulwQAgF8ofgDAEnfccYe2bNmiuXPn6rvvvtOyZcvUtWtXpaamFjnmnDlz9Pvf/14+n0+bNm3Kt09ycrIOHTqkDRs2KC4uTrfccou+//77Ij8nAABFRfEDABZIS0vTJ598omnTpunaa69V/fr11a5dO40ZM0a9e/cuUkxjjJKTk3XPPffo7rvv1qxZs/LtV7VqVSUkJKhVq1aaOXOmMjMz9dFHH+Xp5/P5VLlyZa1cudK1/P3331d0dLQyMjIkSaNHj1azZs0UGRmpRo0aady4cTp16tR58+zataseeugh17Jbb71VgwYNch5nZ2frscceU+3atVWlShUlJiZq3bp1hdsRAIByg+IHACwQFRWlqKgoLV26VNnZ2cUSc+3atcrIyFC3bt3Uv39/LViwQL/88ssFt6lcubIk6eTJk3nWxcTE6JZbbtH8+fNdy+fNm6dbb71VkZGRkqTo6GjNmTNHO3fu1EsvvaS33npLM2bM8PRaRowYoY0bN2rBggX6+uuvdeedd6pHjx7atWuXp7gAgLKF4gcALBAaGqo5c+Zo7ty5qlq1qjp27Kgnn3xSX3/9dZ6+o0ePdoql3L9nnnkmT79Zs2YpKSlJISEhatWqlRo1aqSFCxeeN4eMjAw99dRTCgkJUZcuXfLt069fPy1dutSZ5fH5fFq+fLn69evn9HnqqafUoUMHNWjQQL169dJjjz2m9957z99d4jhw4ICSk5O1cOFCXXPNNWrcuLEee+wxderUScnJyUWOCwAoeyh+AMASd9xxh3788UctW7ZMPXr00Lp163TFFVdozpw5rn6jRo3S1q1bXX/33Xefq09aWpqWLFmi/v37O8v69++f76Vvffv2VVRUlKKjo7V48WLNmjVLbdq0yTfHm266SZUqVdKyZcskSYsXL1ZMTIy6devm9Hn33XfVsWNHJSQkKCoqSk899ZQOHDhQ1N2ibdu26cyZM2rWrJmr4Pu///s/7dmzp8hxAQBlT2igEwAAlJ6IiAjdcMMNuuGGGzRu3Dj94Q9/0Pjx4133v8TFxalJkyau7apXr+56PH/+fGVlZSkxMdFZZoxRTk6OvvvuOzVr1sxZPmPGDHXr1k2xsbGKj4+/YH5hYWH6r//6L82fP19JSUmaP3++7rrrLoWG/vpxtXHjRvXr108TJ05U9+7dFRsbqwULFuj5558/b8zg4GAZY1zLzr5HKD09XSEhIdq8ebNCQkJc/aKioi6YLwCgfGHmBwAs1qJFiwLv08nPrFmz9Oijj7pmh7766itdc801mj17tqtvQkKCmjRpUmDhk6tfv35atWqVduzYoTVr1rguefv0009Vv359jR07VldddZWaNm2q/fv3XzBefHy8Dh065Dw+c+aMtm/f7jy+/PLLdebMGf30009q0qSJ6y8hIaFQOQMAygeKHwCwQGpqqq677jr97W9/09dff629e/dq4cKFmj59uvr06eNXrK1bt+rLL7/UH/7wB7Vq1cr117dvX82dO1enT58ucq6dO3dWQkKC+vXrp4YNG7pml5o2baoDBw5owYIF2rNnj15++WW9//77F4x33XXXafny5Vq+fLm+/fZbDR8+XGlpac76Zs2aqV+/fhowYICWLFmivXv36rPPPtOUKVO0fPnyIr8OAEDZQ/EDABaIiopSYmKiZsyYoc6dO6tVq1YaN26chg4dqldffdWvWLNmzVKLFi10ySWX5Fl322236aefftKKFSuKnGtQUJD69u2rr776yjXrI0m9e/fWww8/rBEjRqht27b69NNPNW7cuAvGGzx4sAYOHKgBAwaoS5cuatSoka699lpXn+TkZA0YMECPPvqomjdvrltvvVWff/656tWrV+TXAQAoe4LMuRdCAwAAAEAFxMwPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwAsUPAAAAACtQ/AAAAACwwv8DV0tSlhfOUz8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x400 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "top_preds = np.argsort(-predictions)\n",
    "inds = top_preds[0]\n",
    "\n",
    "# Visualize the explanations\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10,4))\n",
    "axes[0].imshow(test_sample[0], cmap='gray')\n",
    "axes[0].axis('off')\n",
    "\n",
    "# plot the test image and the attributions on the image for each class, first triangle\n",
    "for i in range(2):\n",
    "    m = normalize(shap_values[inds[i]])  # normalize to [-1, 1]\n",
    "    axes[i+1].set_title(class_name[inds[i]])\n",
    "    im = axes[i+1].imshow(m, cmap='bwr')\n",
    "    axes[i+1].set_xticks([])\n",
    "    axes[i+1].set_yticks([])\n",
    "cb = fig.colorbar(im, ax=axes.ravel().tolist(), label=\"SHAP value\", orientation=\"horizontal\", aspect=60)\n",
    "cb.outline.set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### 3 - Conclusions\n",
    "The Shapley scores are estimated using KernelSHAP. The example here shows that the KernelSHAP method evaluates the importance of each segmentations/super pixels to the classification of geometric shapes and the results indicate that the model determines the shape by checking whether there is a (sharp) angle or not. For instance, the figure above shows that the sharp angle leads to negative scores against circle, and therefore the prediction is triangle. The interpretation agrees with the human visual preception of the chosen image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e7604e8ec5f09e490e10161e37a4725039efd3ab703d81b1b8a1e00d6741866c"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}