experiments/multiclass/notebooks/Random_Data_Split.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TC Intensity Classification using Random data split\n",
    "In this notebook we proceed to perform the same experiments as in [Classification_Model](Classification_Model.ipynb) but with a different data split. That is, we proceed to use completely randomised datasets. That is, the training, validation and test datasets have been generated at image level. Meaning that images from same sequences may be found in different sets. \n",
    "\n",
    "Note that this clearly may influence the evaluation process, as now test data contains images from typhoon sequences which have been partially (some images) used to traing the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "sys.path.insert(0, '../../..')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use GPU with identifier \"1\", after agreeing with other users with access to the same machine. If you are not using GPU or don't have any restriction on GPU usage skip this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import environ\n",
    "\n",
    "environ[\"CUDA_VISIBLE_DEVICES\"]=\"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load data\n",
    "Time to load the data. The data is stored in chunk HDF5 files, which have been previously been generated using script [generate_dataset.py](../../../scripts/generate_dataset.py). We will use chunks 0-26 for training, 27-24 for validation and 35-47 for testing. Note that loading the chunks might take a while. Grab yourself a tee."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "\n",
    "# Paths where data is stored\n",
    "dataset_dir = '/root/fs9/lucas/data/datasets/random_256/'\n",
    "chunk_filenames = listdir(dataset_dir)\n",
    "train_chunk_filenames = chunk_filenames[:27]\n",
    "valid_chunk_filenames = chunk_filenames[27:35]\n",
    "test_chunk_filenames = chunk_filenames[35:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this task, we will ignore classes 6 and 7 and only focus on Tropical Cyclones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " file 0_chunk.h5 read\n",
      " file 1_chunk.h5 read\n",
      " file 2_chunk.h5 read\n",
      " file 3_chunk.h5 read\n",
      " file 4_chunk.h5 read\n",
      " file 5_chunk.h5 read\n",
      " file 6_chunk.h5 read\n",
      " file 7_chunk.h5 read\n",
      " file 8_chunk.h5 read\n",
      " file 9_chunk.h5 read\n",
      " file 10_chunk.h5 read\n",
      " file 11_chunk.h5 read\n",
      " file 12_chunk.h5 read\n",
      " file 13_chunk.h5 read\n",
      " file 14_chunk.h5 read\n",
      " file 15_chunk.h5 read\n",
      " file 16_chunk.h5 read\n",
      " file 17_chunk.h5 read\n",
      " file 18_chunk.h5 read\n",
      " file 19_chunk.h5 read\n",
      " file 20_chunk.h5 read\n",
      " file 21_chunk.h5 read\n",
      " file 22_chunk.h5 read\n",
      " file 23_chunk.h5 read\n",
      " file 24_chunk.h5 read\n",
      " file 25_chunk.h5 read\n",
      " file 26_chunk.h5 read\n",
      " file 27_chunk.h5 read\n",
      " file 28_chunk.h5 read\n",
      " file 29_chunk.h5 read\n",
      " file 30_chunk.h5 read\n",
      " file 31_chunk.h5 read\n",
      " file 32_chunk.h5 read\n",
      " file 33_chunk.h5 read\n",
      " file 34_chunk.h5 read\n",
      " file 35_chunk.h5 read\n",
      " file 36_chunk.h5 read\n",
      " file 37_chunk.h5 read\n",
      " file 38_chunk.h5 read\n",
      " file 39_chunk.h5 read\n",
      " file 40_chunk.h5 read\n",
      " file 41_chunk.h5 read\n",
      " file 42_chunk.h5 read\n",
      " file 43_chunk.h5 read\n",
      " file 44_chunk.h5 read\n",
      " file 45_chunk.h5 read\n",
      " file 46_chunk.h5 read\n",
      " file 47_chunk.h5 read\n"
     ]
    }
   ],
   "source": [
    "from pyphoon.app.utils import load_h5datachunks\n",
    "\n",
    "X_train, Y_train, idx_train, seqno_train = load_h5datachunks(dataset_dir,\n",
    "                                                             train_chunk_filenames, \n",
    "                                                             features=['data', 'class', 'idx', 'seq_no'],\n",
    "                                                             ignore_classes=[6, 7],\n",
    "                                                             verbose=True\n",
    "                                                             )\n",
    "\n",
    "X_valid, Y_valid, idx_valid, seqno_valid = load_h5datachunks(dataset_dir,\n",
    "                                                             valid_chunk_filenames, \n",
    "                                                             features=['data', 'class', 'idx', 'seq_no'],\n",
    "                                                             ignore_classes=[6, 7],\n",
    "                                                             verbose=True\n",
    "                                                            )\n",
    "\n",
    "X_test, Y_test, idx_test, seqno_test = load_h5datachunks(dataset_dir, \n",
    "                                                         test_chunk_filenames, \n",
    "                                                         features=['data', 'class', 'idx', 'seq_no'],\n",
    "                                                         ignore_classes=[6, 7],\n",
    "                                                         verbose=True\n",
    "                                                         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Balance dataset\n",
    "The distribution of the classes is not balanced. Hence, we proceed to balance it which gives our model equal chances to perform good on all classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test...\n",
      "valid...\n",
      "train...\n"
     ]
    }
   ],
   "source": [
    "from pyphoon.app.utils import balance_dataset\n",
    "\n",
    "# test\n",
    "print(\"test...\")\n",
    "for i in range(len(Y_test)):\n",
    "    # Get labels and valid positions\n",
    "    pos = balance_dataset(Y_test[i], [2,3,4,5])\n",
    "    # Retrieve corresponding samples\n",
    "    X_test[i] = X_test[i][pos]\n",
    "    Y_test[i] = Y_test[i][pos]\n",
    "    idx_test[i] = idx_test[i][pos]\n",
    "    seqno_test[i] = seqno_test[i][pos]\n",
    "\n",
    "# valid\n",
    "print(\"valid...\")\n",
    "for i in range(len(Y_valid)):\n",
    "    # Get labels and valid positions\n",
    "    pos = balance_dataset(Y_valid[i], [2,3,4,5])\n",
    "    # Retrieve corresponding samples\n",
    "    X_valid[i] = X_valid[i][pos]\n",
    "    Y_valid[i] = Y_valid[i][pos]\n",
    "    idx_valid[i] = idx_valid[i][pos]\n",
    "    seqno_valid[i] = seqno_valid[i][pos]\n",
    "    \n",
    "# train\n",
    "print(\"train...\")\n",
    "for i in range(len(Y_train)):\n",
    "    # Get labels and valid positions\n",
    "    pos = balance_dataset(Y_train[i], [2,3,4,5])\n",
    "    # Retrieve corresponding samples\n",
    "    X_train[i] = X_train[i][pos]\n",
    "    Y_train[i] = Y_train[i][pos]\n",
    "    idx_train[i] = idx_train[i][pos]\n",
    "    seqno_train[i] = seqno_train[i][pos]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us plot the histograms of the sets to verify that the data is equally balanced now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAFTCAYAAACNs2XWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xu8X1V95//XuwTQeuEiKaUJNkxl6qC/qjSDtDr+LLRc1BrmN0ixtkSKTS+0tVNnWnBmRFE6OtMWtR11qDAGiwI/tCVVFFMu9eev5RJBkYsOKaIk5RINIErFBj/zx14Hvhy+Jzk5l+/5np3X8/E4j+/ea6+992fvJCvn891rrZ2qQpIkSZLUTz+w0AFIkiRJkuaPSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJPVckhVJKsmHFjqWUUlydZIFeydRkg+1e75ioGws/hwW+t5IkkbPpE+SFqEkz03yp0luTvJgku8l+cckn0xySpI9FzrG2WoJ0uDPI0m2JLkhyQeTHJtkt3k6951J7pyPY8+3YQmnJGnXtmShA5Ak7ZwkbwHOoPvi7u+BtcC3gf2BlwMfBH4DWLlAIc61t7XP3YC9gecBvwycAmxI8rqq+t+T9jkJ+MHRhfgkpwPvBDYvYAxTWeh7I0kaMZM+SVpEkryZLgm6C3hNVV07pM6rgDeNOrb5UlVvnVyWZH/gT4HXAH+TZGVV3Tewz9dHF+GTVdXdwN0LGcNUFvreSJJGz+6dkrRItO56bwX+GXjFsIQPoKo+ARwzjeP9yyTvTLKhdZt8JMnXkpyTZPmQ+kmyOsnftfrfTXJXksuT/MKkuj+R5KOtm+Rgt8x3J9l9Bpc/+RrvBU4ErgYOBN486fxPGrc2nfiTvLzt96PAj07qXvqhgWNVO8cPt66mm5M8muT1bft2u1i27rl/lWRrku8k+VySo4bUe2s7zsuHbHvSGMEW++q2+tWB2O/c3r1p5T+Q5NeTXJ/k2y2u65P8RpIn/b4wcA/2a39n7m5/1rckOXnYdUuSFoZP+iRp8TgZ2B24sKpu3l7FqnpkGsf7f4BfB64C/g74Hl3XyTcAP9+eng12TzyLrtviV4GLgQeBA4B/TffE7SLoEj7gWqCAda3+M4HnAL8J/Ge6xHVWqur7Sd5B16X1tUn+fVVtb4KS6cR/J92T1N9t+7x7YP8vTDrevsA1dF1rPw58H7h3GqEfRNct90vA/2wx/ALwqSS/WFUXTeMYU3kbcBzwAuA9wAOt/IEp93jch4FfpHuK/EG6P79/C7wPeCnwuiH77A38/3R/dy4B9qS7l+cl+X5VrZ3xlUiS5oxJnyQtHi9tn1fM0fE+DJw9OUFsT5w+RZec/cbApl+jG6P2/Kp6eNI++w2srgaeAhxXVZdOqrcP8IR9Z+lzwDbgh4AVdAndVHYYf1XdCbx14ondsK6lA/4vunv4K1W1bSdifhnwR1X1HwfO/2d0ieAHknyqqr61E8d7TFW9tT1dfAHw7nY9O5TktXQJ343Ay6rq2638PwN/C/xikk9W1Ucm7foC4Fzg16rq0bbPu4GbgD+gG28qSVpgdu+UpMXjgPa5aS4OVlWbhz0RrKrPALcARw/Z7Z+BR4fs840hdf9pSL37q+r7Mwh3qBb/N9vq0mnssjPx78j3gP+wkwkfdE8Yz5x0/g3ABXRPzv7tDGKZrV9pn6dNJHwtru/QJW/QPQGe7GHg9yYSvrbPrXRP//5VkqfPU7ySpJ1g0idJu6g2xu2XkvxNG+O2bWIMGN1TrGWTdrmA7mnarUn+a5Jjkuw15NAX0SVWf5Xk/CQnJfmx+byU9rmjd89NN/7punNw8pidcENVPTSk/Or2+aKZhzRjh9J1T716yLa/pfvzHBbX7VM8lbyrfe4zJ9FJkmbFpE+SFo+J2SAnJ2Mz9Sd03RMPAS4H/phuTNjbgK8Be0yq/+/bz7eB0+i6gH4jyaVJnjNRqaquA/4NcCVwPF0Xv41Jvty6Ec6ZJE+hG1sHsGUH1acV/064Zwb7wNTj/iaON5tEdKb2ArZW1fcmb2hPMr/B8LimGis48fRzXt6jKEnaOY7pk6TF43PAEcCRdOOoZizJDwG/A9wM/PTkJ0/DkrPWhe/dwLvb/i+lm0HzNcDzkjxvortoVf098Kp0L4n/SbrZRH8b+EiSLVX1N7OJf8BL6f4vu3dH49d2Jv5p2tGTxansP0X5D7fPBwfKJrrCDvv/eu8Znn+YB4F9k+xeVU+YZCfJEmA/YEbjDCVJC88nfZK0ePwvujFp/y7JIdur2JKt7fkXdP8HfGZIwre8bZ9SVd1XVR+vqhPonuj9GPD8IfUeqaq/q6q30CWZAKt2ENu0tNcI/Ke2OnmCke2aRvyPMn9PqQ5N8owh5S9vnzcOlN3fPg8cUn/lFMefGF+3M/HfSPf34WVDtr2sHeuGnTieJGmMmPRJ0iIxMbMkXbfLTyYZ+kt/kmPoui5uz53t86VJHksO2sQbf86kJ0tJ9kzykiHn2p3Hu1c+3Mp+OslTh5xz/8F6s9Ge1F1Ilyh9HfjDHdSfdvzNN4GlU1zHbO0FvGVSHCvpXonwIPCXA5uua58ntyduE/UPnHyMARMT2zx7J2I6r33+1yQ/OHCeHwTe2VZn9XRZkrRw7N4pSYtIVf1h++X/DOD6JH8HbKAbp7Y/3VOZg1vZ9o5zT5IL6bo3fiHJZ+iSkZ8Dvkv3TroXDuzyVOBzSTYCn6cb8/eUVv9fAeuq6rZW9/eBI5L8f3SvUPg23fv/jqV7cnXOzlxzkre2xR+g69L4PLqumXvQJUWvm8bsmzsTP3SvxfjXwKeTfBZ4BPhiVf31zsQ+hc8Cb0jyYrpZLife0/cDdK8+eKwbZVVd287/MuC6JFfS/Tn/PN04zGFPAK8A/iPw50k+BjwEPFBVfzZVQFX1kSSrgBOAW5L8FV331ePo3it4UVVdMMvrliQtEJM+SVpkqurMJP8v3YvOf4bupe1PoXvC8wXgXcBfTONQpwB30CUcp9JNhLKO7gnSxybVnZi6/2eAn6ZLBh4C/oHuXX7nDdR9H11y92IeH3O3qZX/cVV9bacuuEtwoXtFwkN0Cdv5LcbPTPMVEDsTP8A76BLMnwdeQte9cS0wF0nfV4Ffp3uC9ut0LzS/ATizqi4fUn8V8N/b528Dt9Ml1p+hS9KeoKouT/Im4FfpXjK/B909mzLpa15LN1Pnr9C90xDgNroJft4//cuTJI2bVM10HLokSZIkadw5pk+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT4tOkt2SfDvJzryDSpKeJMmKJDXxDrwkn0qyejp1Z3CuNyf54GzilSRpJkz6NO9agjbx8/0k/zSw/rqdPV5VPVpVT6+qr89HvJIWlySfTnLmkPJVSe7ZmSStqo6tqrVzENPLk2yadOw/rKo3zPbYknYtc/171MBxr0nyS3MZq8aXSZ/mXUvQnl5VTwe+Dvz8QNmTXvY702/RJe2y1gK/lCSTyn8ZuKCqti1ATJI0J3b29yhpGJM+Lbgk70hyUZKPJnmI7pe3n2rfQD2Q5O4k702ye6u/pHWxWtHW/6Jt/1SSh5L8fZKDFvCSJI3WXwHPAv7NREGSfYBXAecneWWSG5N8K8ldSd461YGSXJ3kDW15tyR/lOQbSe4AXjmp7slJbmvtzh1Jfq2VPw34FPAjA9/G/0iStyb5i4H9X53kltbOXZ3kXw1suzPJf0hyU5IHWxv5lLm4WZL6pbVV/6W1Q99IckGSvdu2pyW5MMnW1tZcm2SfJH8M/Gvgg62N+uOFvQrNN5M+jYt/C3wE2Au4CNgGvBHYD3gJcAzwa9vZ/xeB/wLsS/ct2NvnM1hJ46Oq/gm4GDhpoPgE4MtV9UXgO23b3nSJ228kOW4ah/5VusTxRcBK4PhJ2+9r258JnAycneTQqvoOcCzwjwPfxv/j4I5J/iXwUeB3gaXAZcBfJ9lj0jUcAxwE/ATw+mnELGnX8x+Ao4CXAsuBfwbObtveACwBltH9TvVbwPeq6k3A9cAbWhv1ppFHrZEy6dO4+FxV/XVVfb+q/qmqrq+qa6tqW1XdAZwD/N/b2f+SqtpQVf8MXAC8cCRRSxoXa4HjB56GndTKqKqrq+pLrX25iS7Z2l57MuEE4N1VdVdVbQX+6+DGqvpkVf1Ddf4W+AwDTxt34BeAT1bV+tZu/RHwVOCnB+q8t6r+sZ37r7FdkzTcrwOntfbiu8DbgF9oXd7/me6LpR9rv1Nd376Y0i7GpE/j4q7BlSTPTfLJNgnDt4Az6b6hmso9A8sPA0+fhxgljamq+hzwDeC4JD8GHEbXe4AkL05yVZItSR6k+wVpe+3JhB/hiW3T1wY3Jjm2dUPfmuQB4BXTPO7EsR87XlV9v51r2UAd2zVJ29USuwOBy1r3zQeAG+l+x38WcC7wt8AlSTYl+cMkuy1cxFooJn0aFzVp/X8CNwPPqapnAm8BJk/SIEmDzqd7wvdLwOVVdW8r/wiwDjiwqvYCPsD02pO76X6ZmvDYa2KS7Al8jO4J3f5VtTddF82J405u0yb7R+BHB4438Yvb5mnEJUkAVFXRtRtHVNXeAz9PqapvVNUjVfWWqnou8DLgNcCJE7svVNwaPZM+jatnAA8C32mTG2xvPJ8kQZf0/SzdWLzB1y48A9haVd9NchjdGODpuBj4nSTL28Qwpw1s2wPYE9gCbEtyLN2Ymgn3As9Kstd2jv3KJEe2SareBDwC/N00Y5OkCR8A3pnkQIAkP5Tk59vyzyY5JMkPAN+imzPh+22/e4F/sRABa/RM+jSu3gSsBh6ie+p30cKGI2ncVdWddEnT0+ie7E34TeDMNjvwW+gSrun4c+By4IvADcDHB871EPA77Vj30yWS6wa2f5lu7OAdrcvVj0yK9St0TyT/lK5b6s/TTcP+vWnGJkkT/hvwN8CVrZ37O+DQtm0ZcCnd71M30/VImPid6mzgpCT3J/lvow1Zo5buqbAkSZIkqY980idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9tmShA5ip/fbbr1asWLHQYUiaQ5///Oe/UVVLFzqO2bBtkvqnD20T2D5JfTTd9mnRJn0rVqxgw4YNCx2GpDmU5GsLHcNs2TZJ/dOHtglsn6Q+mm77ZPdOSZIkSeoxkz5JkiRJ6jGTPkmSJEnqMZM+SZIkSeoxkz5JkiRJ6jGTPkmSJEnqMZM+SZIkSeoxkz5JkiRJ6jGTPkmSJEnqMZM+SZIkSeoxkz5JkiRJ6rElCx3AKKw47ZPzctw73/nKeTnufPJePG6+7gUsvvvhvVgY3vfHeS+eyLb6cd6LheF9f5z34nHei8cttv+3fNInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPbbDpC/JeUnuS3LzkG1vSlJJ9mvrSfLeJBuT3JTk0IG6q5Pc3n5WD5T/ZJIvtX3emyRzdXGSJEmStKubzpO+DwHHTC5MciBwFPD1geJjgYPbzxrg/a3uvsAZwIuBw4AzkuzT9nk/8KsD+z3pXJIkSZKkmdlh0ldVnwW2Dtl0NvD7QA2UrQLOr841wN5JDgCOBtZX1daquh9YDxzTtj2zqq6pqgLOB46b3SVJkiRJkibMaExfklXA5qr64qRNy4C7BtY3tbLtlW8aUi5JktQ7SfZOckmSLye5LclPJdk3yfo2BGb9RG+omQybkaRhdjrpS/KDwJuBt8x9ODs895okG5Js2LJly6hPL0mSNFvvAT5dVc8FXgDcBpwGXFFVBwNXtHWY2bAZSXqSmTzp+zHgIOCLSe4ElgM3JPlhYDNw4EDd5a1se+XLh5QPVVXnVNXKqlq5dOnSGYQuSZK0MJLsBbwMOBegqr5XVQ/QDY9Z26qt5fGhLjs1bGaElyJpkdnppK+qvlRVP1RVK6pqBV2XzEOr6h5gHXBS645wOPBgVd0NXA4clWSf9k3UUcDlbdu3khzeZu08Cbh0jq5NkiRpnBwEbAH+V5Ibk3wwydOA/dvvRAD3APu35Z0dNiNJQ03nlQ0fBf4e+PEkm5Kcsp3qlwF3ABuBPwd+E6CqtgJvB65vP2e2MlqdD7Z9/gH41MwuRZKeKMlu7RerT7T1g5Jc28bHXJRkj1a+Z1vf2LavGDjG6a38K0mOXpgrkdQTS4BDgfdX1YuA7/B4V04A2sR2NWTfGXFojCToGp/tqqrX7mD7ioHlAk6dot55wHlDyjcAz99RHJI0A2+kGy/zzLb+LuDsqrowyQeAU+jGyJwC3F9Vz0lyYqv3C0kOAU4Engf8CPA3Sf5lVT066guR1AubgE1VdW1bv4Qu6bs3yQFVdXfrvnlf27694TEvn1R+9bATVtU5wDkAK1eunLNkUtLiMqPZOyVp3CVZDrySricBrQv5EXS/ZMGTx81MjKe5BDiy1V8FXFhVj1TVV+l6JBw2miuQ1DdtKMxdSX68FR0J3Eo3PGZiBs7VPD7UZaeGzYzqOiQtPjt80idJi9S76d4l+oy2/izggara1tYHx8A8Nj6mqrYlebDVXwZcM3BMx81Imq3fBi5o3cvvAE6m+xL+4jaE5mvACa3uZcAr6L5werjVpaq2JpkYNgNPHDYjSU9i0iepd5K8Crivqj6f5OUjON8auunUefaznz3fp5O0iFXVF4CVQzYdOaTuTg+bkaRh7N4pqY9eAry6vVbmQrpune+hm+584suuwVfEPDZupm3fC/gmU4+neQJfJyNJksaZSZ+k3qmq06tqeZto6kTgyqp6HXAVcHyrNnnczMR4muNb/WrlJ7bZPQ+ie0HydSO6DEmSpDlh905Ju5I/AC5M8g7gRtoLktvnh5NsBLbSJYpU1S1JLqabaGEbcKozd0qSpMXGpE9Sr1XV1bSpzKvqDobMvllV3wVeM8X+ZwFnzV+EkiRJ88vunZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYztM+pKcl+S+JDcPlP33JF9OclOSv0yy98C205NsTPKVJEcPlB/TyjYmOW2g/KAk17byi5LsMZcXKEmSJEm7suk86fsQcMyksvXA86vqJ4D/DZwOkOQQ4ETgeW2f9yXZLcluwP8AjgUOAV7b6gK8Czi7qp4D3A+cMqsrkiRJkiQ9ZodJX1V9Ftg6qewzVbWtrV4DLG/Lq4ALq+qRqvoqsBE4rP1srKo7qup7wIXAqiQBjgAuafuvBY6b5TVJkiRJkpq5GNP3K8Cn2vIy4K6BbZta2VTlzwIeGEggJ8olSZIkSXNgVklfkv8EbAMumJtwdni+NUk2JNmwZcuWUZxSkiRJkha1GSd9SV4PvAp4XVVVK94MHDhQbXkrm6r8m8DeSZZMKh+qqs6pqpVVtXLp0qUzDV2SJEmSdhkzSvqSHAP8PvDqqnp4YNM64MQkeyY5CDgYuA64Hji4zdS5B91kL+tasngVcHzbfzVw6cwuRZIkSZI02XRe2fBR4O+BH0+yKckpwJ8BzwDWJ/lCkg8AVNUtwMXArcCngVOr6tE2Zu+3gMuB24CLW12APwB+L8lGujF+587pFUqSJEnSLmzJjipU1WuHFE+ZmFXVWcBZQ8ovAy4bUn4H3eyekiRJvZbkTuAh4FFgW1WtTLIvcBGwArgTOKGq7m+znL8HeAXwMPD6qrqhHWc18J/bYd9RVWtHeR2SFpe5mL1TkiRJ0/czVfXCqlrZ1k8Drqiqg4Er2jp07zc+uP2sAd4P0JLEM4AX031xfkaSfUYYv6RFxqRPkiRpYa2ie1cxPPGdxauA86tzDd3kdwcARwPrq2prVd0PrAeOGXXQkhYPkz5JkqTRKeAzST6fZE0r27+q7m7L9wD7t+Wdff+xJA21wzF9kiRJmjMvrarNSX6IbkK8Lw9urKpKUlPsu9NaYrkG4NnPfvZcHVbSIuOTPkmSpBGpqs3t8z7gL+nG5N3bum3SPu9r1Xf2/cfDzuc7jiWZ9EmSJI1CkqclecbEMnAUcDPde45Xt2qD7yxeB5yUzuHAg60b6OXAUUn2aRO4HNXKJGkou3dKkiSNxv7AX3ZvYmAJ8JGq+nSS64GL27uQvwac0OpfRve6ho10r2w4GaCqtiZ5O3B9q3dmVW0d3WVIWmxM+iRJkkagvZv4BUPKvwkcOaS8gFOnONZ5wHlzHaOkfrJ7pyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ+k3knylCTXJflikluSvK2VH5Tk2iQbk1yUZI9Wvmdb39i2rxg41umt/CtJjl6YK5IkSZo5kz5JffQIcERVvQB4IXBMksOBdwFnV9VzgPuBU1r9U4D7W/nZrR5JDgFOBJ4HHAO8L8luI70SSZKkWTLpk9Q71fl2W929/RRwBHBJK18LHNeWV7V12vYjk6SVX1hVj1TVV4GNwGEjuARJkqQ5Y9InqZeS7JbkC8B9wHrgH4AHqmpbq7IJWNaWlwF3AbTtDwLPGiwfso8kSdKisMOkL8l5Se5LcvNA2b5J1ie5vX3u08qT5L1t/MtNSQ4d2Gd1q397ktUD5T+Z5Ettn/e2b9claVaq6tGqeiGwnO7p3HPn61xJ1iTZkGTDli1b5us0kiRJMzKdJ30fohvLMug04IqqOhi4oq0DHAsc3H7WAO+HLkkEzgBeTPfL1xkTiWKr86sD+00+lyTNWFU9AFwF/BSwd5IlbdNyYHNb3gwcCNC27wV8c7B8yD6D5zinqlZW1cqlS5fOy3VIkiTN1A6Tvqr6LLB1UvHg+JfJ42LOb+NprqH7BesA4GhgfVVtrar76bpaHdO2PbOqrqmqAs4fOJYkzUiSpUn2bstPBX4OuI0u+Tu+VVsNXNqW17V12vYrW5u0Djixze55EN0XU9eN5iokSZLmxpIdVxlq/6q6uy3fA+zflqca/7K98k1DyodKsobuCSLPfvazZxi6pF3AAcDaNtPmDwAXV9UnktwKXJjkHcCNwLmt/rnAh5NspPuS60SAqrolycXArcA24NSqenTE1yJJkjQrM036HlNVlaTmIphpnOsc4ByAlStXjuSckhafqroJeNGQ8jsYMvtmVX0XeM0UxzoLOGuuY5QkSRqVmc7eeW/rmkn7vK+VTzX+ZXvly4eUS5IkSZLmwEyTvsHxL5PHxZzUZvE8HHiwdQO9HDgqyT5tApejgMvbtm8lObzN2nnSwLEkSZIkSbO0w+6dST4KvBzYL8kmulk43wlcnOQU4GvACa36ZcAr6F5g/DBwMkBVbU3yduD6Vu/MqpqYHOY36WYIfSrwqfYjSZIkSZoDO0z6quq1U2w6ckjdAk6d4jjnAecNKd8APH9HcUiSJEmSdt5Mu3dKkiRJkhYBkz5JkiRJ6jGTPkmSJEnqMZM+SZIkSeoxkz5JkqQRSrJbkhuTfKKtH5Tk2iQbk1yUZI9Wvmdb39i2rxg4xumt/CtJjl6YK5G0WJj0SZIkjdYbgdsG1t8FnF1VzwHuB05p5acA97fys1s9khwCnAg8DzgGeF+S3UYUu6RFyKRPkiRpRJIsB14JfLCtBzgCuKRVWQsc15ZXtXXa9iNb/VXAhVX1SFV9le79yIeN5gokLUYmfZIkSaPzbuD3ge+39WcBD1TVtra+CVjWlpcBdwG07Q+2+o+VD9lHkp7EpE+SJGkEkrwKuK+qPj/Cc65JsiHJhi1btozqtJLGjEmfJEnSaLwEeHWSO4EL6bp1vgfYO8mSVmc5sLktbwYOBGjb9wK+OVg+ZJ8nqKpzqmplVa1cunTp3F6NpEXDpE+SJGkEqur0qlpeVSvoJmK5sqpeB1wFHN+qrQYubcvr2jpt+5VVVa38xDa750HAwcB1I7oMSYvQkh1XkSRJ0jz6A+DCJO8AbgTObeXnAh9OshHYSpcoUlW3JLkYuBXYBpxaVY+OPmxJi4VJnyRJ0ohV1dXA1W35DobMvllV3wVeM8X+ZwFnzV+EkvrE7p2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjs0r6kvz7JLckuTnJR5M8JclBSa5NsjHJRUn2aHX3bOsb2/YVA8c5vZV/JcnRs7skSZIkSdKEGSd9SZYBvwOsrKrnA7sBJwLvAs6uqucA9wOntF1OAe5v5We3eiQ5pO33POAY4H1JdptpXJIkSZKkx822e+cS4KlJlgA/CNwNHAFc0ravBY5ry6vaOm37kUnSyi+sqkeq6qvARuCwWcYlSZIkSWIWSV9VbQb+CPg6XbL3IPB54IGq2taqbQKWteVlwF1t322t/rMGy4fsI0mSJEmahdl079yH7indQcCPAE+j6545b5KsSbIhyYYtW7bM56kkSZIkqRdm073zZ4GvVtWWqvpn4OPAS4C9W3dPgOXA5ra8GTgQoG3fC/jmYPmQfZ6gqs6pqpVVtXLp0qWzCF2SJEmSdg2zSfq+Dhye5Afb2LwjgVuBq4DjW53VwKVteV1bp22/sqqqlZ/YZvc8CDgYuG4WcUmSJEmSmiU7rjJcVV2b5BLgBmAbcCNwDvBJ4MIk72hl57ZdzgU+nGQjsJVuxk6q6pYkF9MljNuAU6vq0ZnGJUmSJEl63IyTPoCqOgM4Y1LxHQyZfbOqvgu8ZorjnAWcNZtYJEmSJElPNttXNkiSJEmSxphJnyRJkiT1mEmfJEmSJPUcGJRNAAAUvklEQVSYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJ0ggkeUqS65J8McktSd7Wyg9Kcm2SjUkuSrJHK9+zrW9s21cMHOv0Vv6VJEcvzBVJWixM+iRJkkbjEeCIqnoB8ELgmCSHA+8Czq6q5wD3A6e0+qcA97fys1s9khwCnAg8DzgGeF+S3UZ6JZIWFZM+Sb2T5MAkVyW5tX2b/sZWvm+S9Ulub5/7tPIkeW/71vymJIcOHGt1q397ktULdU2SFr/qfLut7t5+CjgCuKSVrwWOa8ur2jpt+5FJ0sovrKpHquqrwEbgsBFcgqRFyqRPUh9tA95UVYcAhwOntm/GTwOuqKqDgSvaOsCxwMHtZw3wfuiSROAM4MV0v1CdMZEoStJMJNktyReA+4D1wD8AD1TVtlZlE7CsLS8D7gJo2x8EnjVYPmQfSXoSkz5JvVNVd1fVDW35IeA2ul+IBr81n/xt+vntW/hrgL2THAAcDayvqq1VdT/dL2jHjPBSJPVMVT1aVS8EltN9mfTc+TxfkjVJNiTZsGXLlvk8laQxZtInqdfaxAcvAq4F9q+qu9ume4D92/JU35r7bbqkeVFVDwBXAT9F90XTkrZpObC5LW8GDgRo2/cCvjlYPmSfyec5p6pWVtXKpUuXzvl1SFocTPok9VaSpwMfA363qr41uK2qim4szVycx2/SJe1QkqVJ9m7LTwV+jq4nwlXA8a3aauDStryurdO2X9narnXAiW12z4PouqZfN5qrkLQYmfRJ6qUku9MlfBdU1cdb8b2t2ybt875WPtW35tP6Nt1v0iVN0wHAVUluAq6n6z7+CeAPgN9LspFuzN65rf65wLNa+e/RxiFX1S3AxcCtwKeBU6vq0ZFeiaRFZcmOq0jS4tJmtzsXuK2q/mRg08S35u/kyd+m/1aSC+kmbXmwqu5OcjnwhwOTtxwFnD6Ka5DUP1V1E11388nldzBk9s2q+i7wmimOdRZw1lzHKKmfTPok9dFLgF8GvtRmyQN4M12yd3GSU4CvASe0bZcBr6Cb9vxh4GSAqtqa5O1038gDnFlVW0dzCZIkSXPDpE9S71TV54BMsfnIIfULOHWKY50HnDd30UmSJI2WY/okSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx2aV9CXZO8klSb6c5LYkP5Vk3yTrk9zePvdpdZPkvUk2JrkpyaEDx1nd6t+eZPVsL0qSJEmS1Jntk773AJ+uqucCLwBuA04Drqiqg4Er2jrAscDB7WcN8H6AJPsCZ9C9EPkw4IyBFyFLkiRJkmZhxklfkr2AlwHnAlTV96rqAWAVsLZVWwsc15ZXAedX5xpg7yQHAEcD66tqa1XdD6wHjplpXJIkSZKkx83mSd9BwBbgfyW5MckHkzwN2L+q7m517gH2b8vLgLsG9t/UyqYqlyRJkiTN0mySviXAocD7q+pFwHd4vCsnAFVVQM3iHE+QZE2SDUk2bNmyZa4OK0mSJEm9NZukbxOwqaqubeuX0CWB97Zum7TP+9r2zcCBA/svb2VTlT9JVZ1TVSurauXSpUtnEbokSZIk7RpmnPRV1T3AXUl+vBUdCdwKrAMmZuBcDVzaltcBJ7VZPA8HHmzdQC8HjkqyT5vA5ahWJkmSJEmapSWz3P+3gQuS7AHcAZxMl0henOQU4GvACa3uZcArgI3Aw60uVbU1yduB61u9M6tq6yzjkiRJkiQxy6Svqr4ArByy6cghdQs4dYrjnAecN5tYJEmSJElPNtv39EmSJEmSxphJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSdIIJDkwyVVJbk1yS5I3tvJ9k6xPcnv73KeVJ8l7k2xMclOSQweOtbrVvz3J6oW6JkmLg0mfJEnSaGwD3lRVhwCHA6cmOQQ4Dbiiqg4GrmjrAMcCB7efNcD7oUsSgTOAFwOHAWdMJIqSNIxJnyRJ0ghU1d1VdUNbfgi4DVgGrALWtmprgePa8irg/OpcA+yd5ADgaGB9VW2tqvuB9cAxI7wUSYuMSZ8kSdKIJVkBvAi4Fti/qu5um+4B9m/Ly4C7Bnbb1MqmKpekoUz6JEmSRijJ04GPAb9bVd8a3FZVBdQcnmtNkg1JNmzZsmWuDitpkTHpkyRJGpEku9MlfBdU1cdb8b2t2ybt875Wvhk4cGD35a1sqvInqapzqmplVa1cunTp3F2IpEXFpE+SJGkEkgQ4F7itqv5kYNM6YGIGztXApQPlJ7VZPA8HHmzdQC8HjkqyT5vA5ahWJklDLVnoACRJknYRLwF+GfhSki+0sjcD7wQuTnIK8DXghLbtMuAVwEbgYeBkgKramuTtwPWt3plVtXU0lyBpMTLpkyRJGoGq+hyQKTYfOaR+AadOcazzgPPmLjpJfWb3TkmSJEnqMZM+SZIkSeqxWSd9SXZLcmOST7T1g5Jcm2RjkouS7NHK92zrG9v2FQPHOL2VfyXJ0bONSZIkSZLUmYsnfW8EbhtYfxdwdlU9B7gfOKWVnwLc38rPbvVIcghwIvA84BjgfUl2m4O4JEmSJGmXN6ukL8ly4JXAB9t6gCOAS1qVtcBxbXlVW6dtP7LVXwVcWFWPVNVX6WaoOmw2cUmSJEmSOrN90vdu4PeB77f1ZwEPVNW2tr4JWNaWlwF3AbTtD7b6j5UP2ecJkqxJsiHJhi1btswydEmSJEnqvxknfUleBdxXVZ+fw3i2q6rOqaqVVbVy6dKlozqtJEmSJC1as3lP30uAVyd5BfAU4JnAe4C9kyxpT/OWA5tb/c3AgcCmJEuAvYBvDpRPGNxHkiRJkjQLM37SV1WnV9XyqlpBNxHLlVX1OuAq4PhWbTVwaVte19Zp269sLx1dB5zYZvc8CDgYuG6mcUmSJEmSHjebJ31T+QPgwiTvAG4Ezm3l5wIfTrIR2EqXKFJVtyS5GLgV2AacWlWPzkNckiRJkrTLmZOkr6quBq5uy3cwZPbNqvou8Jop9j8LOGsuYpEkSZIkPW4u3tMnSZIkSRpTJn2SeifJeUnuS3LzQNm+SdYnub197tPKk+S9STYmuSnJoQP7rG71b0+yeti5JEmSxp1Jn6Q++hBwzKSy04Arqupg4Iq2DnAs3QRSBwNrgPdDlyQCZwAvpuuyfsZEoihJkrSYmPRJ6p2q+izdhFGDVgFr2/Ja4LiB8vOrcw3da2cOAI4G1lfV1qq6H1jPkxNJSZKksWfSJ2lXsX9V3d2W7wH2b8vLgLsG6m1qZVOVS5IkLSomfZJ2Oe0doTVXx0uyJsmGJBu2bNkyV4eVJEmaEyZ9knYV97Zum7TP+1r5ZuDAgXrLW9lU5U9SVedU1cqqWrl06dI5D1ySJGk2TPok7SrWARMzcK4GLh0oP6nN4nk48GDrBno5cFSSfdoELke1MkmSpEVlTl7OLknjJMlHgZcD+yXZRDcL5zuBi5OcAnwNOKFVvwx4BbAReBg4GaCqtiZ5O3B9q3dmVU2eHEaSJGnsmfRJ6p2qeu0Um44cUreAU6c4znnAeXMYmiRJ0sjZvVOSJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkaUSSnJfkviQ3D5Ttm2R9ktvb5z6tPEnem2RjkpuSHDqwz+pW//YkqxfiWiQtHiZ9kiRJo/Mh4JhJZacBV1TVwcAVbR3gWODg9rMGeD90SSJwBvBi4DDgjIlEUZKGMemTJEkakar6LLB1UvEqYG1bXgscN1B+fnWuAfZOcgBwNLC+qrZW1f3Aep6cSErSY0z6JEmSFtb+VXV3W74H2L8tLwPuGqi3qZVNVS5JQ5n0SZIkjYmqKqDm6nhJ1iTZkGTDli1b5uqwkhaZGSd9SQ5MclWSW5PckuSNrdzByJIkSdN3b+u2Sfu8r5VvBg4cqLe8lU1V/iRVdU5VrayqlUuXLp3zwCUtDrN50rcNeFNVHQIcDpya5BAcjCxJkrQz1gETX3qvBi4dKD+pfXF+OPBg6wZ6OXBUkn3a70xHtTJJGmrGSV9V3V1VN7Tlh4Db6PqTOxhZkiRpiCQfBf4e+PEkm5KcArwT+LkktwM/29YBLgPuADYCfw78JkBVbQXeDlzffs5sZZI01JK5OEiSFcCLgGtxMLIkSdJQVfXaKTYdOaRuAadOcZzzgPPmMDRJPTbriVySPB34GPC7VfWtwW0ORpYkSZKkhTWrpC/J7nQJ3wVV9fFW7GBkSZIkSRoTs5m9M8C5wG1V9ScDmxyMLEmSJEljYjZj+l4C/DLwpSRfaGVvpht8fHEbmPw14IS27TLgFXSDkR8GToZuMHKSicHI4GBkSZIkSZozM076qupzQKbY7GBkSZIkSRoDs57IRZIkSZI0vkz6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHxibpS3JMkq8k2ZjktIWOR5LAtknS+LJ9kjRdY5H0JdkN+B/AscAhwGuTHLKwUUna1dk2SRpXtk+SdsZYJH3AYcDGqrqjqr4HXAisWuCYJMm2SdK4sn2SNG3jkvQtA+4aWN/UyiRpIdk2SRpXtk+Spm3JQgewM5KsAda01W8n+co0d90P+Macx/OuWe0+LzHN0oxjmuW92JFFd6/m+X5MZRzvE3nXTsX1o/MZy3wZt7YJbJ8GzeO/x0V3nxaobYIxvFe7QtsE49c+2TY9zt+dHmfb9ETz0T6NS9K3GThwYH15K3uCqjoHOGdnD55kQ1WtnHl4c8+Ypm8c4zKm6RvXuKZpl2ubYDzjMqbpGceYYDzjGseYdtIu1z4Z0/SNY1zGNH3zEde4dO+8Hjg4yUFJ9gBOBNYtcEySZNskaVzZPkmatrF40ldV25L8FnA5sBtwXlXdssBhSdrF2TZJGle2T5J2xlgkfQBVdRlw2Twdfqe7NYyAMU3fOMZlTNM3rnFNyy7YNsF4xmVM0zOOMcF4xjWOMe2UXbB9MqbpG8e4jGn65jyuVNVcH1OSJEmSNCbGZUyfJEmSJGke9CbpS3JgkquS3JrkliRvHFInSd6bZGOSm5IcOgYxvTzJg0m+0H7eMs8xPSXJdUm+2GJ625A6eya5qN2na5OsmM+YdiKu1yfZMnCv3jDfcbXz7pbkxiSfGLJt5PdqGjEt1H26M8mX2jk3DNk+0n9/48K2aafiGrv2ybZpTmOybRoztk/Tjsm2aediG7u2aRpxjfxejbxtqqpe/AAHAIe25WcA/xs4ZFKdVwCfAgIcDlw7BjG9HPjECO9TgKe35d2Ba4HDJ9X5TeADbflE4KIxiev1wJ8twN+t3wM+MuzPaSHu1TRiWqj7dCew33a2j/Tf37j82DbtVFxj1z7ZNs1pTLZNY/Zj+zTtmGybdi62sWubphHXyO/VqNum3jzpq6q7q+qGtvwQcBuwbFK1VcD51bkG2DvJAQsc00i1a/92W929/Uwe2LkKWNuWLwGOTJIxiGvkkiwHXgl8cIoqI79X04hpXI3039+4sG2avnFsn2yb5jSmcbVLtk1g+7QTMdk2TdM4tk3TjGsczem/vd4kfYPao+IX0X3rMWgZcNfA+iZG1JBsJyaAn2qP5z+V5HkjiGW3JF8A7gPWV9WU96mqtgEPAs8ag7gA/l17xH1JkgOHbJ9r7wZ+H/j+FNsX4l7tKCYY/X2C7j+bzyT5fJI1Q7Yv2L+/cWHbNK14xq59sm2as5jAtmls2T7tMBbbpukZx7ZpOnHB6O/VSNum3iV9SZ4OfAz43ar61kLHAzuM6QbgR6vqBcCfAn813/FU1aNV9UJgOXBYkufP9zmnYxpx/TWwoqp+AljP498UzYskrwLuq6rPz+d5dsY0YxrpfRrw0qo6FDgWODXJy0Z03kXBtml6xrF9sm3aMdumxc32acdsm3ZsHNsmGOv2aaRtU6+SviS70zUQF1TVx4dU2QwMZu7LW9mCxVRV35p4PF/d+3Z2T7LffMY0cO4HgKuAYyZteuw+JVkC7AV8cxQxbS+uqvpmVT3SVj8I/OQ8h/IS4NVJ7gQuBI5I8heT6oz6Xu0wpgW4TxPn3dw+7wP+EjhsUpWR//sbF7ZNO28c2yfbptnFZNs0nmyfdo5t03aNY9s0rbgWon0addvUm6Sv9Qc+F7itqv5kimrrgJPSORx4sKruXsiYkvzwRF/mJIfR/ZnM21/+JEuT7N2Wnwr8HPDlSdXWAavb8vHAlVU1r/3EpxPXpH7Mr6br5z9vqur0qlpeVSvoBhtfWVW/NKnaSO/VdGIa9X1q53xakmdMLANHATdPqjbSf3/jwrZpp+Iau/bJtmnuYrJtGj+2T9OOybZpGsaxbZpuXKO+VwvRNi2ZcbTj5yXALwNfSte/GeDNwLMBquoDwGV0M+FsBB4GTh6DmI4HfiPJNuCfgBPn+S//AcDaJLvRNZIXV9UnkpwJbKiqdXSN7YeTbAS20v0DmW/Tiet3krwa2Nbiev0I4nqSMbhXO4ppIe7T/sBftv+DlwAfqapPJ/l1WLB/f+PCtmn6xrF9sm2au5hsm8aP7dP02DbNwji2TUPiGvW9GnnblPn/P1ySJEmStFB6071TkiRJkvRkJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GP/B9gimz3n24k4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ba5d193b208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "y_train = np.concatenate(Y_train)\n",
    "y_valid = np.concatenate(Y_valid)\n",
    "y_test = np.concatenate(Y_test)\n",
    "\n",
    "plt.subplot(1,3,1)\n",
    "plt.hist(y_train)\n",
    "plt.title(\"Train\")\n",
    "plt.subplot(1,3,2)\n",
    "plt.hist(y_valid)\n",
    "plt.title(\"Validation\")\n",
    "plt.subplot(1,3,3)\n",
    "plt.hist(y_test)\n",
    "plt.title(\"Test\")\n",
    "plt.suptitle(\"Class Distribution\", fontsize=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Preprocessing\n",
    "\n",
    "### 2.1 Data overview\n",
    "First of all, let us gain some knowledge of our data by printing the amount of samples in training, validation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: 59708\n",
      "valid: 17536\n",
      "test: 25904\n",
      "total: 103148\n"
     ]
    }
   ],
   "source": [
    "# Number of samples in dataset\n",
    "n1 = sum([len(x) for x in X_train])\n",
    "n2 = sum([len(x) for x in X_valid])\n",
    "n3 = sum([len(x) for x in X_test])\n",
    "print(\"train:\", n1)\n",
    "print(\"valid:\", n2)\n",
    "print(\"test:\", n3)\n",
    "print(\"total:\", n1+n2+n3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Load preprocessing parameters\n",
    "In this experiment, as already highlighted we will be using a different dataset set up. Thus, we will also use different preprocessing parameters. You may find those at `'../preprocessing_random.h5'`. \n",
    "\n",
    "As before, following parameters are found in this file:\n",
    "\n",
    "- Mean image: Image containing the mean for each pixel location.\n",
    "- Pixel mean: Overal pixel intensity mean (scalar value).\n",
    "- Pixel standard deviation: Overall pixel standard deviation (scalar value).\n",
    "- Pixel maximum value: Overall maximum pixel intensity value (scalar value).\n",
    "- Pixel minimum value: Overall minimum pixel intensity value (scalar value).\n",
    "\n",
    "Depending on your needs, you may use `MeanImagePreprocessor` or`DefaultImagePreprocessor` to preprocess your data. Generation of these files is easy, and is done by `pyphoon.app.preprocess.generate_preprocess_params`. Set variable `generate_params` below to True to do so, however, the repository already provides precomputed preprocessing parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_params = False\n",
    "if generate_params:\n",
    "    from pyphoon.app.preprocess import generate_preprocess_params\n",
    "    generate_preprocess_params(X_train, \"../preprocessing_random.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this task, we will use `MeanImagePreprocessor`, which uses the dataset mean image to centre the data and the minimum and maximum pixel values to normalise the images. To this end, we have previously computed these parameters and stored them under file `random.h5`. Note that the next step may take a while..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "\n",
    "# Define preprocessor\n",
    "with h5py.File('../preprocessing_random.h5') as f:\n",
    "    mean = f.get('image_mean').value\n",
    "    scale_factor = f.get('max_value').value - f.get('min_value').value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Preprocessor\n",
    "We define the preprocessing pipeline here. However, the preprocessing of the data is done in the data generator when training the model. This way, we reduce the memory requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyphoon.app.preprocess import MeanImagePreprocessor\n",
    "\n",
    "# Define preprocessor\n",
    "preprocessor = MeanImagePreprocessor(mean, scale_factor, add_axis=[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model\n",
    "We use the same architecture and hyper-parameters than in the [Classification_Model](Classification_Model.ipynb) notebook. You may refer to section 3.3 to load pretrained weights instead of training. However, make sure to define the model architecture first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Input, Conv2D, Activation, MaxPooling2D, Flatten, \\\n",
    "    Dense, Dropout\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Model Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_img = Input(shape=(128, 128, 1), name=\"in\")\n",
    "############################################################################\n",
    "# Convolutional Layers\n",
    "############################################################################\n",
    "x = Conv2D(64, (3, 3), strides=(1, 1), padding='same', name='conv2d_1')(\n",
    "    input_img)\n",
    "x = Activation('relu', name='activation_1')(x)\n",
    "x = BatchNormalization(name=\"batch_normalization_1\")(x)\n",
    "x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='max_pooling2d_1')(x)\n",
    "\n",
    "x = Conv2D(128, (3, 3), strides=(1, 1), padding='same', name='conv2d_2')(x)\n",
    "x = Activation('relu', name='activation_2')(x)\n",
    "x = BatchNormalization(name=\"batch_normalization_2\")(x)\n",
    "x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='max_pooling2d_2')(x)\n",
    "\n",
    "x = Conv2D(128, (3, 3), strides=(1, 1), padding='same', name='conv2d_3')(x)\n",
    "x = Activation('relu', name='activation_3')(x)\n",
    "x = BatchNormalization(name=\"batch_normalization_3\")(x)\n",
    "x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='max_pooling2d_3')(x)\n",
    "\n",
    "x = Conv2D(256, (3, 3), strides=(1, 1), padding='same', name='conv2d_4')(x)\n",
    "x = Activation('relu', name='activation_4')(x)\n",
    "x = BatchNormalization(name=\"batch_normalization_4\")(x)\n",
    "x = MaxPooling2D((2, 2), strides=(2, 2), padding='same', name='max_pooling2d_4')(x)\n",
    "############################################################################\n",
    "# Dense Layers\n",
    "############################################################################\n",
    "x = Flatten()(x)\n",
    "\n",
    "x = Dense(512, use_bias=True, name=\"dense_1\")(x)\n",
    "x = Activation('relu', name=\"activation_5\")(x)\n",
    "x = BatchNormalization(name=\"batch_normalization_5\")(x)\n",
    "x = Dropout(0.2, name=\"dropout_1\")(x)\n",
    "\n",
    "x = Dense(256, use_bias=True, name=\"dense_2\")(x)\n",
    "x = Activation('relu', name=\"activation_6\")(x)\n",
    "x = BatchNormalization(name=\"fc_bn2\")(x)\n",
    "\n",
    "############################################################################\n",
    "# Output\n",
    "############################################################################\n",
    "x = Dense(4, use_bias=True, name=\"dense_3\")(x)\n",
    "x = Activation('softmax', name=\"out\")(x)\n",
    "\n",
    "############################################################################\n",
    "# Modelshinjuku to asa\n",
    "############################################################################\n",
    "model = Model(input_img, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[\n",
    "    'accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Callbacks\n",
    "Before training, we will define some usefull callbacks to supervise the training process. One of the most usefull callbacks is [Tensorboard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard), which enables easy visualisation of loss and accuracy curves.\n",
    "\n",
    "Make sure to use the following command to run the tensorboard client.\n",
    "\n",
    "```\n",
    "$ tensorboard --logdir=<path/to/log-directory> --port <port>\n",
    "```\n",
    "\n",
    "Once the model is training you may check its performance by navigating to `localhost:<port>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyphoon.app.callbacks import StoreModelWeights, OurTensorBoard, LossHistory\n",
    "\n",
    "foldername = 'results_random_2'\n",
    "storeweights_callback = StoreModelWeights(foldername, naming=\"val_acc\")\n",
    "tensorboard_callback = OurTensorBoard(foldername)\n",
    "loss_callback = LossHistory()\n",
    "\n",
    "callbacks = [loss_callback, storeweights_callback, tensorboard_callback]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Training\n",
    "Finally, To feed our model with input data we will use a data generator, provided by method `data_generator_from_chunklist` from `pyphoon.app.utils`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyphoon.app.utils import DataGeneratorFromChunklist\n",
    "\n",
    "# Generator\n",
    "batch_size = 32\n",
    "generator = DataGeneratorFromChunklist(batch_sz=batch_size,\n",
    "                                       preprocess_algorithm=preprocessor.apply,\n",
    "                                       target_enc='ohe',\n",
    "                                       crop=128\n",
    "                                       )\n",
    "\n",
    "# Number steps\n",
    "n_steps_train = sum([np.ceil(len(x)/batch_size) for x in Y_train])\n",
    "n_steps_valid = sum([np.ceil(len(x)/batch_size) for x in Y_valid])\n",
    "\n",
    "# Number of full epochs, i.e. #times algorithm sees all the data\n",
    "epochs = 10\n",
    "\n",
    "# Train\n",
    "model.fit_generator(\n",
    "    generator=generator.feed(X_train, Y_train),\n",
    "    steps_per_epoch=n_steps_train,\n",
    "    validation_data=generator.feed(X_valid, Y_valid),\n",
    "    validation_steps=n_steps_valid,\n",
    "    epochs=epochs,\n",
    "    callbacks=callbacks\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Load pre-trained weights\n",
    "Here we load the weights that provided the best performance and evaluate the performance on the test data. l."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights('../weights_random.hdf5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Results\n",
    "### 4.1 Test accuracy\n",
    "We evaluate the model that performed best on the validation set on the test set. This is the final performance of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on test set: 0.828211859172\n"
     ]
    }
   ],
   "source": [
    "# Generator\n",
    "batch_size = 32\n",
    "generator = DataGeneratorFromChunklist(batch_sz=batch_size, \n",
    "                                       preprocess_algorithm=preprocessor.apply, \n",
    "                                       target_enc='ohe',\n",
    "                                       crop=128)\n",
    "# Number steps\n",
    "n_steps = sum([np.ceil(len(x)/batch_size) for x in Y_test])\n",
    "\n",
    "# Get predictions\n",
    "y_pred = model.predict_generator(\n",
    "    generator=generator.feed(X_test, Y_test, shuffle_batches=False, shuffle_samples=False),\n",
    "    steps=n_steps\n",
    ")\n",
    "y_pred = np.argmax(y_pred, axis=1)\n",
    "\n",
    "# Ground truth\n",
    "y_true = (np.concatenate(Y_test) - 2).tolist()\n",
    "\n",
    "# Test accuracy\n",
    "acc_test = (np.array(y_true) == np.array(y_pred)).mean()\n",
    "print(\"Accuracy on test set:\", acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Confusion Matrix, precision, recall\n",
    "Now we take a closer look in an attempt to see if our model is biased towards some category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted 2</th>\n",
       "      <th>predicted 3</th>\n",
       "      <th>predicted 4</th>\n",
       "      <th>predicted 5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5586</td>\n",
       "      <td>769</td>\n",
       "      <td>105</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>809</td>\n",
       "      <td>4893</td>\n",
       "      <td>660</td>\n",
       "      <td>114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>147</td>\n",
       "      <td>646</td>\n",
       "      <td>5169</td>\n",
       "      <td>514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>26</td>\n",
       "      <td>131</td>\n",
       "      <td>513</td>\n",
       "      <td>5806</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   predicted 2  predicted 3  predicted 4  predicted 5\n",
       "2         5586          769          105           16\n",
       "3          809         4893          660          114\n",
       "4          147          646         5169          514\n",
       "5           26          131          513         5806"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import pandas as pd\n",
    "\n",
    "pd.DataFrame(confusion_matrix(y_true, y_pred), \n",
    "             columns=['predicted 2','predicted 3','predicted 4','predicted 5'], \n",
    "             index=['2', '3', '4', '5']\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.85      0.86      0.86      6476\n",
      "          1       0.76      0.76      0.76      6476\n",
      "          2       0.80      0.80      0.80      6476\n",
      "          3       0.90      0.90      0.90      6476\n",
      "\n",
      "avg / total       0.83      0.83      0.83     25904\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "# Get complete report\n",
    "print(classification_report(y_true, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusions\n",
    "From this short experiment, we can conclude that the way the data is distributed among all datasets (training, validation and test) has a big impact on the final evaluated performance of the model.\n",
    "\n",
    "Performance using random split is high, as expected. This might be due to several reasons:\n",
    "- Images from different sequences might be subject to some measurement biases due to variations in the state of the sensors throughout the years. Also, depending on the period, different satellites were being used.\n",
    "- Images one hour apart from the same sequence are practically identical. Therefore, practically speaking the model is evaluated on data that it has almost \"already seen\" during training.\n",
    "\n",
    "We argue that random split is not a realistic set up. Reasons:\n",
    "- In general, we want to make our predictions using data from the past. That is, from other sequences that we registered and analysed before.\n",
    "- When analysing data from a new sequence, predictions from it are usually not available or might be subject to some errors."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}