BankReviewSentimentNoteBook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review Sentiment Classification Notebook\n",
    "\n",
    "\n",
    "## Summary\n",
    "Text Classification aims to assign a text instance into one or more class(es) in a predefined set of classes.\n",
    "\n",
    "## Description \n",
    "### Use Case Description\n",
    "A company, such as bank, wants to analyze customer feedback in order to provide additional insight to enhance market campaign prediction. The bank collects customers feedback from public website. The task is to build a pipeline that automatically analyzes customer feedback messages, to provide the overall sentiment for the bank. The aim is to help the bank who wants to more accurately predict the success of telemarketing calls for selling bank long-term deposits gain extra features from social media.\n",
    "\n",
    "#### Use Case Data\n",
    "The data used in this use case is [BankReview dataset](https://www.creditkarma.com/reviews/banking/single/id/Simple#single-review-listingPaper), a publicly available data set collected from credit karma website. The data comprises approximately 120 customers feedback. \n",
    "\n",
    "We shared the review data in the data folder. You can use this shared data to follow the steps in this template, or you can collect more feedbacks from credit karma website.\n",
    "\n",
    "Each instance in the data set has 2 fields:\n",
    "\n",
    "* sentiment - the polarity of the feedback (1 = strongly negative, 2 = negative, 3 = neutral, 4 = positive, 5 = strongly positive)\n",
    "* review - the text of the feedback "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1 - Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data from Local\n",
    "Start by loading data from local to IPython Notebook using Pandas. Then, use some simple visualization techniques to peak at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>I absolutely love my bank. There's a reason th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>International transaction fees for items bough...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>This bank bends over backwards for its custome...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>I have used this bank for going on two years a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>This card makes it so easy to save.... for rai...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                             review\n",
       "0          4  I absolutely love my bank. There's a reason th...\n",
       "1          1  International transaction fees for items bough...\n",
       "2          5  This bank bends over backwards for its custome...\n",
       "3          5  I have used this bank for going on two years a...\n",
       "4          5  This card makes it so easy to save.... for rai..."
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the bank review dataset\n",
    "import csv\n",
    "import pandas as pd\n",
    "\n",
    "text_df = pd.read_csv('./data/BankReviewTrainingSample.csv')\n",
    "text_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_df.columns = ['label_column', 'text_column'] #rename column names\n",
    "label_column = text_df[\"label_column\"]\n",
    "text_df[\"label_column\"] = label_column.map(lambda label: 1 if label == 4 or label == 5 else 0) #replace label 4 to label 1\n",
    "text_df = text_df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label_column</th>\n",
       "      <th>text_column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>I absolutely love my bank. There's a reason th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>International transaction fees for items bough...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>This bank bends over backwards for its custome...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>I have used this bank for going on two years a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>This card makes it so easy to save.... for rai...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label_column                                        text_column\n",
       "0             1  I absolutely love my bank. There's a reason th...\n",
       "1             0  International transaction fees for items bough...\n",
       "2             1  This bank bends over backwards for its custome...\n",
       "3             1  I have used this bank for going on two years a...\n",
       "4             1  This card makes it so easy to save.... for rai..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x16d5e108b70>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEuRJREFUeJzt3X+s3XV9x/Hn21aG9mrLDz1rCtutAdkYRLQnBEPmzqVo\nKi62yQiB4FaWxht1Y2xumd3MovthAn+gc8Rka8S1W5ALI2obKiqrvSMzFm0FLT+GVCxKLa1KW73I\nVOZ7f9xvtdS253vPz97PfT6Sm/v9fs/nez7v9z23r/vt93zPOZGZSJJmvxcNuwBJUm8Y6JJUCANd\nkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCzB/kZGeeeWaOjo52tO+zzz7LggULelvQSc6e\n5wZ7Ll+3/e7YseN7mfmKduMGGuijo6Ns3769o30nJydptVq9LegkZ89zgz2Xr9t+I+LJOuNqnXKJ\niD+LiIcj4qGIuD0iTo2IpRFxf0Tsiog7IuKUjquVJHWtbaBHxBLgT4BmZl4AzAOuBm4CPpSZ5wAH\ngDX9LFSSdGJ1nxSdD7wkIuYDLwX2ApcBd1W3bwBW9b48SVJdUeftcyPiBuADwHPA54AbgG3V0TkR\ncTZwT3UEf/S+48A4QKPRWDYxMdFRoVNTU4yMjHS072xlz3ODPZev237HxsZ2ZGaz7cDMPOEXcBrw\neeAVwIuBTwFvA3YdMeZs4KF297Vs2bLs1NatWzved7ay57nBnsvXbb/A9myTr5lZ65TL5cA3M/O7\nmflT4BPApcCi6hQMwFnAnpn8xZEk9VadQP8WcElEvDQiAlgOPAJsBa6sxqwGNvanRElSHW0DPTPv\nZ/rJz68AO6t91gHvAd4dEbuAM4Bb+1inJKmNWi8sysz3Ae87avMTwMU9r0iS1JGBvlJUkoZpdO3m\nocy7fsVg3ubAN+eSpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgD\nXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBWibaBHxHkR8eARXz+IiD+NiNMj4t6IeLz6ftogCpYk\nHVudzxR9LDMvysyLgGXAj4BPAmuBLZl5LrClWpckDclMT7ksB76RmU8CK4EN1fYNwKpeFiZJmpmZ\nBvrVwO3VciMz91bLTwONnlUlSZqxyMx6AyNOAb4D/FZm7ouIg5m56IjbD2TmL51Hj4hxYByg0Wgs\nm5iY6KjQqakpRkZGOtp3trLnucGeB2fnnkMDnxNg6cJ5XfU7Nja2IzOb7cbNn8F9vhn4Smbuq9b3\nRcTizNwbEYuB/cfaKTPXAesAms1mtlqtGUz5C5OTk3S672xlz3ODPQ/OdWs3D3xOgPUrFgyk35mc\ncrmGX5xuAdgErK6WVwMbe1WUJGnmagV6RCwA3gh84ojNNwJvjIjHgcurdUnSkNQ65ZKZzwJnHLXt\n+0xf9SJJOgn4SlFJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQ\nJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYWo+5miiyLiroj4n4h4NCJeHxGnR8S9\nEfF49f20fhcrSTq+ukfoHwY+k5m/AbwGeBRYC2zJzHOBLdW6JGlI2gZ6RCwE3gDcCpCZP8nMg8BK\nYEM1bAOwql9FSpLaq3OEvhT4LvCvEfFARHw0IhYAjczcW415Gmj0q0hJUnuRmSceENEEtgGXZub9\nEfFh4AfA9Zm56IhxBzLzl86jR8Q4MA7QaDSWTUxMdFTo1NQUIyMjHe07W9nz3GDPg7Nzz6GBzwmw\ndOG8rvodGxvbkZnNduPqBPqvAtsyc7Ra/22mz5efA7Qyc29ELAYmM/O8E91Xs9nM7du312zhhSYn\nJ2m1Wh3tO1vZ89xgz4MzunbzwOcEWL9iQVf9RkStQG97yiUznwa+HRGHw3o58AiwCVhdbVsNbOyw\nVklSD8yvOe564LaIOAV4AvhDpv8Y3BkRa4Angav6U6IkqY5agZ6ZDwLHOtxf3ttyJEmd8pWiklQI\nA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQ\nJakQBrokFcJAl6RCGOiSVIhan1gUEbuBHwL/Bzyfmc2IOB24AxgFdgNXZeaB/pQpSWpnJkfoY5l5\n0RGfPL0W2JKZ5wJbqnVJ0pB0c8plJbChWt4ArOq+HElSp+oGegKfi4gdETFebWtk5t5q+Wmg0fPq\nJEm1RWa2HxSxJDP3RMQrgXuB64FNmbnoiDEHMvO0Y+w7DowDNBqNZRMTEx0VOjU1xcjISEf7zlb2\nPDfY8+Ds3HNo4HMCLF04r6t+x8bGdhxxuvu4agX6C3aIeD8wBbwdaGXm3ohYDExm5nkn2rfZbOb2\n7dtnNN9hk5OTtFqtjvadrex5brDnwRldu3ngcwKsX7Ggq34jolagtz3lEhELIuJlh5eBNwEPAZuA\n1dWw1cDGjquVJHWtzmWLDeCTEXF4/Mcz8zMR8WXgzohYAzwJXNW/MiVJ7bQN9Mx8AnjNMbZ/H1je\nj6IkSTPnK0UlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RC\nGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYWo84lFJ4Wdew5x3RA+D3D3jW8Z+JyS1AmP0CWpELUDPSLm\nRcQDEXF3tb40Iu6PiF0RcUdEnNK/MiVJ7czkCP0G4NEj1m8CPpSZ5wAHgDW9LEySNDO1Aj0izgLe\nAny0Wg/gMuCuasgGYFU/CpQk1VP3CP0fgb8EflatnwEczMznq/WngCU9rk2SNAORmSceEPG7wBWZ\n+a6IaAF/AVwHbKtOtxARZwP3ZOYFx9h/HBgHaDQayyYmJjoqdP8zh9j3XEe7duXCJQsHP2llamqK\nkZGRoc0/DPY8Nwyr5517Dg18ToClC+d11e/Y2NiOzGy2G1fnssVLgbdGxBXAqcDLgQ8DiyJifnWU\nfhaw51g7Z+Y6YB1As9nMVqtVr4Oj3HLbRm7eOfirLHdf2xr4nIdNTk7S6c9rtrLnuWFYPQ/j0meA\n9SsWDKTftqdcMvOvMvOszBwFrgY+n5nXAluBK6thq4GNfatSktRWN9ehvwd4d0TsYvqc+q29KUmS\n1IkZncPIzElgslp+Ari49yVJkjrhK0UlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5J\nhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEG0DPSJOjYgvRcRXI+Lh\niPjbavvSiLg/InZFxB0RcUr/y5UkHU+dI/QfA5dl5muAi4AVEXEJcBPwocw8BzgArOlfmZKkdtoG\nek6bqlZfXH0lcBlwV7V9A7CqLxVKkmqpdQ49IuZFxIPAfuBe4BvAwcx8vhryFLCkPyVKkuqIzKw/\nOGIR8Engb4D11ekWIuJs4J7MvOAY+4wD4wCNRmPZxMRER4Xuf+YQ+57raNeuXLhk4eAnrUxNTTEy\nMjK0+YfBnueGYfW8c8+hgc8JsHThvK76HRsb25GZzXbj5s/kTjPzYERsBV4PLIqI+dVR+lnAnuPs\nsw5YB9BsNrPVas1kyp+75baN3LxzRuX2xO5rWwOf87DJyUk6/XnNVvY8Nwyr5+vWbh74nADrVywY\nSL91rnJ5RXVkTkS8BHgj8CiwFbiyGrYa2NivIiVJ7dU55F0MbIiIeUz/AbgzM++OiEeAiYj4B+AB\n4NY+1ilJaqNtoGfm14DXHmP7E8DF/ShKkjRzvlJUkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJA\nl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1Ih6nxI\n9NkRsTUiHomIhyPihmr76RFxb0Q8Xn0/rf/lSpKOp84R+vPAn2fm+cAlwB9FxPnAWmBLZp4LbKnW\nJUlD0jbQM3NvZn6lWv4h8CiwBFgJbKiGbQBW9atISVJ7MzqHHhGjwGuB+4FGZu6tbnoaaPS0MknS\njERm1hsYMQL8F/CBzPxERBzMzEVH3H4gM3/pPHpEjAPjAI1GY9nExERHhe5/5hD7nuto165cuGTh\n4CetTE1NMTIyMrT5h8Ge54Zh9bxzz6GBzwmwdOG8rvodGxvbkZnNduNqBXpEvBi4G/hsZn6w2vYY\n0MrMvRGxGJjMzPNOdD/NZjO3b99eq4Gj3XLbRm7eOb+jfbux+8a3DHzOwyYnJ2m1WkObfxjseW4Y\nVs+jazcPfE6A9SsWdNVvRNQK9DpXuQRwK/Do4TCvbAJWV8urgY2dFCpJ6o06h7yXAr8P7IyIB6tt\nfw3cCNwZEWuAJ4Gr+lOiJKmOtoGemf8NxHFuXt7bciRJnfKVopJUCANdkgphoEtSIQx0SSqEgS5J\nhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQI\nA12SClHnQ6I/FhH7I+KhI7adHhH3RsTj1ffT+lumJKmdOkfo64EVR21bC2zJzHOBLdW6JGmI2gZ6\nZt4HPHPU5pXAhmp5A7Cqx3VJkmYoMrP9oIhR4O7MvKBaP5iZi6rlAA4cXj/GvuPAOECj0Vg2MTHR\nUaH7nznEvuc62rUrFy5ZOPhJK1NTU4yMjAxt/mGw57lhWD3v3HNo4HMCLF04r6t+x8bGdmRms924\n+R3PUMnMjIjj/lXIzHXAOoBms5mtVqujeW65bSM37+y63BnbfW1r4HMeNjk5Sac/r9nKnueGYfV8\n3drNA58TYP2KBQPpt9OrXPZFxGKA6vv+3pUkSepEp4G+CVhdLa8GNvamHElSp+pctng78EXgvIh4\nKiLWADcCb4yIx4HLq3VJ0hC1PSmdmdcc56blPa5FktQFXykqSYUw0CWpEAa6JBXCQJekQhjoklQI\nA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5Jhegq\n0CNiRUQ8FhG7ImJtr4qSJM1cx4EeEfOAjwBvBs4HromI83tVmCRpZro5Qr8Y2JWZT2TmT4AJYGVv\nypIkzVQ3gb4E+PYR609V2yRJQzC/3xNExDgwXq1ORcRjHd7VmcD3elNVfXHToGd8gaH0PGT2PDfM\nqZ7Hbuq631+vM6ibQN8DnH3E+lnVthfIzHXAui7mASAitmdms9v7mU3seW6w5/INqt9uTrl8GTg3\nIpZGxCnA1cCm3pQlSZqpjo/QM/P5iPhj4LPAPOBjmflwzyqTJM1IV+fQM/PTwKd7VEs7XZ+2mYXs\neW6w5/INpN/IzEHMI0nqM1/6L0mFOOkCvd3bCUTEr0TEHdXt90fE6OCr7K0aPb87Ih6JiK9FxJaI\nqHUJ08ms7ttGRMTvRURGxKy+IqJOvxFxVfU4PxwRHx90jb1W4/f61yJia0Q8UP1uXzGMOnspIj4W\nEfsj4qHj3B4R8U/Vz+RrEfG6nhaQmSfNF9NPrn4DeBVwCvBV4PyjxrwL+Odq+WrgjmHXPYCex4CX\nVsvvnAs9V+NeBtwHbAOaw667z4/xucADwGnV+iuHXfcAel4HvLNaPh/YPey6e9D3G4DXAQ8d5/Yr\ngHuAAC4B7u/l/CfbEXqdtxNYCWyolu8ClkdEDLDGXmvbc2ZuzcwfVavbmL7mfzar+7YRfw/cBPzv\nIIvrgzr9vh34SGYeAMjM/QOusdfq9JzAy6vlhcB3BlhfX2TmfcAzJxiyEvi3nLYNWBQRi3s1/8kW\n6HXeTuDnYzLzeeAQcMZAquuPmb6Fwhqm/8LPZm17rv4renZmbh5kYX1S5zF+NfDqiPhCRGyLiBUD\nq64/6vT8fuBtEfEU01fLXT+Y0oaqr2+Z0veX/qt3IuJtQBP4nWHX0k8R8SLgg8B1Qy5lkOYzfdql\nxfT/wO6LiAsz8+BQq+qva4D1mXlzRLwe+PeIuCAzfzbswmark+0Ivc7bCfx8TETMZ/q/at8fSHX9\nUestFCLicuC9wFsz88cDqq1f2vX8MuACYDIidjN9rnHTLH5itM5j/BSwKTN/mpnfBL7OdMDPVnV6\nXgPcCZCZXwROZfo9XkpW6997p062QK/zdgKbgNXV8pXA57N6tmGWattzRLwW+Bemw3y2n1uFNj1n\n5qHMPDMzRzNzlOnnDd6amduHU27X6vxef4rpo3Mi4kymT8E8Mcgie6xOz98ClgNExG8yHejfHWiV\ng7cJ+IPqapdLgEOZubdn9z7sZ4WP8yzw15l+hvy91ba/Y/ofNEw/6P8B7AK+BLxq2DUPoOf/BPYB\nD1Zfm4Zdc797PmrsJLP4Kpeaj3EwfZrpEWAncPWwax5Az+cDX2D6CpgHgTcNu+Ye9Hw7sBf4KdP/\n61oDvAN4xxGP80eqn8nOXv9e+0pRSSrEyXbKRZLUIQNdkgphoEtSIQx0SSqEgS5JhTDQJakQBrok\nFcJAl6RC/D+9n3AB+blYqAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x16d5d963080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib \n",
    "\n",
    "%matplotlib inline\n",
    "text_df[\"label_column\"].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2 - Text Preprocessing\n",
    "\n",
    "Unstructured text such as tweets, product reviews, or search queries usually requires some preprocessing before it can be analyzed. This experiment includes a number of optional text preprocessing and text cleaning steps, such as replacing special characters and punctuation marks with spaces, normalizing case, removing duplicate characters, removing user-defined or built-in stop-words, and word stemming.\n",
    "\n",
    "In this step, the user can visualize the most frequent words for each class. In the sentiment analysis use case, the first word cloud, shown below, represents the top positive sentiment-bearing words while the second word cloud shows the most frequent negative sentiment-bearing words in the input training corpus. This visualization helps us validate the quality of the training data before start to build the classification model in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "stop_words_df = pd.read_csv('./data/StopWords.csv')\n",
    "stop_words = set(stop_words_df[\"Col1\"].tolist())\n",
    "for item in string.ascii_lowercase: #load stop words\n",
    "    if item != \"i\":\n",
    "        stop_words.add(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "text_column = []\n",
    "for line in text_df.text_column:\n",
    "    value = \" \".join(item.lower()\n",
    "                     for item in RegexpTokenizer(r'\\w+').tokenize(line)\n",
    "                     if item.lower() not in stop_words)\n",
    "    text_column.append(value)\n",
    "text_df.text_column = text_column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label_column</th>\n",
       "      <th>text_column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>i absolutely love my bank reason bank customer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>international transaction fees for items bough...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>bank bends backwards for its customers i ve on...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>i used bank for going on years it best bank ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>card makes it so easy to save for rainy days f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label_column                                        text_column\n",
       "0             1  i absolutely love my bank reason bank customer...\n",
       "1             0  international transaction fees for items bough...\n",
       "2             1  bank bends backwards for its customers i ve on...\n",
       "3             1  i used bank for going on years it best bank ap...\n",
       "4             1  card makes it so easy to save for rainy days f..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the most frequent 50 words in both positive and negative classes\n",
    "Use wordcloud package to visualize the most frequent words in both positive and negative class as a word cloud. These most frequent words will give us initial understanding of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing wordcloud module\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "positive_text = \" \".join(text_df[text_df.label_column == 1].text_column.tolist())\n",
    "negative_text = \" \".join(text_df[text_df.label_column == 0].text_column.tolist())\n",
    "\n",
    "# Generate word \n",
    "\n",
    "try:\n",
    "    from wordcloud import WordCloud\n",
    "except ImportError:\n",
    "    print (\"Missing wordcloud module\")\n",
    "else:\n",
    "    import matplotlib.pyplot as plt\n",
    "    try:\n",
    "        positive_word_cloud = WordCloud(max_words=50, margin=10, random_state=1).generate(positive_text)\n",
    "        default_colors = positive_word_cloud.to_array()\n",
    "        plt.title(\"Top 50 words in positive class\")\n",
    "        plt.imshow(default_colors)\n",
    "        plt.axis(\"off\")\n",
    "        image = plt.show()\n",
    "        negative_word_cloud = WordCloud(max_words=50, margin=10, random_state=1).generate(negative_text)\n",
    "        default_colors = negative_word_cloud.to_array()\n",
    "        plt.title(\"Top 50 words in negative class\")\n",
    "        plt.imshow(default_colors)\n",
    "        plt.axis(\"off\")\n",
    "        image = plt.show()\n",
    "    except IOError as error:\n",
    "        print (\"Missing WordCloud data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3 - Feature Engineering \n",
    "\n",
    "The aim of this step is to convert variable-length unstructured text data into equal-length numeric feature vectors. The user has the option to try two types of text representations: N-Gram TF and Unigrams TF-IDF. Here, we chose the unigrams TF-IDF feature extraction method.\n",
    "\n",
    "Then, we performed feature selection to reduce the size of the data. We used the Chi-squared score function to rank the hashing features in descending order, and returned the top 30 most relevant features with respect to the sentiment label, out of all the extracted unigrams."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stem and Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.stem.porter import PorterStemmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize stemmer to remove morphological affixes from words, leaving only the word stem, and convert the text_column in data frame as a list for future process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = PorterStemmer()\n",
    "text_list = text_df[\"text_column\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenize the sentences in text_list and remove morphological affixes from words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stem_tokens(tokens, stemmer_model):\n",
    "    '''\n",
    "    :param tokens: tokenized word list\n",
    "    :param stemmer: remove stemmer\n",
    "    :return:  tokenized and stemmed words\n",
    "    '''\n",
    "    return [stemmer_model.stem(original_word) for original_word in tokens]\n",
    "\n",
    "\n",
    "def tokenize(text):\n",
    "    '''\n",
    "    :param text: raw test\n",
    "    :return: tokenized and stemmed words\n",
    "    '''\n",
    "    tokens = text.strip().split(\" \")\n",
    "    return stem_tokens(tokens, stemmer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unigrams TF-IDF feature extraction\n",
    "\n",
    "#### Create the Word Dictionary\n",
    "First, extract the set of unigrams (words) that will be used to train the text model. In addition to the unigrams, the number of documents where each word appears in the text corpus is counted (DF). It is not necessary to create the dictionary on the same labeled data used to train the text model -- any large corpus that fairly represents the frequency of words in the target domain of classification can be used, even if it is not annotated with the target categories.\n",
    "\n",
    "#### TF-IDF Calculation\n",
    "When the metric word frequency of occurrence (TF) in a document is used as a feature value, a higher weight tends to be assigned to words that appear frequently in a corpus (such as stop-words). The inverse document frequency (IDF) is a better metric, because it assigns a lower weight to frequent words. You calculate IDF as the log of the ratio of the number of documents in the training corpus to the number of documents containing the given word. Combining these numbers in a metric (TF/IDF) places greater importance on words that are frequent in the document but rare in the corpus. This assumption is valid not only for unigrams but also for bigrams, trigrams, etc.\n",
    "\n",
    "This experiment converts unstructured text data into equal-length numeric feature vectors where each feature represents the TF-IDF of a unigram in a text instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize the TfidfVectorizer to compute tf-idf for each word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 2)\t0.221753831939\n",
      "  (0, 49)\t0.368544633283\n",
      "  (0, 13)\t0.406986984004\n",
      "  (0, 26)\t0.242029788625\n",
      "  (0, 75)\t0.124843206538\n",
      "  (0, 46)\t0.160383216913\n",
      "  (0, 60)\t0.227581612941\n",
      "  (0, 19)\t0.16500416833\n",
      "  (0, 18)\t0.202485683627\n",
      "  (0, 10)\t0.153999605087\n",
      "  (0, 65)\t0.211429246413\n",
      "  (0, 1)\t0.198425214393\n",
      "  (0, 28)\t0.160383216913\n",
      "  (0, 52)\t0.266452574283\n",
      "  (0, 77)\t0.106567331634\n",
      "  (0, 4)\t0.232457541703\n",
      "  (0, 85)\t0.202485683627\n",
      "  (0, 27)\t0.194596902167\n",
      "  (0, 40)\t0.16996771502\n",
      "  (0, 67)\t0.227581612941\n",
      "  (1, 4)\t0.311352804109\n",
      "  (1, 84)\t0.579671474006\n",
      "  (1, 36)\t0.442011991718\n",
      "  (1, 44)\t0.609644004953\n",
      "  (2, 13)\t0.285643697073\n",
      "  :\t:\n",
      "  (118, 28)\t0.131942194735\n",
      "  (118, 52)\t0.109601048415\n",
      "  (118, 77)\t0.175339139512\n",
      "  (118, 4)\t0.0956177299141\n",
      "  (118, 84)\t0.356038999546\n",
      "  (118, 30)\t0.141988120997\n",
      "  (118, 73)\t0.154283341055\n",
      "  (118, 35)\t0.16008871037\n",
      "  (118, 72)\t0.192475782951\n",
      "  (118, 78)\t0.163238141632\n",
      "  (118, 38)\t0.732912257642\n",
      "  (118, 58)\t0.15159500074\n",
      "  (118, 59)\t0.178019499773\n",
      "  (119, 49)\t0.702475767354\n",
      "  (119, 26)\t0.153776093072\n",
      "  (119, 75)\t0.158640807457\n",
      "  (119, 10)\t0.195690437441\n",
      "  (119, 28)\t0.203802223118\n",
      "  (119, 88)\t0.198309392736\n",
      "  (119, 16)\t0.252142964795\n",
      "  (119, 8)\t0.203802223118\n",
      "  (119, 41)\t0.234158589118\n",
      "  (119, 35)\t0.24727825041\n",
      "  (119, 38)\t0.2264160419\n",
      "  (119, 57)\t0.257302685823\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<120x92 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 1606 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english', max_df=160000,\n",
    "                        min_df=9, norm=\"l2\", use_idf=True)\n",
    "tfs = tfidf.fit_transform(text_list)\n",
    "print(tfs)\n",
    "tfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform Feature Selection using Chi-squared Test\n",
    "The Chi-squared statistic measures dependence between stochastic variables, so a transformer based on this function \"weeds out\" the features that are the most likely to be independent of class and therefore irrelevant for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<120x30 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 492 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = text_df.label_column.tolist()  # Load labels\n",
    "# Perform chi-squared test to select top 30 features\n",
    "selected_features = SelectKBest(chi2, k=30).fit_transform(tfs, labels)\n",
    "selected_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4 - Train and Evaluate Models with Different Top K Words\n",
    "\n",
    "In this step, users can train and evaluate text classification models using state-of-the-art ML algorithms ranging from Two-Class Logistic Regression, Two-Class Support Vector Machine and Two-Class Boosted Decision Tree for binary text classification to One-vs-All Multiclass, Multiclass Logistic Regression and Multiclass Decision Forest for multi-class text classification. The user will also find the Sweep Parameters module that allows her to get the optimal values for the underlying learning algorithm parameters. In the case of binary models, the user has the advantage of visualizing the weights assigned to different words/unigrams that help her justify the decision of the trained model. Here, we used Two-Class Logistic Regression classifier as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Select Top Features\n",
    "Select top k words based on Chi-squared test score to formalize the feature vector.\n",
    "The features for this task is the words in different sentence, if we choose to use all words as features, then the feature space will become very large and sparse, which makes the training process much slower.\n",
    "Besides that feature selection will also simplify our model and reduce the chance of overfitting.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_top_features(chi_square, top_k_features, tf_idf_features, label_list):\n",
    "    '''\n",
    "    :param chi2: chi squared feature selctor\n",
    "    :param top_k: number of top features\n",
    "    :param tfs: tf-idf feature list\n",
    "    :param labels: labels of the feature list\n",
    "    :return: feature list after chi squared test\n",
    "    '''\n",
    "    return SelectKBest(chi_square, k=top_k_features).fit_transform(tf_idf_features, label_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Data for Training and Testing\n",
    "Split the transformed feature vectors into 80% as training set and 20% as test set.\n",
    "Training set is used to train a classifier and test set is used to test the performance of the trained classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dsvmadmin\\AppData\\local\\AmlWorkbench\\Python\\lib\\site-packages\\sklearn\\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "def split_data(selected_features_list, label_list):\n",
    "    '''\n",
    "    :param selected_features: feature list after feature selection\n",
    "    :param labels: labels of the feature list\n",
    "    :return: 80% training set and 20% test set\n",
    "    '''\n",
    "    return train_test_split(selected_features_list, label_list, test_size=0.2, random_state=12345)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Sweep Parameter\n",
    "To find the optimal parameters for the logistic regression, we test both l1 and l2 regularization with different regularization parameters C (0.1, 1 and 10). Usually a classifier will have several parameters to tune in order to get the best performance. These regularization parameters are used to avoid overfitting on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_sweep_parameter_dict():\n",
    "    '''\n",
    "    :return: parameters of logistic regression model to tune\n",
    "    '''\n",
    "    return {\"penalty\": (\"l1\", \"l2\"), \"C\": [0.1, 1, 10]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sweep Parameters with Logistic Regression using Cross Validation\n",
    "Use 3-fold cross validation to find the optimal parameters for logistic regression algorithm, then train the algorithm with all training data and the optimal parameters. 3-fold cross validation will first split the training set into 3 equal sized subsamples. Of the 3 subsamples, a single subsample is retained as the validation set for testing the model, and the remaining k-1 subsamples are used as training data. The cross validation process is repeated 3 times with each of the 3 subsamples used exactly once as the validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dsvmadmin\\AppData\\local\\AmlWorkbench\\Python\\lib\\site-packages\\sklearn\\grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn import grid_search\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def sweep_logistic_regression_cross_val(train_feature, train_label):\n",
    "    '''\n",
    "    :param train_x: feature of training set\n",
    "    :param train_y: label of training set\n",
    "    :return: trained logistic regression model\n",
    "    '''\n",
    "    model = LogisticRegression()\n",
    "    classifier = grid_search.GridSearchCV(model, make_sweep_parameter_dict())\n",
    "    classifier.fit(train_feature, train_label)\n",
    "    return classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate Model using Testing Data\n",
    "The probability of each record in test set is calculated by the trained model to calculate the AUC. AUC (area under the curve) is an important metric to evaluate the performance of the classifier. The value of the AUC lies in range [0, 1] and the bigger AUC means better performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prediction(classifier, test_feature):\n",
    "    '''\n",
    "    :param clf: the trained classifier\n",
    "    :param test_x: feature of test set\n",
    "    :return: score probability of test set\n",
    "    '''\n",
    "    return [classifier.predict_proba(feature)[0][1] for feature in test_feature]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute AUC\n",
    "Compute the AUC based on the label and the scores of test set and prepare input to plot ROC curve. In statistics, a ROC (receiver operating characteristic curve), is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. And AUC is the area under ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "def calc_auc(test_label, predict_probability_list):\n",
    "    '''\n",
    "    :param test_y: label of test set\n",
    "    :param predict: score probability of test set\n",
    "    :return: falst positive list, true positive list,, AUC\n",
    "    '''\n",
    "    fpr_list, tpr_list = metrics.roc_curve(test_label,\n",
    "                                           predict_probability_list, pos_label=1)[:-1]\n",
    "    auc = metrics.auc(fpr_list, tpr_list)\n",
    "    return fpr_list, tpr_list, auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Put it all together\n",
    "We select different top k words list based on Chi-squared test, and then a 3-fold cross validation is used to find the optimal parameters for the logistic regression algorithm based on the top k words. We trained the logistic regression algorithm with the optimal parameters, test its performance on test set and plot the ROC curve for each top k words list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8jFf7+PHPyUIsoaUoEqT2WEOaIBGJfV/aqn0nlLZ+\nRakHpb7dH/UoT63VKlVqqaLVatVW+9LEvkdlUcQSS0gkM+f3x4x5IiKZRCaTiev9es1LZu577vua\nicw155z7XEdprRFCCCEex8neAQghhMjdJFEIIYRIlyQKIYQQ6ZJEIYQQIl2SKIQQQqRLEoUQQoh0\nSaIQeY5Saq5SapK948hpSqkpSqlvn/AYT+V7J9IniULkCKVUoFJql1LqplLqulJqp1LqxWw4bn+l\n1I6Uj2mth2mt/+9Jj52FWKz+oFZKbVVK3VBK5bd1XJlhr/dO5G6SKITNKaWKAD8Bs4BiQFngPSDR\nnnHZi1KqAtAY0EBHuwYjhBUkUYicUAVAa71Ma23QWt/TWv+mtT78YAel1ECl1Anzt+yNSqnyKbZp\npdQwpdQZpVScUuoLZVIdmAs0VErdUUrFmfdfpJR63/xzsFIqWik1Vil1RSn1j1Kqs1KqrVLqtLl1\n868U53JSSr2jlDqnlLqmlFqhlCpm3lbBHEs/pVSkUuqqUmqCeVtr4F9AN3Msh9J5P/oCe4BFQL+U\nG8yxf6GU+lkpdVsptVcpVTHF9s+VUlFKqVtKqYNKqcZpncD8/DdSPXZYKdXF/N79x/x+3FJKHVFK\n1UzjvXtOKfWT+T2/rpT6UyklnxlPIfmli5xwGjAopb5RSrVRSj2bcqNSqhOmD9mXgBLAn8CyVMdo\nD7wI1AZeBVpprU8Aw4DdWuvCWutnHnP+5wE3TC2Zd4EFQG+gPqZv9pOUUl7mfd8AOgNNgDLADeCL\nVMcLBKoCzYB3lVLVtda/Ah8C35tjqZPO+9EXWGq+tVJKlUq1vTumFtezwFnggxTb9gN1MbXMvgNW\nKqXc0jjHN+bXCIBSqo759f8MtASCMCXwopjez2tpHGM0EI3pd1IK0+9Iav48hSRRCJvTWt/C9OGq\nMX1Ixyql1qX4gBwGfKS1PqG1Tsb0gVs3ZasC+FhrHae1jgS2YPqwtFYS8IHWOglYDjwHfK61vq21\nPgYcBx58sA8DJmito7XWicAU4BWllEuK471nbhUdAg6leG6GlFKBQHlghdb6IHAO6JlqtzVa633m\n92Jpyteqtf5Wa31Na52stf4MyI8paaW2DqiilKpsvt8HUxK7b34/3IFqgDK/7/+kcYwkoDRQXmud\npLX+U0txuKeSJAqRI8wfRv211h5ATUzf1meYN5cHPjd3ccQB1wGF6RvwA5dS/HwXKJyJ01/TWhvM\nP98z/3s5xfZ7KY5XHliTIpYTgAHTN+rsiKUf8JvW+qr5/nek6n5K7/hKqTHmLrqb5viKYkp8D9Fa\nJwDfA73N3UU9gCXmbZuB/2JqKV1RSs03jyOl9m9MLZrflFIRSql3MvE6RR4iiULkOK31SUz98zXN\nD0UBQ7XWz6S4FdBa77LmcNkcXhTQJlUsblrrmCeNRSlVAFM3TxOl1CWl1CXgLaCOuWsoXebxiLHm\nYzxr7mq7iSmppuUboBemLrK7WuvdlkC1nqm1rg94Y+qCevuRF2NqcY3WWr+AadB9lFKqWUZxirxH\nEoWwOaVUNaXUaKWUh/m+J6ZvuHvMu8wFxiulapi3F1VKdbXy8JcBD6VUvmwKdy7wwYNuL6VUCfMY\nirWxVEhnwLczptaJN6bupLpAdUxjMn2tOL47kAzEAi5KqXeBtFoCAJgTgxH4DHNrAkAp9aJSyl8p\n5QrEAwnm/R6ilGqvlKqklFKYEpIhrf1E3ieJQuSE24A/sFcpFY8pQRzFNFiK1noN8AmwXCl1y7yt\njZXH3gwcAy4ppa5mtLMVPsfUv/+bUuq2OVZ/K5+70vzvNaXUX2ls7wd8rbWO1FpfenDD1A3UK9U4\nSFo2Ar9iujjgAqYP+KgMnrMYqAWknN9RBNNY0Q3zca5h6mZKrTKwCbgD7AZma623ZHA+kQcpGZsS\nIu9SSvUFQrXWgfaORTguaVEIkUcppQoCw4H59o5FODabJQql1FfmCT1HH7NdKaVmKqXOmicC1bNV\nLEI8bZRSrTCNZVzGdGWVEFlmyxbFIqB1OtvbYOoDrQyEAnNsGIsQTxWt9UatdSGtdSfzfAwhssxm\niUJrvR3T9fCP0wlYrE32AM8opUrbKh4hhBBZk9FVFrZUloev2Ig2P/bIDFGlVCimVgeFChWqX61a\ntRwJUGTg/l3unz+HMVnh5GK/iyISlMLw2KkEQjzdYhOSuJtswKC5qrUukZVj2DNRWE1rPR/zgJyv\nr68+cOCAnSMSAFzYzYXevaBEVcp//aXdwghYPQyjNrCgyWS7xSBEbvLgalalFN8vXc316zeYPfPL\nC1k9nj0TRQzgmeK+h/kx4WicXaFo2Yz3s5Fk5QoqHzUrWTvdQYi8KyYmhtdee41u3brRq1cvak42\n/V3Mnpn1L3P2vDx2HdDXfPVTA+DmYwqTCSGEyIDWmgULFuDt7c2mTZu4c+dOth3bZi0KpdQyIBh4\nTikVDUwGXAG01nOBDUBbTEXH7gIDbBWLEELkZefOnWPIkCFs2bKFkJAQFixYQMWKFTN+opVslii0\n1j0y2K6BEbY6vxBCPC2OHDnCwYMHmT9/PoMHD8ZUniv7OMRgthBCiIcdPXqUv/76i759+9K5c2ci\nIiIoXry4Tc4lJTyEEMKB3L9/nylTplCvXj0mTJhAQkICgM2SBEiiEEIIh7F3717q1avHe++9R7du\n3QgLC8PNLa2VcLOXdD0JIYQDiImJoXHjxpQqVYqffvqJdu3a5di5pUUhhBC52OnTpwEoW7Ys33//\nPceOHcvRJAGSKIQQIleKi4sjNDSUatWqsX37dgC6dOlCkSKPXdTQZqTrSQghcpl169bx2muvcenS\nJd5++21efPFFu8YjiUIIIXKRwYMHs3DhQmrVqsXatWvx9fW1d0iSKIQQwt5SFvHz9fWlfPnyjBs3\njnz58tk5MhNJFA5q3ftfoDdttGsMBbjNc7fy8bfbeTouDLFbHAanOArxgt3OL8STiIqKYtiwYXTv\n3p0+ffowbNgwe4f0CEkUDir/tk0UvxbNzTIV7BaDk9FIRCnFsepFKZmvit3iAGhZoYVdzy9EZhmN\nRubNm8e4ceMwGAx06dLF3iE9liQKB3a5pCftNv5gt/P/dWgxb4b/m/k1ezO5fu77FiREbnXmzBkG\nDx7M9u3bad68OfPnz8fLy8veYT2WJAohhMhhx48f5/Dhw3z11Vf0798/24v4ZTdJFEIIkQMOHTpE\neHg4/fr1o1OnTkRERPDss8/aOyyryIQ7IYSwocTERCZNmoSvry+TJk2yFPFzlCQBkiiEEMJmdu/e\njY+PD++//z49e/bMsSJ+2U26noQQwgZiYmJo0qQJzz//PBs2bKBNmzb2DinLpEUhhBDZ6MSJE4Cp\niN+KFSs4duyYQycJkEQhhBDZ4saNGwwcOBBvb2/+/PNPADp37oy7u7udI3ty0vUkhBBPaM2aNQwf\nPpzY2FjGjx9v9yJ+2U0ShRBCPIGBAwfy9ddfU7duXX7++Wfq1atn75CynSQKIYTIpJRF/Bo0aEDl\nypUZM2YMrq6udo7MNiRRCCFEJly4cIGhQ4fSs2dP+vbtS2hoqL1DsjkZzBZCCCsYjUa++OILatas\nyY4dO0hKSrJ3SDlGWhSZZNRG/lrwCc6/77JrHCWuRBL93HP89vdvdoshIu6U3c4tRE46deoUgwcP\nZseOHbRs2ZJ58+ZRoUIFe4eVYyRRZEKyMZkpu6ZQd9VqKlyGv0vZMZhSsMP7En9sG23HIEyKuBSw\ndwhC2NSpU6c4duwYixYtom/fvrm+iF92k0RhpURDImO3jWVz1GbaFipDIe9i1Jn9od3imbDmCLcT\nk/mho4/dYuDiIQquCaVsYDn7xSCEjYSFhREeHs6AAQPo2LEjERERPPPMM/YOyy4kUVghPimeNze/\nyb5L+xjvN56ym34FoPyzle0WUwFukGhMorIdY+DWVUg22O/8QthAQkICU6dO5dNPP6Vs2bL06NED\nNze3pzZJgAxmZ+hGwg0GbRzEwcsH+ajxR/Ss3tPeIQkhbGTnzp3UrVuXjz76iL59+xIeHu6QRfyy\nm7Qo0nEp/hKhv4dy8c5FPg/5nCaeTewdkhDCRmJiYggJCaFs2bJs3LiRli1b2jukXEMSxWP8ffNv\nQn8P5fb928xtPhff533tHZIQwgaOHz+Ot7c3ZcuWZfXq1YSEhFC4cGF7h5WrSNdTGo5fO06/X/uR\naEjkq1ZfSZIQIg+6fv06/fv3p0aNGmzfvh2ADh06SJJIg7QoUjlw6QBvbH4D93zuzG8xnwpFK9g7\nJCFENlu9ejUjRozg2rVrTJgwAT8/P3uHlKtJokhhe/R2Rm0dRZnCZZjfYj7PF3re3iEJIbJZ//79\n+eabb6hXrx6//vordevWtXdIuZ4kCrOfIn5i0o5JVC1WlTnN5/Csm+OsZyuESF/KIn6NGjWievXq\njB49GhcX+Qi0hk3HKJRSrZVSp5RSZ5VS76SxvZxSaotSKkwpdVgp1daW8TzOdye+Y/yf46lXqh4L\nWy2UJCFEHnL+/HlatmzJ4sWLAQgNDWXcuHGSJDLBZolCKeUMfAG0AbyBHkop71S7TQRWaK19gO7A\nbFvFkxatNXMOzeGjfR8R4hnC7OazKeRaKCdDEELYiMFgYObMmdSsWZM9e/ZYWhUi82yZUv2As1rr\nCACl1HKgE3A8xT4aKGL+uShw0YbxPMSojXy6/1OWnlhKp4qdmNJoCi5O8g1DiLzgxIkTDBo0iN27\nd9OmTRvmzp1LuXJSaiarbPnJWBaISnE/GvBPtc8U4Del1BtAIaB5WgdSSoUCoUC2/LKTjElM3jmZ\n9RHr6ePdhzG+Y3BScqWwEHnF2bNnOXXqFEuWLKFXr15PXRG/7GbvT8cewCKttQfQFlii1KOf2Frr\n+VprX621b4kSJZ7ohAnJCYzaMor1Eet5w+cN3vZ9W5KEEHnAwYMH+eqrrwDTfIjz58/Tu3dvSRLZ\nwJYtihjAM8V9D/NjKQ0CWgNorXcrpdyA54Arjzvo/fPnudCnb5YCMmgDZ26coUnSHfq6l6Pkph1E\nsiNTx0g2am4dPUZMcU9en5W552an81fjqVhSJgYJce/ePd577z2mTZuGp6cnPXv2xM3NjSJFimT8\nZGEVWyaK/UBlpZQXpgTRHUhdUS8SaAYsUkpVB9yA2PQOakxIyFIwScZkTt84zb3ku7xQ9AWKuxXL\n0nESkw2cLlyaM1X8KeGeP0vHyA4l3PPT0tueC2IIYX/bt29n8ODBnDlzhkGDBjFt2jQp4mcDNksU\nWutkpdTrwEbAGfhKa31MKTUVOKC1XgeMBhYopd7CNLDdX2dwaYKTmxvllyzOVCz/3PmH138P5VK8\n4rPgudTzCMrSawI4GnOTcbN2sKCvLxPlg1oIu4mJiaFZs2Z4enqyadMmmjVrZu+Q8iybXuajtd4A\nbEj12Lspfj4OBNgyhoibEYT+FsrdpLvMazGPeqXq2fJ0QggbO3LkCLVq1aJs2bKsWbOGkJAQChWS\ny9ptKU+P4h67doz+v/QnyZjEV62/kiQhhAO7evUqffr0oXbt2pYifu3bt5ckkQPybKLYf2k/gzYO\nooBLARa3WUy1YtXsHZIQIgu01qxYsQJvb2+WL1/O5MmT8fdPfaW9sKU8OcNsS+QWxmwbg6e7J/Na\nzKNUIRlLEMJR9evXjyVLluDr68sff/xBrVq17B3SUyfPJYr159YzaeckvIt7M7vZbJ5xe3rXuRXC\nUaUs4tekSRNq167N//t//0/qM9lJnup6+vb4t/xrx7/wfd6XBS0XSJIQwgFFRETQvHlzFi1aBMCg\nQYMYM2aMJAk7yhOJQmvNF+Ff8Mn+T2herjmzm0lxPyEcjcFgYMaMGdSqVYv9+/fj5JQnPp7yBIdP\n0UZt5ON9H7Ps5DK6VOrCuw3fleJ+QjiY48ePM3DgQPbu3Uu7du2YO3cuHh4e9g5LmDn0J2qSMYlJ\nOyfxc8TP9K/Rn1H1R0ldFyEc0Pnz5zl37hzfffcd3bt3l7/jXMZhE0VCcgKjt41me/R2RtYbyaCa\ng+Q/lxAOZP/+/YSHhzNkyBDatWtHREQE7u7u9g5LpMEhOwFv37/N0N+H8mf0n0xqMInBtQZLkhDC\nQdy9e5cxY8bQoEEDPvroIxLM9dskSeReDpcoNJqBGwdy+OphPg36lFervmrvkIQQVtq6dSu1a9fm\ns88+Y8iQIYSFhUkRPwfgcF1PCckJ/H3zb2Y1nUVg2UC7xFBHnaXY5UTI/5SvrX3leMb7CGEWHR1N\nixYtKF++PJs3byYkJMTeIQkrOVyi0Gjmt5yPT0kfu5zf5cJ21uZ/F7bZ5fS5U36p+y8e79ChQ9Sp\nUwcPDw/Wrl1LcHAwBQsWtHdYIhOUoy04XrloQX3m5l27nT/yv+0pEHsYp1cXU7yw/dajyDXyFYLn\na4GMEYlUYmNjGTlyJMuWLWPr1q00adLE3iE91ZRSB7XWvll5rsO1KOzq2jnKXf2TL51fZXCNYHtH\nI0SupLVm+fLlvPnmm9y8eZP33nuPhg0b2jss8QSsShRKqXxAOa31WRvHk7vtW0ASLpwpJwPoQjxO\nnz59WLp0Kf7+/ixcuJAaNWrYOyTxhDK86kkp1Q44Avxuvl9XKbXG1oHlOgm3MIYtYb2hAS94vWDv\naITIVYxGo6WQX0hICNOnT2fnzp2SJPIIay6PnQr4A3EAWutwoJItg8qVDi3D6f4dFiW3oq6nFBsU\n4oGzZ8/SrFkzvv76a8BUxO+tt97C2dnZzpGJ7GJNokjSWselesyxRsCflNEIe+dxsXBNjqlK1PIo\nau+IhLC75ORkpk2bRq1atQgLCyNfvnz2DknYiDVjFCeUUq8CTkopL+BNYI9tw8plzm6C6+dYU3w8\nVUq5UzCfXAMgnm5Hjx5lwIABHDhwgE6dOjF79mzKlClj77CEjVjTongdqA8YgR+ARGCkLYPKdfbO\nRbuX5surtfApJ91OQkRGRnLhwgWWL1/OmjVrJEnkcdZ8NW6ltR4HjHvwgFLqJUxJI++LPQ3n/uC6\n31hubEfGJ8RTa+/evRw6dIjQ0FDatm1LREQEhQsXtndYIgdY06KYmMZjE7I7kFxr3zxwzs+Oou0B\n8JFEIZ4y8fHxjBo1ioYNG/Lpp5+SmJgIIEniKfLYFoVSqhXQGiirlJqeYlMRTN1Qed+9OAhfBrVe\nYd8VJ9zzu1CxhPxxiKfH5s2bGTJkCBEREbz22mt8/PHH5M8vFQmeNul1PV0BjgIJwLEUj98G3rFl\nULlG2LeQFA/+QwlfGUdtz6I4OUmpCvF0iI6OplWrVnh5ebFt2zaCgoLsHZKwk8cmCq11GBCmlFqq\ntU7IwZhyB6MB9s2Hco24V7wmJy9tZFgTmWgn8r6wsDB8fHzw8PBg/fr1NGnShAIFCtg7LGFH1oxR\nlFVKLVdKHVZKnX5ws3lk9nZ6I8RdAP+hHIm5icGo8fF8ysuKizzt8uXLdOvWjXr16rFtm6k8cuvW\nrSVJCKsSxSLga0ABbYAVwHIbxpQ77J0DRTygWnvCo24AUFcujRV5kNaab7/9Fm9vb3788Ufef/99\nGjVqZO+wRC5iTaIoqLXeCKC1Pqe1ngjk7RVHLh+H89vBbzA4uxAeFYfHswV4TsqKizyoZ8+e9OnT\nh6pVqxIeHs6ECRNwdXW1d1giF7FmHkWiUsoJOKeUGgbEACVtG5ad7ZsHLgWgXj8AwiPjqFdeup1E\n3mE0GlFKoZSiZcuWNGzYkBEjRkh9JpEma1oUbwGFMJXuCACGAANtGZRd3b0Oh76H2q9CwWJcvpXA\nxZsJMtFO5BmnT58mJCSEr776CoABAwbw5ptvSpIQj5Vhi0Jrvdf8422gD4BSqqwtg7KrvxZD8j3w\nHwpAWKSpHqJPOWlRCMeWnJzM9OnTmTx5Mm5ubjJILayWbotCKfWiUqqzUuo58/0aSqnFwN70nuew\nDMmwbwF4BUEpUx398Kg4XJ0VNcrIutDCcR0+fJgGDRowbtw42rRpw/Hjx+nZs6e9wxIO4rGJQin1\nEbAU6AX8qpSaAGwBDgFVcia8HHbqZ7gVDf7DLA+FR92geukiuLlKs1w4rujoaKKioli5ciWrV6+m\ndOnS9g5JOJD0up46AXW01veUUsWAi+b7p6w9uFKqNfA54Ax8qbX+OI19XgWmYFrj4pDW2n5fc/bO\ng2fKQ5XWABiMmiPRN3m5vofdQhIiq3bt2sXhw4cZNmyYpYhfoUKF7B2WcEDpdT0laK3vAWitrwMn\nM5kknIEvMM298AZ6KKW8U+1TGRgPBGitawD/L5PxZ59/DsOFneAXCk6m1sOZK7eJv2+QgWzhUO7c\nucPIkSMJDAzks88+sxTxkyQhsiq9FsULSqkHpcQVUCHFfbTWL2VwbD/grNY6AkAptRxTK+V4in2G\nAF9orW+Yj3klk/Fnn73zwLUg+PS2PBQuA9nCwfz222+EhoYSGRnJiBEj+PDDD6WIn3hi6SWKl1Pd\n/28mj10WiEpxPxrT2tspVQFQSu3E1D01RWv9a+oDKaVCgVCAiu42uFIj/iocWWlKEgX+13oIi4zj\nmYKuVCheMPvPKUQ2i4qKol27dlSsWJHt27cTGBho75BEHpFeUcA/cuj8lYFgwAPYrpSqlXqNbq31\nfGA+QOWiBbN/ve6DX4Mh0XJJ7APhUXHU8XgGpaRirMi9Dh48SP369fH09GTDhg00btwYNzc3e4cl\n8hBrJtxlVQzgmeK+h/mxlKKBdVrrJK31eeA0psSRcwxJsH8hVGwKJapaHr6TmMzpK7dlfELkWpcu\nXaJr1674+vpaivi1aNFCkoTIdrZMFPuBykopL6VUPqA7sC7VPj9iak1gnqtRBYiwYUyPOr4Wbv/z\n0CWxAIej49BaCgGK3EdrzTfffIO3tzfr16/nww8/lCJ+wqasqfUEgFIqv9Y60dr9tdbJSqnXgY2Y\nxh++0lofU0pNBQ5ordeZt7VUSh0HDMDbWutrmXsJT2jvPCj2AlRq8dDD4VGm3q+6HpIoRO7SvXt3\nVqxYQUBAAF9++SXVqlWzd0gij8swUSil/ICFQFGgnFKqDjBYa/1GRs/VWm8ANqR67N0UP2tglPmW\n82IOQvQ+aP0JOD3cuAqLjMPruUI8WyifXUITIqWURfzatm1L48aNGT58OE5OtuwUEMLEmv9lM4H2\nwDUArfUh8kqZ8b3zIJ871H14jp/WmvCoOBmfELnCyZMnCQoKYuHChQD069eP119/XZKEyDHW/E9z\n0lpfSPWYwRbB5Kjbl+HoD+DTC9weruN08WYCsbcTJVEIu0pKSuLDDz+kTp06HD9+nMKFC9s7JPGU\nsmaMIsrc/aTNs63fwHR1kmM7+DUYk00zsVN5MNFOEoWwl/DwcAYMGEB4eDivvPIKs2bN4vnnn7d3\nWOIpZU2ieA1T91M54DKwyfyY40pONF0SW7klFK/4yObwqBvkc3GiemmpGCvs49KlS1y6dInVq1fz\n0ksZFUEQwrasSRTJWuvuNo8kJx37EeKvPDLB7oHwqDhqlilCPhfpAxY5Z8eOHRw+fJjhw4fTunVr\nzp07R8GCUhVA2J81n4T7lVIblFL9lFLuNo/I1rSGvXPguSqmSXapJBmMHI6+SV1Pqe8kcsbt27d5\n/fXXady4MTNmzLAU8ZMkIXKLDBOF1roi8D5QHziilPpRKeW4LYzo/XAxzNSaSKM0x6lLt0lMNspE\nO5EjNm7cSM2aNZk9ezYjR47kr7/+kiJ+Itexqm9Fa71La/0mUA+4hWlBI8e0dy7kLwq10851YeaJ\ndj4ykC1sLCoqivbt21OwYEF27NjBjBkz5MomkStlmCiUUoWVUr2UUuuBfUAs4Jj1Am5dNJXsqNcH\n8qf9BxkeGUfxQvnweFbWExbZT2vNvn37APD09OSXX34hLCxMSnCIXM2aFsVRoAHwqda6ktZ6tNba\nMdfM3r8QjAbwG/LYXcKjblDXUyrGiuz3zz//8PLLL+Pv728p4te8eXMp4idyPWuuenpBa220eSS2\nlpRgmjtRtS08WyHNXW7eTeJcbDxdfMrmbGwiT9Nas2jRIkaNGkVCQgKffPIJAQEB9g5LCKs9NlEo\npT7TWo8GViulHlkDwooV7nKXo6vg7jVoMOyxuxyKfjDRTq54Etnn1VdfZdWqVTRu3Jgvv/ySKlWq\n2DskITIlvRbF9+Z/M7uyXe6jtWkQu6Q3VGj82N3Co+JQCmp7Fs3B4EReZDAYUErh5OREhw4daNq0\nKUOHDpX6TMIhPfZ/rdZ6n/nH6lrrP1LegOo5E142idwNl4489pLYB8Kj4qhYojBF3FxzMDiR15w4\ncYLGjRtbivj17duX1157TZKEcFjW/M8dmMZjg7I7EJvaMwcKPAu1Xn3sLlIxVjyppKQk3n//ferW\nrcupU6coWlRapiJvSG+MohumVem8lFI/pNjkDsSl/axcKC4KTv4Ejd6EfI+f6Rp5/S7X4+/jIxPt\nRBaEhYXRv39/Dh8+TLdu3Zg5cyYlS5a0d1hCZIv0xij2YVqDwgP4IsXjt4EwWwaVrfYvABS8ODjd\n3Swr2kmLQmTB5cuXuXr1Kj/++COdOnWydzhCZKvHJgqt9XngPKZqsY7p/l04+A1Ubw/PeKa7a1hk\nHAVcnalayvHLWYmcsX37do4cOcKIESNo3bo1Z8+epUABmagp8p7HjlEopbaZ/72hlLqe4nZDKXU9\n50J8AkdWQEIc+D/+ktgHwqPiqFW2KC7OMuAo0nfr1i2GDx9OkyZNmDlzpqWInyQJkVel96n4YLnT\n54ASKW4P7uduWsOeufB8bSjXMN1dE5MNHL94SwoBigxt2LCBGjVqMG/ePEaNGiVF/MRTIb3LYx/M\nxvYEnLXWBqAhMBQolAOxPZnz2yH2hKk1kUE5jhP/3Oa+wSjjEyJdUVFRdOrUiaJFi7Jr1y4+++wz\nChXK/X9p41cbAAAgAElEQVQKQjwpa/pZfsS0DGpFYDGmORTf2TSq7LB3HhR8Dmq+nOGuYZE3AOSK\nJ/EIrTV79uwBTEX8fvvtN/766y/8/f3tHJkQOceaRGHUWicBLwEztNZvALm7GNL183BqA/gOANeM\nC66FR8VRqkh+SheVPmbxPxcvXqRz5840bNjQUsQvJCSEfPny2TkyIXKWNYkiWSnVFegD/GR+LHdP\nXd7/JTg5g29acwUfJRPtREpaa7788ku8vb357bffmDZtmhTxE081a2dmh2AqMx6hlPICltk2rCeQ\neAf+WgLenaBImQx3vx5/nwvX7kohQGHxyiuvMGTIEOrWrcuRI0cYPXo0Li7WFFoWIm/K8H+/1vqo\nUupNoJJSqhpwVmv9ge1Dy6JDyyDxJvi/Zt3uMtFO8HARv86dO9OyZUuGDBki9ZmEwLoV7hoDZ4GF\nwFfAaaVU7myHG42wbz6UqQcevlY9JSzyBk4KantIXZ6n1dGjRwkICLAU8evTp49UehUiBWv+Ev4D\ntNVaB2itGwHtgM9tG1YWRWyGq6etuiT2gbCoOKqUcqdQfulaeNrcv3+f9957j3r16nHu3DmefVa6\nH4VIizWfjvm01scf3NFan1BK5c7LPvbOg8KloEYXq3Y3GjWHouJoV7u0jQMTuc3Bgwfp378/R48e\npWfPnsyYMYMSJXL/PFIh7MGaRPGXUmou8K35fi9yY1HAa+fgzG8QPB5crMtj56/FcyshWcYnnkLX\nrl0jLi6O9evX0759e3uHI0SuZk2iGAa8CYw13/8TmGWziLJq7zxwcrX6kliA8EhZ+vRpsmXLFo4c\nOcKbb75Jy5YtOXPmDG5uGc+zEeJpl+4YhVKqFtAaWKO17mi+/VtrnZAz4Vkp4RaELzXNwi5s/RoA\n4VFxFMrnTKWShW0YnLC3mzdvMnToUJo2bcqcOXMsRfwkSQhhnfSqx/4LU/mOXsDvSinrv6rntPDv\n4P4d01KnmRAWdYM6ns/g7GTdwLdwPOvXr8fb25svv/ySMWPGcPDgQSniJ0Qmpdf11AuorbWOV0qV\nADZgujw2dzEaYd888PSHsvWsflpCkoGT/9wmNOgFGwYn7CkqKoqXX36ZatWq8eOPP/Liiy/aOyQh\nHFJ6XU+JWut4AK11bAb72s/Z3+F6RKZbE0djbpJs1DKQncdordm1axfwvyJ+Bw4ckCQhxBNI78P/\nBaXUD+bbGqBiivs/pPM8C6VUa6XUKaXUWaXUO+ns97JSSiulrJsll9KeOeBeBqp3zNTTLEufSsXY\nPCM6OpqOHTsSEBBgKeIXHBwsRfyEeELpdT2lrs/938wcWCnljGmt7RZANLBfKbUu5ZwM837uwEhg\nb2aOD8CVkxCxBZpOAufM1SkMi4qj7DMFKOkuA5qOzmg0smDBAt5++22Sk5OZPn06gYGB9g5LiDwj\nvTWz/3jCY/thqgsVAaCUWg50Ao6n2u//gE+AtzN9hn3zwTk/1O+f6aeGR8ZJayKPePnll/nxxx9p\n2rQpCxYs4IUXZNxJiOxky3GHskBUivvRpFrHQilVD/DUWv+c3oGUUqFKqQNKqQNamx+8d8NUALBW\nVyj0XKYCu3I7gZi4e/jI+ITDSk5Oxmg0LcL48ssvs2DBAjZt2iRJQggbsNsAtVLKCZgOjM5oX631\nfK21r9ba11LCKexbSLqb6UFsSDnRThKFIzp8+DANGzZkwYIFAPTu3ZvBgwejrKzvJYTIHKsThVIq\nsxefx2Bab/sBD/NjD7gDNYGtSqm/gQbAOqsGtI0GU7dT+QAoXTuTYZkGsl2cFDXLSsVYR5KYmMjk\nyZOpX78+Fy5ckNpMQuQQa8qM+ymljgBnzPfrKKWsKeGxH6islPIyFxHsDqx7sFFrfVNr/ZzWuoLW\nugKwB+iotT6Q4ZFP/QJxkaYqsVkQHhVHtdLuuLk6Z+n5Iuft37+fevXqMXXqVHr06MGJEyd46aWX\n7B2WEE8Fa1oUM4H2wDUArfUhTCvepUtrnQy8DmwETgArtNbHlFJTlVKZu5Y1tb1zoagnVG2b6aca\njJrD0Tel28nB3Lhxgzt37rBhwwYWL15M8eLF7R2SEE8Na4oCOmmtL6Tq/zVYc3Ct9QZMM7pTPvbu\nY/YNtuaYAPz9JzR/D5wzv4bE2St3uJOYjI8UAsz1Nm/ezJEjRxg5ciQtW7bk9OnTUn5DCDuwpkUR\npZTyA7RSylkp9f+A0zaO6/G0BpcCUK9vlp4eHnUDkIl2uVlcXBxDhgyhWbNmzJs3z1LET5KEEPZh\nTaJ4DRgFlAMuYxp0tm5Balup0w0KFsvSU8Oj4iji5oJX8ULZHJTIDmvXrsXb25uvvvqKsWPHShE/\nIXKBDPtutNZXMA1E5xIa/DJ/SewDYZFx1PF8BiepGJvrREZG0rVrV6pXr866devw9c18RRchRPbL\nMFEopRYAOvXjWutQm0SUIQWlvLP0zPjEZE5fvk1L71LZHJPIKq01O3bsoHHjxpQrV45NmzbRoEED\nqc8kRC5iTdfTJuAP820nUBJItGVQtnI4+iZGDT7lZCA7N4iMjKRdu3YEBQVZivgFBQVJkhAil7Gm\n6+n7lPeVUkuA320WkQ09qBhbRy6NtSuj0cjcuXMZN24cWmtmzpwpRfyEyMUyf30peAHlszuQnBAe\ndYPyxQtSrJB8Y7Wnl156ibVr19KiRQvmz59PhQoV7B2SECId1oxR3OB/YxROwHXgsWtL5GbhUXE0\neEEmatlDcnIyTk5OODk50a1bNzp16kT//v2lPpMQDiDdRKFMf8V1+F+NJqPW+pGBbUfwz817XL6V\nKDOy7eDQoUMMHDiQIUOGMGzYMHr06GHvkIQQmZDuYLY5KazRWhvMN4dMEiAVY+0hISGBiRMn4uvr\nS3R0NM8//7y9QxJCZIE1YxT7lFI+Wuswm0djQ2FRceRzdsK7TBF7h/JU2LdvH/369ePkyZP069eP\n6dOnU6xY1iZJCiHs67GJQinlYi7sFwgMUUqdA+IBhamxUS+HYswW4ZFxeJcpQn4XqRibE27dusW9\ne/f49ddfadWqlb3DEUI8gfRaFPuAekDnHIrFZpINRo7E3KTbi54Z7yyy7LfffuPYsWO89dZbNG/e\nnFOnTkn5DSHygPTGKBSA1vpcWrccii9bnLp8m3tJBnykEKBN3LhxgwEDBtCqVSsWLlwoRfyEyGPS\na1GUUEqNetxGrfV0G8RjEw8m2slAdvb74YcfGDFiBLGxsYwfP553331XEoQQeUx6icIZKIy5ZeHI\nwiLjKFYoH+WKFbR3KHlKZGQk3bt3p2bNmmzYsAEfHx97hySEsIH0EsU/WuupORaJDYVHxVHX8xmZ\n3JUNtNZs376dJk2aUK5cOTZv3oy/vz+urq72Dk0IYSMZjlE4ulsJSZyLvSPdTtngwoULtGnThuDg\nYEsRv8DAQEkSQuRx6SWKZjkWhQ0djrqJ1jI+8SSMRiP//e9/qVGjBjt27GDWrFk0btzY3mEJIXLI\nY7uetNbXczIQW3mw9KlUjM26zp07s379elq1asW8efMoX94ha0IKIbIoK9VjHUp4VBwVSxSiaAHp\nHsmMpKQknJ2dcXJyokePHrzyyiv06dNHxnmEeApZs3CRw9JaExYZR11PWagoM/766y/8/PyYO3cu\nAD169KBv376SJIR4SuXpFkX0jXtci79PXZloZ5V79+4xdepU/v3vf1OiRAk8PXPXTPakpCSio6NJ\nSEiwdyhC5Gpubm54eHhk24UmeTpRhJkn2vnI+ESG9uzZQ79+/Th9+jQDBw5k2rRpPPts7mqJRUdH\n4+7uToUKFaR1I8RjaK25du0a0dHReHl5Zcsx83SiCI+MI7+LE1Wfd7d3KLlefHw8SUlJ/P777zRv\n3tze4aQpISFBkoQQGVBKUbx4cWJjY7PtmHk7UUTdoFbZorg65+mhmCz79ddfOXbsGKNHj6ZZs2ac\nPHmSfPly9zKxkiSEyFh2/53k2U/Q+8lGjl68JYUA03Dt2jX69etHmzZt+Oabb7h//z5Ark8SOWXr\n1q2UL1+ekJAQWrRowbVr1wC4e/cuAwYMIDg4mJCQEA4ePGh5zrJlywgKCqJx48b07t3b8p7mpFWr\nVrFmzZocP29KS5cupVGjRrRv355bt249tC0yMpKmTZsSFBTEihUrANMSuX369CEwMJCPP/4YgLVr\n11q2WyM8PJyFCxcC8NVXX1keDwwMTHP/U6dOERwcTIMGDShVqhTBwcGMHz8+U6/zgUqVKhEcHExw\ncDCnTp3K0jEy0rt3b6Kjo21ybKtprR3qVsndTVsjPPKGLj/uJ/3ToYtW7f80MBqNeuXKlbpkyZLa\nxcVFT5w4USckJNg7LKsdP348R86zZcsWPWHCBK211kuWLNGzZs3SWms9duxY/d1332mttb58+bJu\n1KiRTkxM1EePHtXt27fXiYmJWmut9+7dq+/evZvl8xsMhiw9r3Pnzjo5Odlmx8/I/fv3dWBgoE5K\nStLLly/Xn3766UPbhw8frnft2qWTk5N1ixYtdFJSkl69erV+//33tdZat2vXTv/zzz/aYDDoLl26\nZCmGgICANH9Oy/nz53WvXr2ydB5rz5EVqX8/vXr10lFRUZk+Tuq/F+CAzuLnbp5tUVgqxkqLwiIy\nMpKePXvi6enJgQMH+L//+z+p9JqBuLg4y887d+60rPddsmRJmjZtyp49e1i1ahUjR460tMj8/Pwo\nUKCA5XlGo5HBgwfTpEkT2rRpAzz8bTc4ONjy79ixY+nbty8dOnTg5s2bAIwePZp9+/Zx9uxZWrZs\nSZMmTXj//fcfivP69esYDAacnZ1JSkqiWbNmBAUF8fLLL2MwGPj7778JCQnhlVdeYdGiRezdu5fg\n4GACAgL4+uuvAfjoo49o0qQJ/v7+hIVlfkHLM2fOUKtWLVxcXGjevDm7d+9+aHtERAS1a9fG2dmZ\nUqVKcebMGfbs2UOLFi0ACAkJYd++fTg5OeHq6mppyQHcvn2bV1991fLerVmzhvPnzzN8+HC2bt3K\nxIkTmT9/PkeOHCE4OJgjR45gMBgYMWIEdevW5ddff80w/r/++ouAgAAaNGjAsmXLANO3+REjRtC4\ncWOmTn209F1sbCxBQUG89tprlvL6D3Tt2pU7d+4wZ84cXnnlFQDatWuH0Whk0aJFNGjQgMDAQI4c\nOQKAv78/Q4cOZezYsZw7dw4/Pz+6dOnCxYsXAVOL0c/Pj6ZNm7Jx40arfifZJc+OUYRHxVHCPT9l\nirrZOxS70lqzZcsWmjZtSvny5dm6dSt+fn64uDjur/699cc4fvFWxjumw7tMESZ3qPHY7UuWLGH9\n+vUYjUZ27NgBPNrv6+HhwT///MM///xD6dKlH3ustWvXUrJkSb788kuMRmO6cXXp0oWGDRsyf/58\n1q1bR58+fTh48CCfffYZ3bp1Y+HChXh6etKjRw+io6Px8PAA4OzZs5QrVw4AFxcXfvrpJwoUKMDE\niRPZvHkzlStX5sqVK2zatAlnZ2datWrFunXrcHd3p0WLFvTq1YuRI0cyfvx4zp49y+TJk1m6dKkl\nrsjISPr27ftQrGXLln1on7i4OIoUMS01XLRo0YeSLEDVqlXZtm0bwcHB7Nmzh7i4uMc+x8vLi1On\nTtGoUSMA3N3duX37NomJiRQrVozdu3cTHx9v2Q4QGhrK4sWL2bp1K2DqYp08eTJJSUm8/vrrtG7d\nOt33ftKkSSxfvpxSpUoREBBgSUxBQUF88cUXtGnThkuXLj209vvu3bspVqwYU6dOZeHChQwfPtyy\nzc/Pj71793Lw4EFcXV1JTEzE1dUVg8HA3Llz2bFjBxcuXGDUqFGsXbuWK1euMHnyZMqUKUNoaCiz\nZs2ifv361Khh+n+6evVqfvjhBzw8PDA1EHJOnm5RPO0VY8+fP0/Lli1p1qyZpYhfo0aNHDpJ5JQ+\nffoQFhaGr68vkZGRAI/8cUZHR1O6dGlKly5t+daXltOnT1s+0JycHv6TS33M+vXrA6aE8eOPP3Lw\n4EHq1TOtOnzq1Cn69OlDcHAwJ06cICYmJs3zxcfHM2jQIJo0acKqVasssdWpUwdnZ9NSwIcOHaJj\nx46EhIRw6dIlYmNjWbJkCUFBQQwePPiR11OuXDm2bt360C1lkgDTB/2DcYlbt27xzDMPt+bHjx/P\n/Pnz6dq1K9WqVaNUqVIZPiclLy8vVq5cSdu2bbly5Qq7du0iICDgsfuXKFGCkiVLUrZs2UeSVlpu\n3bqFp6cn+fLlo3z58ly9ehXAUj6/Zs2a/P333w8958E68F26dOHo0aMPbQsICGDXrl0kJiZSrVo1\nVq5cSf369bl8+TJeXl64uLhQsWJFrl83VUsqXbo0ZcqUAUytLx8fH1xcXKhduzZgSmSTJ0+mf//+\nREREZPh6slOe/MS4EX+f81fj6errYe9Q7MJgMPDf//6Xf/3rXzg7OzNnzpw8VcQvvZZAdnJycuKd\nd95hypQpLFu2jIYNG7Js2TJ69OjBlStX+OOPP5g0aRLFixfnnXfeITg4GFdXVw4cOECNGjUs3U9V\nq1Zlz549tG/fHqPRiJOTE1prEhMTHxkAfZBISpQoQUJCAosWLbJ0d1WtWpUZM2ZQunRpDAbDQ1+C\nKlWqxIULFwDYuHEjVapU4bvvvmPChAmWZJQySfn4+LBq1SoKFSpEUlISrq6uzJ49m7CwMM6dO8eQ\nIUMeisuaFkWVKlU4evQoBoOBTZs20aBBg4f2L1WqFD/++CP37t2jT58+eHl50bBhQ/744w/8/PzY\nsmWL5bWeP3+eqlWrPvT8Ro0aMW3aNJYsWUJYWBgnT57Ey8vL8rrh4VZfyp+t+QZepEgRIiMjKV26\nNBcuXOC5554DTEm1SpUqlisEH0i5kuPOnTupWLHiQ8erX78+b731FiEhIQQEBDBmzBg+++wzSpYs\nSUREBMnJyVy4cMEyXynl78fLy4tDhw7h4+Nj6Zry8vJi4cKFbN++nRkzZjBr1qwMX1N2yZOJIjz6\n6V7RrlOnTvz888+0bduWuXPn5roZ1o6katWqxMbGcunSJaZMmcLw4cOZO3cuSik+//xz8uXLR40a\nNejRowfNmjVDa0358uUt/f4AHTt2ZP369QQFBVG4cGE2bNhA//79CQwMtIxZpKVdu3Z88sknzJw5\nE4APPviAgQMHWrowVq9eTeHChQHTN1tnZ2cMBgP+/v588MEHHDhwgKJFi1K5cuVHjv3ee+/RoUMH\ntNYUK1aM1atX4+fnR1BQEEFBQY/s/6BFkR5XV1eGDBlC48aNefbZZ/nuu+8AeOONN5g1axY///wz\n06ZNw9nZmU8++QSlFB06dGD16tUEBgbStm1bSpcujdFoJDExkeLFiz90/ICAAN544w1q1qyJr6+v\n5Rt/Sp6enrz88st88MEH6caalvfee4/u3btjMBgYOXKkpfW1efNmZs6cSdOmTR/qdrp27Rrt27en\ncOHCFC9enCVLljx0vPz58+Pi4mIZ9zhx4gT+/v7ky5ePoUOHEhgYiLOzM7Nnz34klrFjx9KrVy9L\nixVMLYp9+/YRHx/PjBkzMv36noTK6b6uJ1W5SAF95ta9dPf5z++nmbn5DEemtKJw/jyZCx+Rsojf\n999/T3JyMj179sxTXW8nTpygevXq9g4j11q5ciUuLi506dLF3qE8kbVr15KQkEC3bt3sHQq9e/fm\n/fffp0KFCvYOJdNS/70opQ5qrX2zciybfooqpVoDn2NaVvVLrfXHqbaPAgYDyUAsMFBrfeGRA2VS\neFQcVUq6PzVJ4sCBAwwaNIjQ0FBGjBiRK/7ARM7r2rWrvUPIFp06dbJ3CCIVmw1mK6WcgS+ANoA3\n0EMp5Z1qtzDAV2tdG1gFfPqk59Vacyg67qnodrp37x7jxo3D39+f2NhYWSdCiGz27bffOmRrIrvZ\n8qonP+Cs1jpCa30fWA489FVBa71Fa33XfHcP8MSjz39fu0vc3aQ8P39i9+7d1KlTh08//ZSBAwdy\n/Phx2rdvb++whBB5kC37ZsoCUSnuRwP+6ew/CPglrQ1KqVAgFKCSe/rzIsIiTSva5fXSHffu3cNo\nNLJp0yaaNcsTq9YKIXKpXNGJr5TqDfgCTdLarrWeD8wH02B2escKj4qjUD5nKpfMexVjN2zYwLFj\nx3j77bdp2rQpJ06cyLZ680II8Ti27HqKAVJel+lhfuwhSqnmwASgo9Y6MfX2zAqPiqOWR1GcnfLO\n1T5Xr16ld+/etGvXjqVLl1oKzkmSsA0pCph1T1tRwHbt2vHMM888dOnw4sWLadSoER07duT27dtZ\nOm5GHpR9yTFZLRKV0Q1TayUC8ALyAYeAGqn28QHOAZWtPW56RQHv3U/Wlf71s/5ow4nH7uNIjEaj\nXrZsmX7uuee0q6urnjx5sqXw3NNIigKmT4oC5nxRwIsXL+oJEyboLVu2aK21TkxM1I0bN9bJycn6\n22+/1dOnT3+i4z+Q+nfWpEmTDJ/jEEUBtdbJwOvARuAEsEJrfUwpNVUp1dG827+BwsBKpVS4Umrd\nk5zz2MVbJBl0nrniKTIykn79+uHl5cXBgweZMmWKlALPYVIU0HpPY1HA1DW+Tp48Sd26dXF2dk7z\nPXjw+x83bhxjxox56LEPP/yQgIAAmjVrRnR0NMnJyfj4+NCrVy+mTZvG3r17qVevHr1797a0VGbO\nnEmDBg0ICQnh0KFDGb7GrLLpGIXWegOwIdVj76b4OVuXUssLA9laa/744w+aN29O+fLl2bZtGy++\n+KJllqgAfnkHLh15smM8XwvafPzYzVIUUIoCWlMUMLWM3oMXXniBc+fOERMTg4uLC3///Tfly5cn\nJiaGP//8k507d7J161Y++eQT/vOf/xATE8POnTspWLAgbdu2Zf369bi7u1vKhaxfv56tW7fi5uZm\n00KBeaooYHhUHGWKulGqiGNWjD137hzNmjWjRYsWliJ+DRo0kCRhB1IUUIoCWlMUMLWMXk9AQABb\ntmzBzc0NNzc3Nm/eTEBAAOfPn6du3boA+Pr6cvbsWQCqVatGwYIFAVOrqmzZshQpUoRKlSoBMGXK\nFIYOHcrQoUPTLGmSXXLFVU/ZJTwqziHnTxgMBj7//HMmTpyIq6sr8+bNy1NF/LJdOi2B7CRFAf9H\nigI+WhQwLdWqVSM8PNxy6Xrq96BRo0Z07tyZoUOH4urqysyZM1m9ejX58+cnPDwcMFVaeNBiSPk7\nK1y4MBcvXsTd3d2SSOrVq8c333zD4sWLWbx4cYbxZVWeSRRX7yQSfeMefRs63uzkDh068Msvv9C+\nfXvmzJlj6U4Q9idFAU2kKOCjRQEBRowYwYYNG/jll18YPnw4gwYNYuDAgQQGBlK8ePFHWl0VKlQg\nNjaWwMBAXFxcmDhxoiUpNGrUiEaNGpE/f34WL178SHzvvvsu7du3p2rVqpZuxiFDhhAZGUliYiLf\nfPNNpl+ztfJMUcBNxy8zePEBVgxtiJ9XMTtEljn379/HxcUFJycnVqxYgcFgoHv37nmqiF92k6KA\n6ZOigNlPigKa5JkWRXhUHM5Oilpli9o7lAzt27ePQYMGMXToUF5//XXLoJkQT0KKAgpbyTOD2WFR\nN6j2vDsF8uXegd+7d+8yevRoGjZsyI0bNx5Z6EQIkbtIUUCTPJEojEbN4aibuXr+xI4dO6hVqxbT\np09nyJAhHDt2LN3+aSGEyC3yRNfTudg73E5MztWJ4sHCQlu2bMn56fdCCPEE8kSiCIsyXSOd2yba\nrV+/nhMnTjB27FhCQkI4fvw4Li554i0XQjxF8kTXU3hUHO5uLrzwXGF7hwJAbGwsPXv2pGPHjixb\ntsxSIE6ShGOQooBZl9migA+MHDmS3r17A45TFDAuLs7yejp06MCdO3cA+P3332nYsCHNmjVLdyLm\nk+jduzfR0dE2OXaaslokyl63tIoCtp6xXff+ck/6FbJygNFo1EuXLtXFixfXrq6ueurUqU91Eb/s\nJkUB0+eIRQG11vrSpUu6Q4cOlgJ9jlIU8O7du/rSpUtaa61nz56tZ8+erbU2Fey7ffu23rFjh37z\nzTezfPyUUv/OevXqpaOiotJ9jkMUBcwpd+8nc+rSrVwxPhEZGcmAAQOoVKkSYWFhTJo0SYr4OTgp\nCmi9rBQFBJgxYwZvvPGGZT9HKQpYoEABSpUqBZgmGzo7O3Pr1i2KFClC4cKFadSoEYcPH37oOV27\nduXOnTvMmTOHV155BTBNrDQajSxatIgGDRoQGBjIkSOmWmb+/v4MHTqUsWPHcu7cOfz8/OjSpYul\npbJq1Sr8/Pxo2rQpGzduzPA1ZpXD94Ucib6JUWO3RGE0Gvn9999p1aoV5cuX588//6R+/fpSn8mG\nPtn3CSevn3yiY1QrVo1xfuMeu12KAuZMUcDr168TGxv7yOxxRyoKePv2bRYuXMjGjRsfeg+UUiQn\nJz+0r5+fH3v37uXgwYO4urpaZtkbDAbmzp3Ljh07uHDhAqNGjWLt2rVcuXKFyZMnU6ZMGUJDQ5k1\naxb169enRo0aAKxevZoffvgBDw8PmxYFdPhEEW4eyLZHojhz5gxDhgxh27ZtbNu2jaCgIPz8/HI8\nDpH9+vTpw9SpUxk0aBCRkZHUqlUrzaKA1atXtxQFfPDHm1pWiwIOGzYMb2/vR4oCgulDOSYmJs1y\nL/Hx8YSGhhITE8Ply5epXLkylStXTrMoIJgWxoqNjeWnn35i6dKlODk5PZIUrSnhYU1RwKFDh/LF\nF19YigJ+/vnnjBgxIt3jPpCyKOCePXvYtWsXb7/99kO1nlJ6UBQQyFRRQCDdooApE4XRaKR///58\n8sknFClSBK215T3QWj8yLhkQEMAff/xBYmIi1apVY+XKldSvX5/Lly/j5eWFi4sLFStW5Pr164Cp\njHmZMmUAU4vMx8cHFxcXateuDZiS2+TJkzEYDEyaNMlmc7PyRKLwLFaA4oXz59g5k5OT+c9//sO7\n7xcEsY4AABCWSURBVL5L/vz5WbhwoRTxy0HptQSykxQF/B9bFQU8f/4848eP5969e5w5c4YVK1bw\n6quvOkxRwAkTJlgGtOF/yTI+Pp5Dhw5Rq1ath/avX78+b731FiEhIQQEBDBmzBg+++wzSpYsSURE\nBMnJyVy4cIFnn30WePh35uXlxaFDh/Dx8bF0TXl5ebFw4UK2b9/OjBkzmDVrVoavMyvyRKLwrZCz\ntZ3at2/Pxo0b6dSpE7Nnz7ZkfJH3SFFAE1sVBXxQ/O7vv/9m4sSJvPrqqw5TFDAyMpLp06fTsGFD\nVq5cSc+ePQkNDWX8+PE0b96cAgUKPFLcL3/+/Li4uFjGQk6cOIG/vz/58uVj6NChBAYG4uzszOzZ\nsx+Jb+zYsfTq1ctS2h5MLYp9+/YRHx/PjBkzMv2areXQRQEv3UygwUd/8G57bwYGetn0vA/+OJ2c\nnFi1ahVGo5GuXbtKEb8cJEUB0ydFAbOfFAU0cegWRXiUaUU7W69BsWfPHgYNGsSwYcN44403LFcr\nCJGbSFFAYSsOfXlsWFQcrs4K79JFbHL8+Ph43nrrLRo1asTt27fTbMILIfIuKQpo4tgtisg4vEsX\nwc01+y9F/f/t3X9w1PWdx/Hnu+THJvLDEqa/wIIMlJQqhpjrcQm4yUGrRo6KIj8GaDwBUY5j2kpH\nBUbp3YkylThlpEc5qiba1laiDNB62KkBYwQqSgJiKIIwaWKFyCnlVxKSfd8f3282a37sbkJ2k03e\nj5kdsrvf/e5n3yT72c/n+93Xp6SkhLy8PP+52o8//rj/tDdjjOlLYrajaGj0caj6LHfdGJnV4Boa\nGoiPj/ef9mqMMX1VzHYUR0+d52J9I+O//sUu2+fWrVupqKjg4YcfJicnh8OHD1s+kzGmz4u5YxSX\n451/u/KLdqdOnWLmzJlMnz6dLVu2WIhfH2ehgJ3XmVDAtWvXMmXKFLKzs/H5fDETCgjOKbter5cp\nU6b4T9ctLCwkMzOTadOmce7cuU7tN5SoL1XQ2ZCo7rqkp49XVdUfv1SmaT/ZqT6fL2gwVjA+n08L\nCwt18ODBmpCQoI899pjW19d3en8msiwUMLhYDAXct2+frlq1qlX7YiEUUFX97xebN2/W/Px8raur\n00mTJmlDQ4O+8MILmp+ff0X7b9Ly/8zr9YZ8TJ8OBRRxmlz218+44Zqrr+h7DJWVlSxcuJDU1FTK\ny8tZsWIF8fHxXdVU0wtYKGD4OhMKuGPHDj755BNycnL8oXuxEgoI+N8vamtrGTt2LEeOHCEtLY1+\n/fq1WYOm//8HH3yQ5cuXf+62NWvWkJWVxeTJk6mqqqKhoYHx48czd+5cnnzySfbt20d6ejrz5s3z\nj1TWr1/PhAkTyMnJoby8PORr7KyYnFs5V3uZD06fJ/f69oPY2uPz+di5cye33norw4cPp7S0lPHj\nx1uIXwz5eM0a6iquLBQw8ZupfGXFinbvt1DA6IQCnjp1ipSUFIqLi5k9ezbvvvsu6enpMRMKeOLE\nCebMmcPFixd59dVXOX78eNAajBw5kuPHj1NdXU1cXBwnT55k+PDhVFdXU1JSQmlpKbt27WLt2rU8\n9dRTVFdXU1paSnJyMrm5uWzfvp0BAwb4M522b9/Orl278Hg8EQ0FjLkRBTiJsdqJxNijR4+SnZ1N\nbm4uu3fvBiAjI8M6CdPK/PnzOXDgABkZGVRWVgKt84Kqqqr8cQrBFqjpbCjg1q1beeedd1qFAmZn\nZ1NRUUF1dXWbz3fhwgUWLFiA1+tly5Yt/ra1FQqYk5PDxx9/TE1NDc8//zw33XQTCxcubPV6miI8\nAi+BnQSEFwq4adMm7rrrLn8o4KBBg/B6vQDk5ORQUVHRbh0DQwFPnz7NW2+9RVZWVrvbN4UCDh06\ntEOhgAkJCUFDAVu2ae/evaxcuZL8/PyQNcjKyqK4uBiPx4PH4+H1118nKyuLEydOkJaWBjjvSceO\nHQMgNTWV5ORkwBlVDR06lIEDBzJq1CgAVq9ezeLFi1m8eHGbkSZdJSZHFAc6eCC7oaGBdevW8eij\nj5KUlMSzzz5rp7zGsGAjga5koYDNIhUK2LRmw80330xZWZk/HTcWQgEvX75MXFwcIsLAgQNJSkoi\nNTWVsrIyfD5fmzXIzMzk9ttvZ/HixcTHx7N+/XqKiopITEykrKwMgP379/tHDIH/Z/379+ejjz5i\nwIAB/o4kPT2dgoICCgsLKSwsbBVa2FVis6Oo/IyRQ67i6uTwFgW67bbbeO2117jjjjvYsGFDqzx5\nY9pjoYCOSIUCTp06lUWLFuH1eklNTSUzMzNmQgGrqqrIy8ujX79+eDweCgoKSExM5J577mHixImk\npKS0GnWNGDGCmpoaJk6cSFxcHKtWrfJ3CpmZmWRmZpKYmNgqTBDgkUceYerUqYwZM8Y/zbho0SIq\nKyupq6ujoKCgw685XDEXCpiRkaHc/gQ3jR5C/qy0drerra31rzpVVFQEwJ133hmtZpoIsFDA4CwU\nsOtZKKAj5kYU9Y0+/n6+LmgQYGlpKQsWLGDJkiUsW7bMOgjTJ1gooImUmDuYfam+EWj7+MT58+dZ\ntmwZkyZNora21j59GmOuiIUCOmJuRHGxvpH+cV8g9SufD+jbvXs3eXl5VFZWsnTpUtasWeOfvzW9\nh6raGiDGhNDVhxRisKNoYMLQQSTEtR4MJScnU1JSEvSUORO7PB4PZ86cISUlxToLY9qhqpw5cwaP\nx9Nl+4y5juJSfaN/2unll1/myJEjrFixAq/Xy6FDh+w7Eb3YsGHDqKqqoqamprubYkyP5vF4/F/G\n7AoR7ShE5BbgZ0A/YLOqPtHi/kSgELgROAPMUtWTwfapwPCkembMmEFRUREZGRksX76chIQE6yR6\nufj4eK69NrJL3hpjWovY6bEi0g84CnwHqALeBuao6vsB2ywBxqnqfSIyG5iuqkHPiYsb9GW9Suqp\nq73E6tWreeCBByyfyRhjQriS02MjedbTt4FjqvqhqtYDLwItz3v7HtD0LZEtwGQJMfnc+Pcaxl1/\nHeXl5Tz00EPWSRhjTIRFcuppKPDXgOtVwD+2t42qNojIWSAF+NxXLkXkXuBe92rdm2+++V5qampE\nGh1jhtCiVn2Y1aKZ1aKZ1aLZmNCbtC0mDmar6iZgE4CI7O/s8Km3sVo0s1o0s1o0s1o0E5H9nX1s\nJKeeqoFrAq4Pc29rcxsRiQMG4RzUNsYY00NEsqN4GxgtIteKSAIwG9jWYpttQJ778wzgdY218Clj\njOnlIjb15B5zWArsxDk99hlVPSwi/4GzJN824JfA8yJyDPg/nM4klE2RanMMslo0s1o0s1o0s1o0\n63QtYi491hhjTHTFXCigMcaY6LKOwhhjTFA9tqMQkVtE5C8ickxEHmrj/kQR+a17/z4RGRH9VkZH\nGLX4kYi8LyIHReRPIjK8O9oZDaFqEbDdnSKiItJrT40MpxYiMtP93TgsIr+OdhujJYy/ka+LSLGI\nHHD/TnK7o52RJiLPiMhpEXmvnftFRNa7dTooIulh7VhVe9wF5+D3cWAkkACUA2NbbLME2Oj+PBv4\nbXe3uxtrkQMkuz/f35dr4W43AHgD2AtkdHe7u/H3YjRwAPiie/1L3d3ubqzFJuB+9+exwMnubneE\nanETkA681879ucCrgAATgH3h7LenjigiEv8Ro0LWQlWLVfWie3UvzndWeqNwfi8A/hNYC9RGs3FR\nFk4tFgEbVPVTAFU9HeU2Rks4tVCgaRGbQcBHUWxf1KjqGzhnkLbne0ChOvYCV4vIV0Ptt6d2FG3F\nfwxtbxtVbQCa4j96m3BqEWgBzieG3ihkLdyh9DWq+vtoNqwbhPN78Q3gGyJSKiJ73TTn3iicWqwG\n5olIFfAH4N+j07Qep6PvJ0CMRHiY8IjIPCAD8HZ3W7qDiHwByAfu7uam9BRxONNP2TijzDdE5HpV\n/axbW9U95gDPqeo6EfknnO9vXaeqvu5uWCzoqSMKi/9oFk4tEJEpwEpgmqrWRalt0RaqFgOA64Bd\nInISZw52Wy89oB3O70UVsE1VL6vqCZzY/9FRal80hVOLBcDvAFR1D+DBCQzsa8J6P2mpp3YUFv/R\nLGQtRGQ88AucTqK3zkNDiFqo6llVHaKqI1R1BM7xmmmq2ukwtB4snL+RrTijCURkCM5U1IfRbGSU\nhFOLSmAygIh8E6ej6ItLJW4Dvu+e/TQBOKuqfwv1oB459aSRi/+IOWHW4qdAf+Al93h+papO67ZG\nR0iYtegTwqzFTuC7IvI+0Aj8WFV73ag7zFo8APyPiPwQ58D23b3xg6WI/Abnw8EQ93jMo0A8gKpu\nxDk+kwscAy4C/xrWfnthrYwxxnShnjr1ZIwxpoewjsIYY0xQ1lEYY4wJyjoKY4wxQVlHYYwxJijr\nKEyPIyKNIlIWcBkRZNsR7SVldvA5d7npo+Vu5MWYTuzjPhH5vvvz3SLytYD7NovI2C5u59sikhbG\nY34gIslX+tym77KOwvREl1Q1LeByMkrPO1dVb8AJm/xpRx+sqhtVtdC9ejfwtYD7Fqrq+13SyuZ2\n/pzw2vkDwDoK02nWUZiY4I4cSkTkXfeS2cY23xKRP7ujkIMiMtq9fV7A7b8QkX4hnu4NYJT72Mnu\nGgaH3Kz/RPf2J6R5DZAn3dtWi8hyEZmBk7n1K/c5k9yRQIY76vC/ubsjj6c72c49BAS6ich/i8h+\ncdae+Il72zKcDqtYRIrd274rInvcOr4kIv1DPI/p46yjMD1RUsC00yvubaeB76hqOjALWN/G4+4D\nfqaqaThv1FVuXMMsIMu9vRGYG+L5/wU4JCIe4Dlglqpej5NkcL+IpADTgW+p6jjgvwIfrKpbgP04\nn/zTVPVSwN1F7mObzAJe7GQ7b8GJ6WiyUlUzgHGAV0TGqep6nEjtHFXNcaM8VgFT3FruB34U4nlM\nH9cjIzxMn3fJfbMMFA887c7JN+LkFrW0B1gpIsOAl1X1AxGZDNwIvO3GmyThdDpt+ZWIXAJO4sRQ\njwFOqOpR9/4C4N+Ap3HWuviliOwAdoT7wlS1RkQ+dHN2PgBSgVJ3vx1p51U4cRWBK5TNFJF7cf6u\nv4qzQM/BFo+d4N5e6j5PAk7djGmXdRQmVvwQOAXcgDMSbrUokar+WkT2AbcBO0VkIc5KXgWq+nAY\nzzE3MEBQRAa3tZGbLfRtnJC52cBS4J878FpeBGYCR4BXVFXFedcOu504q7g9AWwA7hCRa4HlwD+o\n6qci8hxO8F1LAvxRVed0oL2mj7OpJxMrBgF/c9cPmI/zafpzRGQk8KE73bINZwrmT8AMEfmSu81g\nCX9N8b8AI0RklHt9PrDbndMfpKp/wDlQ3NaZR+dwYs/b8grOSmNzcDoNOtpOVb2MM4U0QURScVZv\nuwCcFZEvA7e205a9QFbTaxKRq0SkrdGZMX7WUZhY8XMgT0T24kw7XWhjm5nAeyJShjOlU+ieabQK\neE1EDgJ/xJmWCUlVa3HSNV8SkUOAD9iI86a7w93fbpzRTkvPARubDma32O+nQAUwXFX/7N7W4Xa6\nxz7W4aTCluOsj30YeAZnOqvJJuB/RaRYVWtwzsj6jfs8e3BqZUy7LD3WGGNMUDaiMMYYE5R1FMYY\nY4KyjsIYY0xQ1lEYY4wJyjoKY4wxQVlHYYwxJijrKIwxxgT1/ypqWVMGJnA6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x16d60d53048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pylab as pl\n",
    "\n",
    "top_k_list = [5, 10, 20, 30]  #Select top words to see performance\n",
    "pl.clf()\n",
    "for top_k in top_k_list:\n",
    "    selected_features = select_top_features(chi2, top_k, tfs, labels)\n",
    "    train_x, test_x, train_y, test_y = split_data(selected_features, labels)\n",
    "    clf = sweep_logistic_regression_cross_val(train_x, train_y)\n",
    "    predict = prediction(clf, test_x)\n",
    "    fpr, tpr, roc_auc = calc_auc(test_y, predict)\n",
    "    pl.plot(fpr, tpr, label='ROC curve (area = %0.2f) with Top %d words' % (roc_auc, top_k))\n",
    "pl.plot([0, 1], [0, 1], 'k--')\n",
    "pl.xlim([0.0, 1.0])\n",
    "pl.ylim([0.0, 1.0])\n",
    "pl.xlabel('False Positive Rate')\n",
    "pl.ylabel('True Positive Rate')\n",
    "pl.title('Sentiment Analysis')\n",
    "pl.legend(loc=\"lower right\", prop={'size':8})\n",
    "pl.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}