#6 Deploying Computer Vision Models to the Cloud with Azure ML Service.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# #6 Deploy an image classification model in Azure Container Instance (ACI)\n",
    "\n",
    "![](https://github.com/ashleymcnamara/Developer-Advocate-Bit/blob/master/bit_ai.png?raw=true)\n",
    "\n",
    "Photo credit to the talented Ashley McNamara\n",
    "\n",
    "#### Let's Deploy a  Computer Vision model to the Cloud using the Azure Machine Learning Service to make this simple we will revisit the MNIST dataset from the third Notebook\n",
    "\n",
    "Based on the [Azure ML Documentation](https://docs.microsoft.com/en-us/azure/machine-learning/service/tutorial-train-models-with-aml?WT.mc_id=cvworkshop-github-abornst)\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "\n",
    "> * Set up your testing environment\n",
    "> * Retrieve the model from your workspace\n",
    "> * Test the model locally\n",
    "> * Deploy the model to ACI\n",
    "> * Test the deployed model\n",
    "\n",
    "ACI is not ideal for production deployments, but it is great for testing and understanding the workflow. For scalable production deployments, consider using AKS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Login and Authenticate with Azure\n",
    "#### Generate credientials file becareful with this anyone who has acess to this file has acess to your account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 -m azure.cli login\n",
    "!python3 -m azure.cli ad sp create-for-rbac --sdk-auth > mycredentials.json\n",
    "!export AZURE_AUTH_LOCATION='mycredentials.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, json\n",
    "with open('mycredentials.json') as data_file:    \n",
    "    azure_credentials = json.load(data_file)\n",
    "\n",
    "# delete credentials file \n",
    "os.remove(\"mycredentials.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is an Azure ML Workspace and why do I need one?\n",
    "\n",
    "An AML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows.  In particular, an AML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, operationalization, and the monitoring of operationalized models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CVWorkshop_RG = 'cvdemo_rg'\n",
    "CVWorkshop_LOC = 'eastus2'\n",
    "CVWorkshop_WS = 'cv_ws'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up your Azure Machine Learning workspace\n",
    "\n",
    "### Option 1: You have workspace already\n",
    "If you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace!  You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n",
    "\n",
    "If you have a workspace created another way, [these instructions](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-environment#create-workspace-configuration-file) describe how to get your subscription and workspace information.\n",
    "\n",
    "If this cell succeeds, you're done configuring this library!  Otherwise continue to follow the instructions in the rest of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote the config file config.json to: /data/aml_config/config.json\n",
      "Workspace configuration succeeded. You are all set!\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "subscription_id = os.environ.get(\"SUBSCRIPTION_ID\", azure_credentials['subscriptionId'])\n",
    "resource_group = os.environ.get(\"RESOURCE_GROUP\", CVWorkshop_RG)\n",
    "workspace_name = os.environ.get(\"WORKSPACE_NAME\", CVWorkshop_WS)\n",
    "\n",
    "from azureml.core import Workspace\n",
    "\n",
    "try:\n",
    "    ws = Workspace(subscription_id = subscription_id, resource_group = resource_group, workspace_name = workspace_name)\n",
    "    ws.write_config()\n",
    "    print('Workspace configuration succeeded. You are all set!')\n",
    "except:\n",
    "    print('Workspace not found. Run the cells below.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 2: You don't have workspace yet\n",
    "\n",
    "\n",
    "#### Requirements\n",
    "\n",
    "Inside your Azure subscription, you will need access to a _resource group_, which organizes Azure resources and provides a default region for the resources in a group.  You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com).  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n",
    "\n",
    "To create or access an Azure ML Workspace, you will need to import the AML library and the following information:\n",
    "* A name for your workspace\n",
    "* Your subscription id\n",
    "* The resource group name\n",
    "\n",
    "#### Supported Azure Regions\n",
    "Specify a region where your workspace will be located from the list of [Azure Machine Learning regions](https://azure.microsoft.com/en-us/global-infrastructure/services/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the Workspace class and check the azureml SDK version\n",
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.create(name = CVWorkshop_WS,\n",
    "                      subscription_id = azure_credentials['subscriptionId'],\n",
    "                      resource_group = CVWorkshop_RG, \n",
    "                      location = CVWorkshop_LOC,\n",
    "                      create_resource_group = True,\n",
    "                      exist_ok = True)\n",
    "ws.get_details()\n",
    "ws.write_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "register model from file"
    ]
   },
   "source": [
    "### Register a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": [
     "register model from file"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found the config file in: /data/aml_config/config.json\n",
      "Registering model sklearn_mnist\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('./data/test-labels.gz', <http.client.HTTPMessage at 0x7fb2596efc88>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core import Workspace\n",
    "ws = Workspace.from_config()\n",
    "\n",
    "from azureml.core.model import Model\n",
    "\n",
    "model_name = \"sklearn_mnist\"\n",
    "model = Model.register(model_path=\"models/sklearn_mnist_model.pkl\",\n",
    "                        model_name=model_name,\n",
    "                        tags={\"data\": \"mnist\", \"model\": \"classification\"},\n",
    "                        description=\"Mnist handwriting recognition\",\n",
    "                        workspace=ws)\n",
    "\n",
    "# download test data\n",
    "import os\n",
    "import urllib.request\n",
    "\n",
    "os.makedirs('./data', exist_ok=True)\n",
    "\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz', filename='./data/test-images.gz')\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz', filename='./data/test-labels.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up the environment\n",
    "\n",
    "Start by setting up a testing environment.\n",
    "\n",
    "### Import packages\n",
    "\n",
    "Import the Python packages needed for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "check version"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure ML SDK Version:  1.0.2\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    " \n",
    "import azureml\n",
    "from azureml.core import Workspace, Run\n",
    "\n",
    "# display the core SDK version number\n",
    "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the model\n",
    "\n",
    "You registered a model in your workspace in the previous tutorial. Now, load this workspace and download the model to your local directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "load workspace",
     "download model"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found the config file in: /data/aml_config/config.json\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "os.stat_result(st_mode=33261, st_ino=827, st_dev=112, st_nlink=1, st_uid=0, st_gid=0, st_size=63684, st_atime=1544229481, st_mtime=1544229481, st_ctime=1544259770)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core import Workspace\n",
    "from azureml.core.model import Model\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "model=Model(ws, 'sklearn_mnist')\n",
    "model.download(target_dir='.', exist_ok=True)\n",
    "import os \n",
    "# verify the downloaded model file\n",
    "os.stat('./sklearn_mnist_model.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test model locally\n",
    "\n",
    "Before deploying, make sure your model is working locally by:\n",
    "* Loading test data\n",
    "* Predicting test data\n",
    "* Examining the confusion matrix\n",
    "\n",
    "### Load test data\n",
    "\n",
    "Load the test data from the **./data/** directory created during the training tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import load_data\n",
    "\n",
    "# note we also shrink the intensity values (X) from 0-255 to 0-1. This helps the neural network converge faster\n",
    "X_test = load_data('./data/test-images.gz', False) / 255.0\n",
    "y_test = load_data('./data/test-labels.gz', True).reshape(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict test data\n",
    "\n",
    "Feed the test dataset to the model to get predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/sklearn/base.py:251: UserWarning: Trying to unpickle estimator LogisticRegression from version 0.19.1 when using version 0.20.0. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  UserWarning)\n"
     ]
    }
   ],
   "source": [
    "import pickle\n",
    "from sklearn.externals import joblib\n",
    "\n",
    "clf = joblib.load('./sklearn_mnist_model.pkl')\n",
    "y_hat = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Examine the confusion matrix\n",
    "\n",
    "Generate a confusion matrix to see how many samples from the test set are classified correctly. Notice the mis-classified value for the incorrect predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 960    0    1    2    0    5    6    3    1    2]\n",
      " [   0 1112    3    1    0    1    5    1   12    0]\n",
      " [   8    8  920   20    9    5   10   11   37    4]\n",
      " [   4    0   17  919    2   22    4   12   21    9]\n",
      " [   1    2    5    3  914    0   10    2    7   38]\n",
      " [  10    2    0   42   10  769   17    7   28    7]\n",
      " [   9    3    7    2    6   20  907    1    3    0]\n",
      " [   2    7   22    5    8    1    1  950    5   27]\n",
      " [  10   14    5   21   14   27    7   11  853   12]\n",
      " [   8    8    2   13   31   14    0   24   12  897]]\n",
      "Overall accuracy: 0.9201\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "conf_mx = confusion_matrix(y_test, y_hat)\n",
    "print(conf_mx)\n",
    "print('Overall accuracy:', np.average(y_hat == y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `matplotlib` to display the confusion matrix as a graph. In this graph, the X axis represents the actual values, and the Y axis represents the predicted values. The color in each grid represents the error rate. The lighter the color, the higher the error rate is. For example, many 5's are mis-classified as 3's. Hence you see a bright grid at (5,3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAFBCAYAAABw9sSEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3XucXVV99/HPNzO5EG5iUIQESGwQSbEFjKBVUUyhAZVoH7DBS1HQ2FYqiD4WtAWk0Ir1/mgtqUFQQcAIbarRAAVEW425AJIEgmMIkoCEEEwIECCZ3/PH3iMnh8nM2ZOz15xz9ved13nNvv/WmZn8zpq11l5bEYGZmVXHiOEugJmZpeXEb2ZWMU78ZmYV48RvZlYxTvxmZhXjxG9mVjEdlfglTZe0UlKPpHNKjHOZpHWSlpUVoybW/pJukbRC0nJJZ5YYa4ykX0i6M4/1qbJi1cTsknS7pO8niLVa0l2S7pC0uORYL5A0V9I9ku6W9JqS4hycv5++1yZJZ5URK4/3kfx3Y5mk70gaU2KsM/M4y8t8T5UUER3xArqAXwMvBUYBdwJTSop1NHAEsCzB+9oXOCJf3h24t8T3JWC3fHkksBB4dcnv72zgKuD7Cb6Xq4G9y46Tx7oCeH++PAp4QYKYXcBvgQNLuv544D5gl3z9WuC9JcU6FFgGjAW6gZuAySl+dlV4dVKN/0igJyJWRcQzwNXAjDICRcRtwIYyrt1PrIciYmm+/DhwN9l/wDJiRURszldH5q/S7vCTNAF4M/D1smIMB0l7klUO5gBExDMR8bsEoacBv46I+0uM0Q3sIqmbLCk/WFKcQ4CFEfFkRGwFfgz8eUmxKqeTEv944IGa9TWUlCCHi6SJwOFkNfGyYnRJugNYB9wYEaXFAr4IfBzoLTFGrQBukLRE0qwS40wCHgG+kTdjfV3SriXG6zMT+E5ZF4+ItcBngd8ADwEbI+KGksItA14vaZykscAJwP4lxaqcTkr8HU3SbsD3gLMiYlNZcSJiW0QcBkwAjpR0aBlxJL0FWBcRS8q4/g68LiKOAI4HPiTp6JLidJM1BX4tIg4HngBK63MCkDQKOBH4bokx9iL7K3oSsB+wq6R3lxErIu4GLgFuAH4E3AFsKyNWFXVS4l/L9jWCCfm2tidpJFnSvzIirksRM2+auAWYXlKI1wInSlpN1iz3JknfLikW8PsaKxGxDrierHmwDGuANTV/Lc0l+yAo0/HA0oh4uMQYfwrcFxGPRMSzwHXAn5QVLCLmRMQrI+Jo4DGy/i1rgk5K/IuAgyRNyms/M4F5w1ymnSZJZG3Fd0fE50uO9SJJL8iXdwGOBe4pI1ZEnBsREyJiItnP6uaIKKX2CCBpV0m79y0Dx5E1JzRdRPwWeEDSwfmmacCKMmLVOIUSm3lyvwFeLWls/ns5jazPqRSSXpx/PYCsff+qsmJVTfdwF6BZImKrpDOABWSjGy6LiOVlxJL0HeCNwN6S1gDnR8ScMmKR1YzfA9yVt70DfCIi5pcQa1/gCkldZJWCayOi9GGWiewDXJ/lK7qBqyLiRyXG+1vgyrwSsgp4X1mB8g+yY4EPlhUDICIWSpoLLAW2ArcDs0sM+T1J44BngQ8l6iCvBEV4WmYzsyrppKYeMzNrgBO/mVnFOPGbmVWME7+ZWcU48ZuZVUzHJf6Sb8Uftlip4zmWY7VKvNTvrQo6LvEDKX9JUv9Cdup7c6z2ipU6nhN/k3Vi4jczswG09A1cklq3cDtFQzwvhnRufrdqklgRqSbahO7uUYXP6e3dxogRXYXPk4rXkXp7tzJiRPGb47dte7bwORExxJ8z9PYO5WfW2r+LEb1ExFD/ow1o+vTpsX79+kLnLFmyZEFElDXvVWEdM2VDO+nuHpk03siRo5PFeuqpx5PFGjduv2SxRo4s7UFTz7Np4yPJYgE8mfBnlup3ccuWJ0q79vr161m8uNgD3CTtXVJxhsSJ38ysoFZuKWmEE7+ZWUG9TvxmZtURuMZvZlYxQZT3KOoknPjNzIoI6G3vvO/Eb2ZWlJt6zMwqJHDnrplZ5bR7jT/plA2SpktaKalH0jkpY5uZNUtEFHq1mmQ1/vwB3l8leyj0GmCRpHkRsSJVGczMdlZEtH1TT8oa/5FAT0SsiohngKuBGQnjm5k1RbvX+FMm/vHAAzXra/JtZmZtJQr+azUt17mbP3TB82+bWUvKRvUMdyl2TsrEvxbYv2Z9Qr5tOxExG5gNnTwts5m1s1ZsvikiZeJfBBwkaRJZwp8JvDNhfDOzpmj3zt1kiT8itko6A1gAdAGXRcTyVPHNzJqiRTtsi0jaxh8R84H5KWOamdn2Wq5z18yslXlaZjOzCnIbv5lZxbjGb2ZWKa15U1YRTvxmZgWEH8RiZlY9buoxM6sYJ34zswrxE7hsSMaM2TVpvKeeejxpvFTWr1+TLNZee70kWawnntyYLBbAH/3RMcli3X77jclilck1fjOzKumAB7E48ZuZFeQav5lZhQR4HL+ZWdV4HL+ZWcW4qcfMrGKc+M3MKiQ6YFTPiOEugJmZpeUav5lZQe3e1JOsxi/pMknrJC1LFdPMrAyRP3e30VcjJE2XtFJSj6Rz+tk/WtI1+f6FkibW7T9A0mZJHxssVsqmnsuB6QnjmZk1Xd9cPUVeg5HUBXwVOB6YApwiaUrdYacDj0XEZOALwCV1+z8P/LCR95As8UfEbcCGVPHMzMoSBf814EigJyJWRcQzwNXAjLpjZgBX5MtzgWmSBCDpbcB9wPJGgrlz18ysoN4o9mrAeOCBmvU1+bZ+j4mIrcBGYJyk3YC/Az7VaPlbrnNX0ixg1nCXw8ysXwXa7WvsLWlxzfrsiJjdpBJdAHwhIjbnfwAMquUSf/7NmA0gqb27zs2s4wRDGtWzPiKmDrB/LbB/zfqEfFt/x6yR1A3sCTwKHAWcJOkzwAuAXklbIuIrOwrWconfzKzVlXAD1yLgIEmTyBL8TOCddcfMA04FfgacBNwc2SfQ6/sOkHQBsHmgpA9ph3N+h6zAB0taI+n0VLHNzJqp2cM58zb7M4AFwN3AtRGxXNKFkk7MD5tD1qbfA5wNPG/IZ6OS1fgj4pRUsczMylTGDVwRMR+YX7ftvJrlLcDJg1zjgkZiuanHzKyATpirx4nfzKwgP4jFzKxi/CAWM7MKGeJwzpbixG9mVpATv5lZxbR7567n6jEzqxjX+M3MihjaXD0tpaUT/+jRY9l//0OGuxhN19OzJGm8d536yWSxlv7i5mSxHn30wWSx9tlnYrJYqX/nP/FvFyWL9cHj0zyHaePG9aVd2527ZmYV1O5t/E78ZmYF+QYuM7OKafMKvxO/mVkRfc/cbWdO/GZmRXhUj5lZ9bjGb2ZWIR7OaWZWQU78ZmYV0+5NPSmfubu/pFskrZC0XNKZqWKbmTVPFP7XalLW+LcCH42IpZJ2B5ZIujEiViQsg5nZTonwOP6GRcRDwEP58uOS7gbGA078ZtZW2r2pZ1ja+CVNBA4HFg5HfDOzneHO3YIk7QZ8DzgrIjb1s38WMAugu3tU4tKZmQ2sE+7cTfogFkkjyZL+lRFxXX/HRMTsiJgaEVO7ujzoyMys2ZJlVkkC5gB3R8TnU8U1M2u2dm/qSVnjfy3wHuBNku7IXyckjG9mtvPyuXqKvFpNylE9PwWUKp6ZWWlaMJkX4UZ0M7OCoteJ38ysUtq8wu/Eb2ZWRHbnbntnfid+M7OCnPjNzCqlNUfqFOHEb2ZWkDt3zcwqxG38ZmYV5MRvZlY1Tvzl2br1WTZseDBJrA0bHkoSB2DGjA8niwVw5RUXJ4s1ZsxuyWKd+sFPJov13W99OVmslx00NVksgB9eviBZrJfsMylJnCeffN7Ev03V5nm/tRO/mVnLiXDnrplZ1biN38ysQgInfjOzymn3xJ/0CVxmZjb8XOM3Myuo3Wv8TvxmZkVEgEf1mJlVi2v8DZI0BrgNGJ3HnRsR56eKb2bWLG2e95PW+J8G3hQRmyWNBH4q6YcR8fOEZTAz2ykezllAZN+pzfnqyPzV3t89M6ueDpidM+lwTkldku4A1gE3RsTClPHNzJoheqPQq9UkTfwRsS0iDgMmAEdKOrT+GEmzJC2WtDiiN2XxzMwakD2Bq8irEZKmS1opqUfSOf3sHy3pmnz/QkkT8+1HSrojf90p6e2DxRqWG7gi4nfALcD0fvbNjoipETFV8v1lZtZ6mp34JXUBXwWOB6YAp0iaUnfY6cBjETEZ+AJwSb59GTA1r1RPBy6VNGAzfrLMKulFkl6QL+8CHAvckyq+mVkz9D2Bq8k1/iOBnohYFRHPAFcDM+qOmQFckS/PBaZJUkQ8GRFb8+1jaKDvNGWVel/gFkm/BBaRtfF/P2F8M7PmyLJ/46/BjQceqFlfk2/r95g80W8ExgFIOkrScuAu4K9qPgj6lXJUzy+Bw1PFMzMryxC6H/eWtLhmfXZEzG5aebKBMn8o6RDginyo/JYdHT/kxC9pMrBmoIubmXWiIQznXB8RAz1abS2wf836hHxbf8esydvw9wQerSvX3ZI2A4cCi9mBhpp6JP2TpFPzZUm6EbgXeEjSUY1cw8ysIxRs32/wQ2IRcJCkSZJGATOBeXXHzANOzZdPAm6OiMjP6QaQdCDwcmD1QMEabeN/F7AyXz4eOAx4NfBN4NMNXsPMrCM0O/HnbfJnAAuAu4FrI2K5pAslnZgfNgcYJ6kHOBvoG/L5OuDO/B6p64G/iYj1A8VrtKlnH7LOBoAT8kL9QtIGBvhzwsys05Q1ZUNEzAfm1207r2Z5C3ByP+d9C/hWkViN1vgfBQ7Ml48D/jtf7gZUJKCZmQ2vRmv83wOuknQv8EKyP0cga/LpKaNgZmYtKWjJaRiKaDTxnw3cDxwAfDwinsi37wt8rYyCmZm1rDafpK2hxJ93PHyun+1faHqJzMxaWuPz77SqHSZ+SUc0epGIWNqc4piZtb42z/sD1vgXk3VgD9Z5G0BX00pUY9u2rWza9OjgBzbBLrvsniQOwM9/Xj88t1x77D4uWaxNj6f5eQFc+qVzk8U67LA3JYt15y9vTRYL4OF19yeLdfn8a5LEmXXSSaVev2Nr/MCkZKUwM2sT0cmduxGRrhpgZtZG2r3G3/DsnJJeIekrkn4oad9829skeeI1M6uUMh7EklKjc/UcRzaXxHjgTcAu+a4/AM4vp2hmZq2onCdwpdRojf8fgbMj4u3AMzXbbyV7gICZWTWU8yCWpBq9getQ6uaQyG0gu5PXzKw62rxzt9Ea/wae/zQYgCN4bvI2M7OOl03S1uwHcKXVaOK/CvgXSRPI3ne3pDcAnyWbmtnMrDKq0tTz98DlZPP1CFiRf70KuLiUkpmZtaIWTeZFNDpXz7PAuySdR/bc3BHA7RHxq6IBJXWR3RW8NiLeUvR8M7Ph1rE3cPUnIn4t6eF8efMQY55J9oSZPYZ4vpnZsGr3Gn+RG7jOkvQbYCOwUdIDkj4iqeEHseR9BG8Gvl68qGZm1gwN1fglfQaYBfwL8LN882uA88jm5P94g/G+mB+7wxnRJM3KY5mZtZyyHr2YUqNNPe8H3h8Rc2u23SxpJXApDSR+SW8B1kXEEklv3NFxETEbmJ2dM6K9v7tm1nn6xnO2sSJt/L/cwbZGm4teC5wo6QRgDLCHpG9HxLsLlMHMbJi1/6ieRpP2N4EP9bP9r2nw6e4RcW5ETIiIicBM4GYnfTNrR9Fb7NVqBnoC15frjnu3pD8Dfp5vOwrYD7iyvOKZmbWedq/xD9TU84q69SX51wPzr7/NXy8vGjQibiWb4M3MrL1EByf+iDgmZUHMzNpBlUb1mJlZrjKJX9IxwCnAAcCo2n0Rke5J1GZmwyrafsqGRp/A9V7gh2Q3Xr0ReATYi2xa5hUllc3MrPV0wINYGh3O+THgjIg4BXgWODciDge+DQx1zh4zs/bU5hPyN5r4XwrclC8/DeyWL38FeG+Ty2Rm1tLaPO83nPgf5bn5ddaSPYoRYBzPPXjdzKzj9Y3qaeemnkY7d38CHAfcBVwLfFnSscA04MaSymZm1nqiOvPxn0E2vw7APwNbyebeuRa4qIRyATB27O5MmfLasi6/nRUr/idJHICHH16dLBakHXr2qledkCzW4sU/TBZryiuOShZr8+aNyWIBnHXJp5LFOv2taWZpeeCB+0q8emvW4oto9AlcG2qWe4FLSiuRmVmL69jEL+mFjV6k9oPBzMxa20A1/vVk/RgDUX5MV9NKZGbW4jq2xg94rh4zs/50auKPiB+nLIiZWTuICo3qMTOzXJtX+J34zcyKqchwTjMze44Tv5lZlXTyE7h2RNI+wCP5jVxFz10NPA5sA7ZGxNSi1zAzG05B+3fuNjof/0hJn5H0ONkkbRPz7ZdI+puCMY+JiMOc9M2sXbX7JG2Nzs55PvBW4N1k0zL3+QWeltnMKqXgnMwtmPgbbeo5BTgtIn4sqbaJZxnwsgLxArhBUgCXRsTs+gMkzQJmAYwaNaZ+t5nZ8KpQG/9+wP07OL9IP8HrImKtpBcDN0q6JyJuqz0g/zCYDbDrrnu293fXzDpSm+f9hpt6lgNH97P9HcCSRoNFxNr86zrgeuDIRs81M2sV0RuFXq2m0dr6p4BvS9qfbEK2kyW9HHgn8OZGLiBpV2BERDyeLx8HXDiEMpuZDZu+J3C1s4Zq/BHxX2S1++OAXrLO3oOAt0bETQOdW2Mf4KeS7iTrFP5BRPyoeJHNzIZRlDOqR9J0SSsl9Ug6p5/9oyVdk+9fKGlivv1YSUsk3ZV/fdNgsRpun4+IBcCCRo/v5/xVwB8P9Xwzs9bQ/CGakrqArwLHAmuARZLmRcSKmsNOBx6LiMmSZpI9EOsvyKbQf2tEPCjpULI8PX6geI228ZuZWXmOBHoiYlVEPANcDcyoO2YGcEW+PBeYJkkRcXtEPJhvXw7sImn0QMEaqvHnN27t8CMuIvZo5DpmZp1gCDX+vSUtrlmfXTecfTzwQM36GqD+Qc+/PyYitkraCIwjq/H3+T/A0oh4mgEUedh6rZHA4XmQixu8hplZRxjCSJ31Zc9WIOkPyZp/jhvs2EYftn5Ff9slLQWmAf+vSAHNzNpWNqyn2VddC+xfsz4h39bfMWskdQN7Ao8CSJpANkT+LyPi14MF29k2/lvIpnIwM6uEvrzf5BkbFgEHSZokaRQwE5hXd8w84NR8+STg5ogISS8AfgCcExH/00iwnU38M9m+fcnMrOM1ezhnRGwla1JfANwNXBsRyyVdKOnE/LA5wDhJPcDZQN+QzzOAycB5ku7IXy8eKF6jnbt3sX3nrsjG5b8Q+OtGrmFm1hnKmXEzIuYD8+u2nVezvAU4uZ/zLgIuKhKr0c7duXXrvcAjwK0RcU+RgGZmba0KD1vPOxEWAQsj4tHyi/Sc3t5etmx5Ikms7u5RSeIAvPSlae9jGzs23WjbvfeekCzWaX+dbsaPy7523uAHNUnK30WAT7zvvclibdrUGS3DHT9lQ972dB2we/nFMTNrbX1z9bTzg1gabeq5k6zzYHV5RTEzaw+tmMyLaDTxXwB8TtL5ZNMwb9f+EhEbmlwuM7MW1ZpP1Sqi0cT/g/zrdTx/dE+QTdVsZtb5AqJ38MNaWaOJ/5hSS2Fm1kaq0tRzH/BA1L1bSWL724zNzDpelRL/vsC6uu0vzPe5qcfMKqETnsDVaOLva8uvtxuwpXnFMTNrcdHhiV/Sl/PFAP5Z0pM1u7vIHh5wR0llMzNrQa35APUiBqvxvyL/KuAQ4Jmafc8AS4HPNhosn0Xu68ChZB8mp0XEzxourZmZ7bQBE39EHAMg6RvAmRGxaSfjfQn4UUSclE89OnYnr2dmll4nN/X0iYj37WwgSXsCRwPvza/5DNv/BWFm1hZix0+ibQspH7Y+iWxGz29Iul3S1yXtmjC+mdlOi2j/uXpSJv5u4AjgaxFxONm0D+fUHyRplqTFkhZv2/ZswuKZmTUiiOgt9Go1KRP/GmBNRCzM1+eSfRBsJyJmR8TUiJja1TUyYfHMzBrjGn+DIuK3wAOSDs43TQNWpIpvZtYs7Z74G72Bq1n+FrgyH9GzCtjpTmMzs9RaMZkXkTTxR8QdwNSUMc3Mmimrxbdeu30RqWv8ZmbtzzV+M7Nqafdx/E78ZmYFuY3fzKxinPjNzCrFnbtmZpUSnT4fv5mZPZ8Tv5lZxTjxl+jpp5+kp2dpklgvecmkJHEApr35HcliAfznNbOTxRo9Ot0jFr757xcli9XdPSpZrK1b085WPmZMuklyP3zep5PEmfPFf0wSp121dOI3M2s94Ru4zMyqJvCoHjOzSnEbv5lZhXg4p5lZ5bTmHPtFOPGbmRXkO3fNzCrGNX4zs4px4jczq5Jo/3H8yR62LulgSXfUvDZJOitVfDOzZgiyB7EU+ddqktX4I2IlcBiApC5gLXB9qvhmZs3izt2hmQb8OiLuH6b4ZmZD5OGcQzUT+M4wxTYz2ylO/AVJGgWcCJy7g/2zgFn5WrJymZk1yom/uOOBpRHxcH87I2I2MBtgxIiu9v7umlnHyQb1uI2/qFNwM4+Zta32b+NPNpwTQNKuwLHAdSnjmpnZc5LW+CPiCWBcyphmZk3X5jV+37lrZlZQK96UVUTSph4zs04QEYVejZA0XdJKST2Szuln/2hJ1+T7F0qamG8fJ+kWSZslfaWRWE78ZmaFBBG9hV6DyWcz+CrZqMcpwCmSptQddjrwWERMBr4AXJJv3wL8A/CxRt+BE7+ZWQF9T+Bqco3/SKAnIlZFxDPA1cCMumNmAFfky3OBaZIUEU9ExE/JPgAa4sRvZlZQCYl/PPBAzfqafFu/x0TEVmAjQxws485dM7OChjCOf29Ji2vWZ+c3qw4LJ34zs4KGkPjXR8TUAfavBfavWZ+Qb+vvmDWSuoE9gUeLFgTc1GNmVlBA9BZ7DW4RcJCkSfl8ZjOBeXXHzANOzZdPAm6OId5C7Bq/mVlBzR7HHxFbJZ0BLAC6gMsiYrmkC4HFETEPmAN8S1IPsIHswwEASauBPYBRkt4GHBcRK3YUT60858TYsXvEwQe/KkmsvfeekCQOwF133ZYsFsCMv5iVLNaqFfcki/WGtx+bLNY3PvfZZLHGjavv0yvXokXzk8V6zWveliTOL395K5s3P1bK9L677LJ7TJ58eKFzli37yZJBmnqSco3fzKygVq4wN8KJ38yskPC0zGZmVeMav5lZxTjxm5lVSN+UDe3M4/jNzCrGNX4zs0LCD2IxM6uaoL1H9aR+5u5HJC2XtEzSdySNSRnfzKwZyngQS0rJEr+k8cCHgakRcSjZbckzBz7LzKz1tHviT93U0w3sIulZYCzwYOL4ZmY7qTWTeRHJEn9ErJX0WeA3wFPADRFxQ6r4ZmbNkA3ndBt/QyTtRfbosEnAfsCukt7dz3GzJC2WtHjr1mdSFc/MrGHt3tSTsnP3T4H7IuKRiHgWuA74k/qDImJ2REyNiKnd3aMSFs/MrDHtnvhTtvH/Bni1pLFkTT3TgMUDn2Jm1mo8jr9hEbFQ0lxgKbAVuB0YtmdOmpkNVbMfxJJa0lE9EXE+cH7KmGZmzdbunbu+c9fMrIBOmKTNid/MrJDW7LAtwonfzKwgJ34zs4px4jczq5h279z1g1jMzCrGNX4zsyLCN3CZmVVK4Bu4zMwqx527Jert3cYTT2xKEuuJJ1YkiQOwceP6ZLEA7r799mSxTv7b5024Wprrv3ZtslirVt2ZLNaJ73xfslgAXSO6ksX62c/+I1msMrV7525LJ34zs9bjG7jMzCrHid/MrEI8V4+ZWQU58ZuZVUqAO3fNzKrF4/jNzCrGTT1mZhXT7ok/6SRtks6UtEzScklnpYxtZtYMEUFEb6FXq0mW+CUdCnwAOBL4Y+Atkianim9m1ixZ8m/81WpS1vgPARZGxJMRsRX4MfDnCeObmTWFE3/jlgGvlzRO0ljgBGD/hPHNzJqi3RN/ss7diLhb0iXADcATwB3AtvrjJM0CZgF0d49KVTwzs8a1YDIvImnnbkTMiYhXRsTRwGPAvf0cMzsipkbE1K4uDzoyM2u2pJlV0osjYp2kA8ja91+dMr6Z2c4LgtYbqVNE6ir19ySNA54FPhQRv0sc38xsp3iStoIi4vUp45mZlcGJ38ysYpz4zcwqpTWHaBbhxG9mVlArTsNQhBO/mVkB7tw1M6uiNk/8SW/gMjNrf1H4XyMkTZe0UlKPpHP62T9a0jX5/oWSJtbsOzffvlLSnw0Wy4nfzKygZk/LLKkL+CpwPDAFOEXSlLrDTgcei4jJwBeAS/JzpwAzgT8EpgP/ml9vh5z4zcwKKmGStiOBnohYFRHPAFcDM+qOmQFckS/PBaZJUr796oh4OiLuA3ry6+2QE7+ZWUElJP7xwAM162vybf0ek09tvxEY1+C522npzt2nn35y/a9+tfj+gqftDawvozzDHGvI8X7yk+92ZKwhavlYX7xoSA+na4vfxYSxDiyjILkFZGUqYoykxTXrsyNidhPLVEhLJ/6IeFHRcyQtjoipZZRnOGOljudYjtUq8VK/t8FExPQSLruW7Z9PMiHf1t8xayR1A3sCjzZ47nbc1GNmNvwWAQdJmiRpFFln7by6Y+YBp+bLJwE3R9aONA+YmY/6mQQcBPxioGAtXeM3M6uCiNgq6QyyZqQu4LKIWC7pQmBxRMwD5gDfktQDbCD7cCA/7lpgBbCVbObj5z3kqlYnJv6U7Wap2+ha8r1J+hhwRkRMzNcvAE6KiEObHauBsiwD5kbEBWXFKvD+WvLn1Ybxhq0tPKWImA/Mr9t2Xs3yFuDkHZx7MXBxo7HU7rce2/DrJ/HvBoyOiEcbPD+AkyNibhPKMlji32lD+GAzaymdWOO3IZA0Kh8/vNMiYjOwuRnXMrPmc+duB5J0q6R/k/QlSY/lr3+RNKLmmNWSLpB0maTfAVfm28dLurrmvB9IOqju+h+X9FtJmyV9E9itbv8Fec27dtupku6S9LSkhyVd0VeO/JDvSoqadSS9VdISSVsk3Sfp4rzjq2//iyX9p6SnJN0v6bRBvi8vy2O8om77LEnrJY2U1CVpTh7vKUm/yt/91vDfAAAEo0lEQVTvDv+vSLpc0vcb+B68T9KK/P3cK+kjdT+TD+bbt+TlWZCP3jBrKif+zvUusp/va4APArOA+gHiZwP3AFOBT0gaC9wCbAHekJ/7EHBTvg9J7wAuAs4HjgBW5tfZIUkfBC4FvgH8EXAC0JcUX5V//QCwb996Pt/IlcBXyG5FP41sJMM/1Vz6cmAy8KfA24C/BCbuqBwRcS/Z6Il31e16F3BtRDxL9j1bC7wDOAT4JPAJ4H0DvcfBSPpAXvbz8ut+FPg74G/y/VPJbtn/FHAwMA340c7ENNuhoneg+dX6L+BW4F7yPpx8298Da2rWVwP/VXfeacCv6s7rIhsr/I58/X+Bf6877yZgdc36BcCymvU1wKcHKG+QtZnXbrsN+Ie6bW8ja0IS8LL8vNfW7D8Q2AZcMECsDwP381z/1gFAL/AnA5zzaeCmAd7f5cD3686pP+Y3wHvqjjkLWJEv/znZnZi7D/fvj1+d/3KNv3P9PCJqe+5/BoyXtEfNtsV157wSmAQ8njfjbCZLRnsBf5Afc0h+rVr1678n6cVkt4//d8HyvxL4ZF858rJcBewKvCQvRy8145Uj4n7gwUGuezWwH9D3/OdTgPsi4n9ryvxXkhZLeiSP+xGyD4ghkfQishtsLq17P5/mue/rjWQfSPdJujJvGtt9qDHNBuL2w2p7om59BHAH+fjgOhvKL87zyvIpoL85IB6pWS40LC0i1km6kax557b865V9+yX9BfBF4GNkf91sAj4EvH2Ay/aS/RVSa2TNcl8F66/ya/ZXrsclHQEcDRwLnAv8k6RXRcRgH2ZmhbjG37mOklSbjF4NPBgRmwY4ZylZm/n6iOipe/Ul/rvza9WqX/+9iFhH1mY+bYC4z5I1KdWX5eX9lKMnsgmq7iH7/f39LISSDiCrzQ/m28DJkl4JvCJf7/M6YGFEfCUilkZED8/VynfkEbL+iVqH9S1ExMNkf4n8QX/vp+a4rRFxc0ScS9YXsivwlgbej1khTvydaz/gi5IOlnQS8H/J5vAeyJXAw8B/SnpDfvv40ZI+VzOy50vAqZI+IOkgSecCRw1y3YuBs/JRLC+TdJikj9bsX002xexLJO2Vb7sQeKekCyUdKunlkk6S9BmAiFhJ1vl5qaTXSDqMrK39qQa+N/9BViOfAyyKrNO3z73AEZKOz9/fP5B1dA/kZuBwSadJmizp48Br6445H/h4/j04OH9Pf5l//5D0FklnSjpc0oHAO4HdyT5ozZrKib9zXUlWi14I/DtZkhsw8UfEk2RNDavImljuIZv/ey/gsfyYa8g6Li8GbierMX9+kOt+jay55ANko3l+RDZSp89HgWPIppa9PT9nAfDmfPsv8tc5ZJ2kfd4L3EeWeP+LrA9g9UBlqXmf1wN/zPa1fchGH12bX2sR2Sihzw1yvQVkzVIXA0vyc/617pivk3Wevwe4E/gJ2Uir+/JDfkfWeX0T2ff9Y8D7I+Ing70fs6J8524HknQr2YiSM4a7LGbWelzjNzOrGCd+M7OKcVOPmVnFuMZvZlYxTvxmZhXjxG9mVjFO/GZmFePEb2ZWMU78ZmYV8/8BgS1Tr3FZYpwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# normalize the diagnal cells so that they don't overpower the rest of the cells when visualized\n",
    "row_sums = conf_mx.sum(axis=1, keepdims=True)\n",
    "norm_conf_mx = conf_mx / row_sums\n",
    "np.fill_diagonal(norm_conf_mx, 0)\n",
    "\n",
    "fig = plt.figure(figsize=(8,5))\n",
    "ax = fig.add_subplot(111)\n",
    "cax = ax.matshow(norm_conf_mx, cmap=plt.cm.bone)\n",
    "ticks = np.arange(0, 10, 1)\n",
    "ax.set_xticks(ticks)\n",
    "ax.set_yticks(ticks)\n",
    "ax.set_xticklabels(ticks)\n",
    "ax.set_yticklabels(ticks)\n",
    "fig.colorbar(cax)\n",
    "plt.ylabel('true labels', fontsize=14)\n",
    "plt.xlabel('predicted values', fontsize=14)\n",
    "plt.savefig('conf.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy as web service\n",
    "\n",
    "Once you've tested the model and are satisfied with the results, deploy the model as a web service hosted in ACI. \n",
    "\n",
    "To build the correct environment for ACI, provide the following:\n",
    "* A scoring script to show how to use the model\n",
    "* An environment file to show what packages need to be installed\n",
    "* A configuration file to build the ACI\n",
    "* The model you trained before\n",
    "\n",
    "### Create scoring script\n",
    "\n",
    "Create the scoring script, called score.py, used by the web service call to show how to use the model.\n",
    "\n",
    "You must include two required functions into the scoring script:\n",
    "* The `init()` function, which typically loads the model into a global object. This function is run only once when the Docker container is started. \n",
    "\n",
    "* The `run(input_data)` function uses the model to predict a value based on the input data. Inputs and outputs to the run typically use JSON for serialization and de-serialization, but other formats are supported.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing score.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile score.py\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "import pickle\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from azureml.core.model import Model\n",
    "\n",
    "def init():\n",
    "    global model\n",
    "    # retreive the path to the model file using the model name\n",
    "    model_path = Model.get_model_path('sklearn_mnist')\n",
    "    model = joblib.load(model_path)\n",
    "\n",
    "def run(raw_data):\n",
    "    data = np.array(json.loads(raw_data)['data'])\n",
    "    # make prediction\n",
    "    y_hat = model.predict(data)\n",
    "    # you can return any data type as long as it is JSON-serializable\n",
    "    return y_hat.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create environment file\n",
    "\n",
    "Next, create an environment file, called myenv.yml, that specifies all of the script's package dependencies. This file is used to ensure that all of those dependencies are installed in the Docker image. This model needs `scikit-learn` and `azureml-sdk`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": [
     "set conda dependencies"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core.conda_dependencies import CondaDependencies \n",
    "\n",
    "myenv = CondaDependencies()\n",
    "myenv.add_conda_package(\"scikit-learn\")\n",
    "\n",
    "with open(\"myenv.yml\",\"w\") as f:\n",
    "    f.write(myenv.serialize_to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the content of the `myenv.yml` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Conda environment specification. The dependencies defined in this file will\n",
      "# be automatically provisioned for runs with userManagedDependencies=False.\n",
      "\n",
      "# Details about the Conda environment file format:\n",
      "# https://conda.io/docs/user-guide/tasks/manage-environments.html#create-env-file-manually\n",
      "\n",
      "name: project_environment\n",
      "dependencies:\n",
      "  # The python interpreter version.\n",
      "  # Currently Azure ML only supports 3.5.2 and later.\n",
      "- python=3.6.2\n",
      "\n",
      "- pip:\n",
      "    # Required packages for AzureML execution, history, and data preparation.\n",
      "  - azureml-defaults\n",
      "- scikit-learn\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"myenv.yml\",\"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create configuration file\n",
    "\n",
    "Create a deployment configuration file and specify the number of CPUs and gigabyte of RAM needed for your ACI container. While it depends on your model, the default of 1 core and 1 gigabyte of RAM is usually sufficient for many models. If you feel you need more later, you would have to recreate the image and redeploy the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "configure web service",
     "aci"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n",
    "                                               memory_gb=1, \n",
    "                                               tags={\"data\": \"MNIST\",  \"method\" : \"sklearn\"}, \n",
    "                                               description='Predict MNIST with sklearn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy in ACI\n",
    "Estimated time to complete: **about 7-8 minutes**\n",
    "\n",
    "Configure the image and deploy. The following code goes through these steps:\n",
    "\n",
    "1. Build an image using:\n",
    "   * The scoring file (`score.py`)\n",
    "   * The environment file (`myenv.yml`)\n",
    "   * The model file\n",
    "1. Register that image under the workspace. \n",
    "1. Send the image to the ACI container.\n",
    "1. Start up a container in ACI using the image.\n",
    "1. Get the web service HTTP endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": [
     "configure image",
     "create image",
     "deploy web service",
     "aci"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating image\n",
      "Image creation operation finished for image sklearn-mnist-svc:1, operation \"Succeeded\"\n",
      "Creating service\n",
      "Running................\n",
      "SucceededACI service creation operation finished, operation \"Succeeded\"\n",
      "CPU times: user 4.08 s, sys: 220 ms, total: 4.3 s\n",
      "Wall time: 6min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from azureml.core.webservice import Webservice\n",
    "from azureml.core.image import ContainerImage\n",
    "\n",
    "# configure the image\n",
    "image_config = ContainerImage.image_configuration(execution_script=\"score.py\", \n",
    "                                                  runtime=\"python\", \n",
    "                                                  conda_file=\"myenv.yml\")\n",
    "\n",
    "service = Webservice.deploy_from_model(workspace=ws,\n",
    "                                       name='sklearn-mnist-svc',\n",
    "                                       deployment_config=aciconfig,\n",
    "                                       models=[model],\n",
    "                                       image_config=image_config)\n",
    "\n",
    "service.wait_for_deployment(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the scoring web service's HTTP endpoint, which accepts REST client calls. This endpoint can be shared with anyone who wants to test the web service or integrate it into an application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": [
     "get scoring uri"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://23.101.137.78:80/score\n"
     ]
    }
   ],
   "source": [
    "print(service.scoring_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test deployed service\n",
    "\n",
    "Earlier you scored all the test data with the local version of the model. Now, you can test the deployed model with a random sample of 30 images from the test data.  \n",
    "\n",
    "The following code goes through these steps:\n",
    "1. Send the data as a JSON array to the web service hosted in ACI. \n",
    "\n",
    "1. Use the SDK's `run` API to invoke the service. You can also make raw calls using any HTTP tool such as curl.\n",
    "\n",
    "1. Print the returned predictions and plot them along with the input images. Red font and inverse image (white on black) is used to highlight the misclassified samples. \n",
    "\n",
    " Since the model accuracy is high, you might have to run the following code a few times before you can see a misclassified sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": [
     "score web service"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHEAAABJCAYAAAC6jrqSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJztnXd4FcX6xz8TAgQIQRASqpQLghekSVOKIFKkKUFApPcmRQWpP6Uq0luCgHTEABoQkN5BgdBFIJGmlGukCASSEEr298dmx3NOTkKAnN3NvfN5nvMkZ88h+2Vmdnb2nbcITdNQKBQKhUKhUCgUCoVCoVDYGy+rBSgUCoVCoVAoFAqFQqFQKB6PMuIoFAqFQqFQKBQKhUKhUKQBlBFHoVAoFAqFQqFQKBQKhSINoIw4CoVCoVAoFAqFQqFQKBRpAGXEUSgUCoVCoVAoFAqFQqFIAygjjkKhUCgUCoVCoVAoFApFGkAZcRQKhUKhUCgUCoVCoVAo0gAeN+IIIYoLIb4RQpwWQtwWQsQIIcKFEJOFEHk8ff5kdPkKIYYKIU4IIe4IIa4LIX4WQnQQQgiLNNmyrRK05RBCTBRCnBVC3BNCXBNC7BBCVLdAy4tCiFFCiP0JOu4IIY4JIYYJIbKYrcdFm63GlV3bSggRIIT4SghxSQhxXwhxUQgxTQjxnFWaEnQNEUKsFEKcF0JoQojfLdZjy/5zRQiR2aHNZlqoQ0viddcqTQjxIkKMQoj9CHENIe4gxDGEGIa116CtxnqCphHJ9KEmhHhgA42Wj3XbrhWEGIEQWjIvS/rPbvOCXfvPrvcbu62rXLTZaW1su/nTxmPdrrpsNVclaFJrhZTrMnUO9U7tP+iG/EAeYBVwGXgIvAx0A94TQpTVNO2qCTokQggvYAPwGrAImAFkBloBC4CXgEFmakrAdm0FIIQoCOwEfIF5wG9ANqA0kM9sPUAnoDewBvgGeADUAsYALYQQVTRNizVblE3Hle3aSgjhDxwA8gKzgV+BUkBPoIYQoqqmaTFmanLgc+Bv4AhgqUEpAdv1XxKMAnJZLSKBPcAcl2NWPvwn24cIUQVr+tBuYx0gFDjr5nhpYCCw1lw5brHDWLflWgF795+d5gW79p/t7jc2XVcZ2uy2Nrbj9WfXsW5XXWCvuQrUWuFJMHcO1TTNkhfQHNCATyw496sJ557icjwDcB64ZVW72K2tEs6/B7gE5LG6LRL0VACyuTk+JqGdPrBIl+3GlR3bCpiacO5WLsdbJRwfbkX/JWgo4vD7r8DvVmmxa/+50VIefQH0UYKmmRZq0YCFVreJ0wsqaG76UIMxGmiadfOVrcb6Y7TOTujbhhbrsM1YT0KfpWuFJF8wO2GsW9J/tpwXbNh/drzf2HFd5aDBVmvjZHTaYv500WTLucpqXXacq9Ra4YnOb+ocamVOnD8Sfma34Nx+CT//43hQ07T7wHUg2nRFyWNZWwkhagDVgPGapv0phEgvhMhstg5HNE07pGnabTcfLU/4WcpMPQ7YblzZtK1qAbFAiMvx5cA9oKPpihLQNO28Ved2h037TyKESAfMBTai74zYAiFEBiGEr9U6ANC0Q9iwD+021pMiwQX5PfTd0o0W6rDlWHfBynWVe2zSf7oUG80L7rG0/2x6v7HdugrsuTZ2h13mTzfYb67SsYUuO81Vaq2QcsyeQ00z4gghfIQQOYUQ+YUQddGtZQDrzdLgQBhwC/hECNFcCPGCEKKEEOIL4BVghAWaJDZrqwYJPy8KIdaiP3xHCyF+E0K0sUBPcuRP+PmXRee39bhywcq2ygjc0xLM0waapsWjj68iQoicFuhKS1g91g0+BEoAH1isw5F3gRjgjhDiqhBihhAim9Wi3GCXPrQ7zdEf5BZqmvbIQh22G+s2Wyskhew/rO0/280LaaT/wNq5yq7rqrSyNrbF/GnXsW5TXbabq9IIthjrSeCZOdREF6MP0F2JjNcFoLUV7k4JeqoDES6aooB3rNJkx7ZCjxfVgKvAT0BrdE+JXxOOd7S6vRJ0pgN+Ro8/LK7GlX3bCvg+oV3Kuhwv69Bm5W3QTrZ0G7W6/xx0FEbfBR2U8L4Q1odTHQAGAO8A7dC9vTTgF8DX6r6TL0inwc8aPNAs7EOHdrPlWE/QtgeIBwpbqMF2Yz1Bh23WCkm+YI8G8Zq1/WfLeSEt9J8d7jd2XFelobWx5fNngg5bjnW76bLrXOWgT60VnlyXx+ZQMxIbG6wGwtETgJUDmgBW7rbfRR+Ma9AbNwd6MqJlQoi3NU3bYqE2O7VV1oSfd4Bamu7CihBiNXo88udCiEWa7kVhJVPRY6eHapoWYaEOO48rA6vbair6DWqFEKI/enuVTDj+AEiPnrhQ4R6r+8/gK/Q5YLKFGpzQNK2yy6HFQohfgLFAv4SfdkD2Idb2oa0RQhRHD1nYpmnaBQul2G6sJ2CntUJiHPoPC/vPxvOCvftPxw73Gzuuq2y/NrbR/An2Heu20mXjucrW2Gysu+K5OdRCy1RpIA4YYsG5X0Z3fezhcjwz+uT7O5DOqraxWVutRbcCj3Hz2aKEz16yuH1GJ+iYbbEO248rG7VVc+BP/tn9eIj+oBSa8L60lfoSNNpux8FG/dcGfcejmsOxQtjAO8GN1vQJ8+fPVmvRNA0NRmt6kldL+9CljWw31hN0jU8YU+9ZqCEtjXXL1gpuXzA+Yaxb1n/JtJW95gUb9p8d7jd2XVelkbWx5fNnMtpsNdbtrMtOc5VaKzyxLo/OoZYlNtY07RfgKNDLgtN/CPgAK100xQA/AgXRF2m2wOK2upzwM9LNZ38m/LQsAZgQYgQwHL3UZA+rdCRg63Flp7bSNG0leoxoOaAGkFfTtB4Jxx7ivnTg/zR26T8hREZ0j4T1QKQQoqgQoij6+AbIlnDMFqUoNU17gJ4U0/pdP5v0YVpACOGN7k5+Az10wQoNaW2sW7lWcMYG/ZcctpoXErBT/9nlfoN911V2Xxvb/fqzzVh3xI667DhX2Qm7jnUz5lArq1MBZEJ3izSbfAk/07n5zNvlp12wqq3CEn7md/OZceyqSVqcSLhAPkPf9eiiJZg9LcS248qGbYWmaY80TTumadoeTdOuCiFyoxt1diUs0BQJ2Kz/MgG5gIbAGYfXzoTP2yS872KFOFeEED7oc5W1CYRd+hAbXIM2pzEQACzVNC3OIg1paqwnYNVawRXZf1jXf0lim3khMZb3n83uN3ZdV9l2bZyAHebPx2H5WE8CW+my8VxlF2w31s2aQz1uxEl4MHN3vBZ6qa39ntbghlMJPzs4HkzYTXsbuIkFngA2bavV6DG/bRzL3Qkh8qDnNflN0zQr2upT9AtkCdBJsz4nD9h3XNmxrZwQQngB09EXairm1wEb9l80ejic68vYudqY8H6NmaKEEM8n8dFo9EX+WhPlOOPSh1jfh2mBzgk/51mowa5j3Y5rBVfs0H+2nBfs3H82vN/Ycl2FTdfGDtjl+rPlWLejLjvOVWkEW4x1AzPnUOFpA7sQYhWQB9gO/IHuFvkKei33GKCmpmnHPCoisaaCwBF0V8dv0DPL5wC6ortl9tY0LdhMTQm6bNdWCbq6oZfdOwnMBzIAPRO0NtI0bbPJenoDM4GLwP+h5ytw5C/NgkR3dhxXdmyrhAVPGLrb4wUgG9AKfawP0zTtczP1uGhryz+hEn3Qx/qkhPd/aJq2xGQ9tuu/pBBCFELvzyBN00wvwyyEmAJUAXagt5cvehnYWugVH2ppmhZrti5S0IdYM1/Zaqw7IoTIi95eh7XEiR4txwZj3ZZrBQeBsv+wuP/sOC/Ytf/seL+x47rKQZut1sYOumwzf9p4rNtOlx3nqgRdaq2Qcj3mzqEmJPVpAawDLgH30BOUhQMzgBc8ff5kdP0L3c3pMnpFnChgNxBooSZbtlWCtkB0y3Q0+u7DZqCqRVoW4lwS0PW1U40r+7YV+g3gW/SHoHvA38AmoJ5V/eagbafN2sp2/ZeM1kJYmOwVfVd2E3AlYVxFA8eAoYCPZW0DCxOSuyb1sqQP7TbWXbQNTdDR1Uodyeizeqzbdq2gaRoaDE0Y25b3nx3nBbv2n13vN3ZbV7los83a2EGTbeZPG4912+my41yVoEutFVKux9Q51OOeOAqFQqFQKBQKhUKhUCgUimfH6sTGCoVCoVAoFAqFQqFQKBSKFKCMOAqFQqFQKBQKhUKhUCgUaQBlxFEoFAqFQqFQKBQKhUKhSAMoI45CoVAoFAqFQqFQKBQKRRpAGXEUCoVCoVAoFAqFQqFQKNIAyoijUCgUCoVCoVAoFAqFQpEGUEYchUKhUCgUCoVCoVAoFIo0gDLiKBQKhUKhUCgUCoVCoVCkAZQRR6FQKBQKhUKhUCgUCoUiDeD9RF/OnE3zy5XXU1qSxSv6GtevXxeux70zZ9PSPxdApvTpbKMJ7NtWdtRkVf8B3LwYfl3TtFyux1VbpUwTqLZKC5pAjfVn1QSQLpOf5pXZz2xJADz8+wqapqWJtlJjPTF2bCs11lOOHfsP1Fh/Vk2g2iotaAI11p9VE6i2SqkmsGdbufJYI44QohvQDSBj7qLUGTo/FeQ9Oefm9klSU9FuM/h3HvMXG46a3OmyY1s9rabY2FjCwsIoVaoUzz//fKpqsqr/AFb0eO2PpHTZsf/UWHfG7m1lF02gxvrTaHLVld6/CDlqdzVdE8Bf3w5xq8kubWUXTaDG+tNoctWlxrozduw/UGP9aTS50/UsbXXy5EmuX79Ozpw5KVmy5FPrsmNb2UUTqLH+NJrc6VJt5V6TO112aKtk0TQtxa/sL5TQrOKVV17RktLU4qufbaVJs3FbPQ2PHj3SWrVqpQkhtPXr16e6Jqv6T9M0DTiUlC6rSO222rVrl+bl5aV5eXlpgYGB2okTJ1JNk/Zf1lZ20xQfH6/95z//0QYMGKCVLVtWq1Chgvb7778/la7/hbHuSU2apuGdI58GWPZKK22l5vXE2LGt1Fh/9rZSYz0xdmwrT61hXMdtUFCQFhQU9My67NhWaqwnxo5tpdbrz65Js2lbub5UThyFQqFQKBQKhUKhUCgUijSAx404M2bMoHTp0ggh8PLywsvLi3fffZcffviBH374gdu3b3taQiKioqIICgqiQoUKCCEYN26c6RoAfv31V9kmXl5evPfee3z44YcsX76c69evW6LJHd9//z0hISEA1KtXz2I1cPfuXbZu3UrPnj0ZMGAAEyZMYPz48YSHhxMeHs7Dhw+tlmgbtm/fTv369RFCIIRg9erVjBgxgvv373P//n2r5SmSIDIyksjISJo0aULevHmZOnUqERERHD16lPLlyxMWFkZYWJjVMpNl3LhxFClSxGoZiqfk9u3bVKlShQYNGqg51WYsWbKEPn36IISgf//+/PHHH4//RwrFUxAREUGzZs1o1qyZXEcIIWjWrJmpOu7evcu8efMYNWoUvXr1kjq8vLwYMWKEqVocCQwMdHrfu3dvevfubZEae/DgwQOioqKIiopi3LhxjBs3joIFC5IuXTqEEKRLl06+1q1bZ7VcAK5du0aRIkWctO3cudNqWQpFsjxRYuOnIX369BQrVoxixYoBcP/+fUJDQwkNDQVg/vz5dOjQwdMyANi3bx8AnTt3Jjw8HAAhBCtWrKBv375kzpzZFB0G+fPn57XXXpO6Vq5ciaZpTJ8+nSxZslCtWjVq1qxJjx49APDzsyYO1eirLl26IMRj8yx5hLCwMGlImjZtWiIdmqYxdOhQAN59913mzJljWXvZAWMRsXjx4kTGmtWrV9OlSxcA5syZg4+Pj+n67Ey3bt1Yu3Yt4eHhZMuWzfTzX7p0ibJly8r3CxcupGHDhuTMmZP9+/fTsGFDGjRoAMD69eupVKmS6Rofx7fffsuQIUP44osvrJZiCwoVKkTdunVp1qwZdevW5eHDh3Tr1g2ApUuX8uDBA4sVJubPP/8kXbp0nDp1ihkzZvDhhx9aLel/np9++gmAjh074ufnR+7cuZk+fTrLly/nl19+IVeuRDk/TaVEiRIUKlSIIUOGkCtXLk6dOuW0ITV16lS59vpfQ9M07ty5A8CiRYuYNWsWp0+fdvrOsGHDABgyZAhZsmQxXaNBcHAw27Ztk2s/d4SGhhIREUHx4sU9rmfhwoWMHDmSixcvymNeXvoedJkyZXjjjTc8rsEdzZo1S9RG/6vj2+Dbb79lyZIlbN68GdDHvbFeN4xujuv3d955h7Vr1/LWW29Zotdg8eLFXLx40UnbunXrKF68OHny5DFFw6VLl1i1ahUnTpxg/vz5aJpG+fLluXLlCqA/u7Zv314+T9uJ+vXrs2PHDg4dOsTLL79stRxbcOnSJdauXcsnn3xCTEwMAC1atCBPnjwIIejbty+FChV6pnN43IjTo0cPaYQA/YI+d+4c7du3B6BXr17kzp2b+vXre1THrVu35EQfFxfn9NmxY8fo3Lkz3377rUc1uPLcc8+xY8cOrl27Jo/duHGDTZs2ERISwq5du9i4cSM3btwA4MsvvzRVH+iDcNeuXYBuRLLCiDNlyhRGjx5NVFRUos8KFCjA1atXuXfvnjz23Xff8cknn1C+fHkzZUqOHj3KokWLALhy5Qrff/89AK+++irt27enTZs2HjcYGmMmNjbW7efLli0DYNKkSZYZcSIiIihRogSg72YZ7WQlly9fZtmyZcTExLBo0SL69u1r6vnj4uLo1auXHLvLli1zejCrUqUK27Ztk5936dKFgwcPkjFjRlN1Pg7jYaRGjRqmnTM2NpZz585x8uRJDhw4gKZp3Lx5k8WLF8vvGAvKdu3aUalSJRo3bkyBAgU8qisgIIBp06bRuHFjYmNjWb9+PeXLl2fevHkAVK9enZ49eya6L1mBYUwaPnw4s2fP5u7duwCptsA+deoUixYt4vLly3IhaswBjRs3tvTB9XHcvHlTzqfZsmWzROuWLVsAiI+PZ+/evRQsWJDcuXPz119/sXnzZlq3bm2qnly5ctG0aVO6dtUTIJcoUYLMmTPL6+zFF19ECIGmaQC0bt2aihUr2vZBNyIigtGjR3Pv3j1Gjx4NwEsvvfRMf/Phw4fs27ePH3/8kfHjxzt95rqe+vzzzwHdC2769Ommr7cc78kpoUSJErJvUxvj786fP5+ePXsSHx9PhQoVaNasGXny5KFx48YA+Pj4WLaGcTXgBAYGmmLUunbtGpUqVeL3338HoGfPngBUqFABcDactGrVikyZMnlcE8DZs2f56KOPnJ5pUkKzZs1Yvny57FMzuXbtGl9//TWjRo1K9NnUqVNZtWoV586d89j5Dxw4wJAheiL3Xbt2yflSCEFAQAAFChTg6NGjgO7dvGLFCho2bMiUKVM8pulpEELw6NEjJk6cKJ9/zOT27dsMGzaMNWvWAPDbb79ZNi+cPXuWFi1aEB4ezr1796TXIPzjrCGE4Ndff2XZsmXkzJnzqc/lcSOOK0IIihYtStWqVQHdO2bnzp0eNeLcu3ePChUqOC2Sy5cvz7vvvgvoF+qKFStk1aWZM2d6TIsr6dOnJ2/ef0qY5c2bl5dffpkBAwawf/9+J08dKxgyZAiRkZF4e3vTvXt3088fFhaWyICzceNGSpUqBUCWLFmIjY1l8+bNdOzYUX7nm2++scSIs27dOjp06MDNmzcB55vp/v372b9/P4MHD+by5cseNeS4Wur//e9/c+rUKY+dL6VEREQAMHToUKcFUHI7fmbi5+dHtmzZiImJITw83Kn/zGDTpk3s2LGDEydOALjdWS9btiwffPABoIerbtq0iSZNmpimMTl+/vlnAC5cuEDhwoXldepJDANugwYN2L17tzyuaRqVKlWiWrVq8ljx4sUpVqwYZ86cISQkBF9fX9q1a+dRfSNGjKB+/fpER0ezYMEC+vbtS758+di2bRsAHTp04Pz584wZM8ajOlKCMdavXLkiDTiBgYEULVo0Vf7+p59+yurVq+V7x+urXLlyLF261JSHoCfh4cOHTJgwgenTp8uHk08//ZRPP/3UdC2GUblBgwa89NJLCCGoUKECu3btYuvWraYYcXLlysUrr7zCkCFDqF69ulMfXrt2jblz5zoZaapXry6vwYIFC9KvXz/50Gkmjx49YtWqVWzZssXJ09Fg9OjR3Lp1S3quGkaEZ91cGDVq1BNf20FBQYwZM8Y0T1DjvuzOgBMYGCiNS8a1afS3ayhRanL27FlA94wVQjBr1ixpLLQD7sLJjHbyNBcuXODixYvSG2n27NlOPx2vyf79+3Py5EkAj29Y+Pn5uTXgFChQQF7zVapUoV+/fnKNA/rm1ezZs0014kRGRnLx4kXatGnD+fPnk/yeJ1N+9O/f38lYa8w5mTNnpmLFirz//vs0adJEjvuaNWty5coVunfvzvr166VHtl1Inz49//d//2fqOa9evUrPnj1Zv36902b+0aNHefXVV03REB8fLz1tNm/eTKtWrRKFoBvX6saNG/H39+fjjz9m27ZtDBw4kAULFjz1uU034hg47hx7OjfHw4cPnS7S999/n/nz55MhQwZAvzCqVasmrYdmGnGS48cffwTghRdeMP3chuV5w4YNAJQqVcp0F77o6Ghq165NbGws3bt3l7tYrjugfn5+NG3a1MmIYyaGcbB9+/aEhoby6NGjZL9/+/Zt3nrrLfr168fbb79NunTpUl3T8OHDAeSiPkeOHE7GQitw3eULDAykdu3abt22je+a7aHj5+fH7NmzadKkCV999RUjR440NUThpZdeomHDhhQuXDjZ77Vt2xawnxFnyZIl8vd69eqZEtL48ccfA//sYnl5eTFt2jTeeecdcufOLW+eVrFjxw6CgoL49ddf5bErV65Qq1YtQB/r/fr1s9yIs2XLFuk1+8cff5AvXz7Wrl1L8eLF8fZ+9qXC33//zdatW5P8/OjRo5QrV47s2bOTK1cuatWqxd9//y0fCvr27etxj113vPrqq3In1ODrr7+2xIiTI0cOAKcQyuPHjwOJ74ueYvHixdStW1dWxwgNDZWGuT179jiFvIAesmuEDs6aNYumTZuabsS5evUqvXr1YvXq1Uka5o3jDRs2JEOGDNITxxMULlzYaSf98OHDHj3f4zAMyq6Eh4cnMqo6Gi88abRw9KCcPn26bQw4xv/fdc0SFBRkmgE6T548+Pv7c/Xq1cd+Nzo6WnqZzJ0716O6/P39Wbt2LR988AE1a9YE9E0MV+NRlSpV+OWXX+R7TdOIj4/3qDbQnwWN+3DTpk0ThU+ZzS+//OLkpdGkSRNKlSpF3759ndadjsaaYsWK2SaPkMG1a9cIDw9n7NixqbbhkxJWr15Nhw4dpKGtUKFCrFy5EsCtod5TxMTESIO7o7f3jh075P3Q2MioXLkyvr6+TJw4kXLlyrFjx45nOrclRpw1a9Ywf/4/tdcdd0rNYOLEidKAA/qE0rNnT7766isAzpw5Y3nM4ddff82kSZPIli2bJUnbjJuz4VFiRcxx+vTpqVixIrt27WLYsGHJLlKDg4Od3HrNWuzHxcVJr7IjR44ghCBz5sx07tyZV199lSpVqnDs2DFAX+AaC7c9e/Zw584dGjVq5BEjjmEkNYwmVodquDPgOBpnXBdESS0qzeDNN9+Uv+/evdvUBI7FihVj+fLlj/1e6dKlAciZMyeHDx/m0aNHHhlHjkRFRTFo0CAABg4c6DZp8aZNm+TvZoVSVa5cGdBzYTVu3JiuXbvSq1cvU86dElasWOH2uPFA7u3tbfn1GRcXx5AhQ5wS5P7rX//C398/1UL1cuTIwfLlywkODpYbFK7cv3+fv/76i8jISCejF+g7XGYmWP7hhx+Af+Z1R6zKweHKuXPnpMfUCy+8INsnNYxu7hg9ejR169bl0qVLzJ49m1WrVj1RaJQQwvS8PQcPHqR169bJ7raD7l3VvHlzSpQokaqG31y5cskHm06dOgFQu3ZtKlasKL/zwgsvWGrEMeZLxw0Vd0aJiIgIp3u1WUaLI0eOmHKPexzBwcFuvYaDgoJMvecUKFCAjz/+mClTphAZGfnY7xtriqFDhz52g+hZeeutt5INP/rzzz/ZvHlzojnVjM2Wy5cvy5AzRwICAmjTpg1+fn589tlnTp8NHDjQY3pCQkK4ffs2+fLlA5De+b/88gvbt2/nr7/+omfPnqRPn97p35mdvzUpjA2WatWqcenSpWfO75ISLl++DOgb1Vu2bCFv3rxygzoqKkrOqzdu3JBrLE9z69Ytp+fPefPmsWHDBmnA6dSpk9z08fX1ld9LjVBU0404x48fp1evXvz5558AtGvXzha7yBkzZpQeFGPHjmXhwoWma3jw4AFxcXG0b9+eTZs2kTVrVnbu3GmqZdMdRYsWlZ4drkRGRrJu3Tq2bt1Kt27dqFGjRqotIDNkyMCXX37Jxx9//NgHiatXr8qbgp+fn6mhVMZYBv1mEBYWRv78+eWxggULArpRynH37fjx49y9e9eUicbqh0TDKGO4Xye1g2e4dRuJmWvXrm2COmeMXQ5N06S3hN0wroeMGTPywgsvmLK4DQ4Olobutm3bJjLinD9/ngsXLsj3hnHF0xjhUMHBwcTExFjucZZSDKOmj48P0dHRlmrp1KmTNDaDfp0uWbIk1XMt1atXj5o1azJu3DhmzZr1xLkT9uzZQ/Xq1VNVkysPHz7k7Nmz0tstPj6eDBkyMGzYMObNm8fly5eT9Sgyi23bttGoUSNpuBk8eLAMZ5wxY0aqhk4YOa6GDh3KqVOnqFWr1lNV0DS8d8zAyI3Qr18/Ll68SJYsWdi+fbvbhzhP0qdPH/r06fNE/yZbtmyWeBAm5fXqLtzKk6FU8I/3weeff86CBQuoUaOGx0NfkyMiIsJt1anAwEBLNg0GDBhA+/btiYmJkd4Hhw4dAv7xBli7di337t2T95fTp0973IiTHEePHuWdd96RiXoNMmfObIpno7sNldKlS/Ptt9/i7+9Po0aNnD4bPXq0R9NI+Pv74+/v73Ssf//+zJgxAyEE/v7+tG/f3pICGynBeC48d+4cI0aM4O233/bYuR48eMDKlStlgYV8+fJx4sQJp+ene/fuybbq06cP33zzjccJiFMhAAAgAElEQVT0OBIaGuoUEte5c2cyZMggI3wKFSrk1uMrNbzArPUzVygUCoVCoVAoFAqFQqFQpAiPe+KcOHECTdOIiopi8eLFzJs3D03TZIjQsGHDPL6LnClTJoYPHy5zDgwbNoyRI0dKC97q1auZPn26/P7jcpqkJvfu3ZMxcaNGjeLAgQMyJOf48ePkzp3bNC0GMTExTp5Ib775Js8995x8r2kakyZNAvTdPyOWdcWKFWzatIk6deqkmpaKFSs6JSt1xzfffMOUKVOkVbN3794ySbWniY+Pl544mqYxY8YMJy+cmJgYJk+eDOC002DWbqTB1KlTnd7XrVsXgKxZs3r83I47WI/Lb+MaRmWFJ46RHFAIYdsdECPRt7uKbZ7AKBlu8NprryX6zoQJE+Tv9evXdxtu5UnOnj2LEMK0ZHbPgre3t8zlA9aGD4Lu1u24K1StWjWPVTzLmDEjn332Gd26dZP3O0fmz59PmTJlaN68OfXq1eOvv/6Sn4WHh3vcE+fnn392CpcqWLAgDRs2ZNCgQaxatYorV654PEHo49i2bRtNmjQhLi4Ob29vPvzwQ3LmzCkrWFaoUIHJkyenWqJjI+RdCMHq1aufygvH+PdmcPDgQZo2bSrPKYSgffv2svqanbh//z5ff/2107HBgwebcm9OKUOHDnV6b0auuldeeQXQKy/NmjWLvn37Urx4cdM8PF1xbQMDK6tqGqGJAwYMSPRZdHQ0xYoVc0r2+sMPP1iSDPfOnTtMmDCBL7/8kocPHyaaByZNmuSU58tT9OrVS67vWrRoQb169ahSpQqnTp2iWLFi3Lp1S343d+7cdOzY0enZx1MYuY3q16/P8ePHpTdH586dbbkGjY+PZ8yYMTIhb7du3ejbt69Hn+V3797N2rVrZe7aOnXqJDpfhgwZpAdjSsIMU4vy5cvLkDiAWrVq0alTJ15//XW3379165b0ZPzXv/71TOdOdSPOzZs36dOnj8yN8Pfffyd6YPXy8pJukWaEAaRLl46BAwfy22+/AbBgwYJks0Gb6W64adMmudgw0DSN6OhoZs6cyXvvvSddWD0V5+7KlClTZJ/5+fnx0UcfOX0eHBzMJ598Auh92apVK/LkycOUKVNo0qQJERERpiZjNh4ujTwhSYV+eQIhhDQY3bhxQ+bHAb3yWoMGDeSDtuONa9++fZQuXdqy0tAlS5YEMKUE3+MeUGvXrk1QUJBbd2UrKtUY5dntzE8//QToiyOjLz3J+++/D8AXX3yR5HeMUCvAUrf3tEC+fPmoUqWKfO9aetgs/v77b5mnI2vWrMyZMwf4x8jrSfLkycM777yT6LijS7Zj0kfQc8V5KsGpYxUq+KfC3759+3juuef49ddfZUUVozqc2Zw5cwbQF/vGZtOhQ4fkvc+47vr168ecOXNo2bJlqqwbjNCy0aNHP1NyVDPCqR49euRkIDWYNWsWs2bNwt/fn7Jly8rQZiurod24cYMRI0YQHBwMINcSVlTvSopmzZolygNjRhUmI2/l1KlT2bt3LydOnOCNN97g/fff56uvvpJhhJqmcfDgQU6fPg3AqlWrOH78uFw3b9++3eNaQV8XuxZpMIo3WBFutWjRIicDOCBDUczACOGvX79+oqqRrpQvX54HDx4kyv2S2vj6+srQwEePHnHy5Elat24t8585EhkZyWeffUZwcLBHn1OvXr0qU2ZER0cjhGDq1Kn06NHD4+3xtEyfPp1Ro0ZJQ6sxf3mS2rVrP3ZT9/vvv5dGSzPWMAbVqlVLlNA/Ofbs2SNTDzyrsSnVrQIDBw5k2bJlTsd8fX0pWbIkzZs35/vvv2ffvn1yJ3fTpk28/PLLqS0jEVmzZmXevHmAvii7dOlSou8YiXNdE1t5klq1asmyeuvWrZNlcTNkyEBwcDCff/65XDBOmjTJlIvaMY6wYsWKTjl5Lly4wKhRo6R1+qeffuKll14CdCNK+/btiY2N9bhGg0mTJhEZGUmWLFlkvKuZhhEfHx/5YNutWzcaNWrE5s2bOXPmDC1btuT27dvyISRjxoyMHDmSzp07m5ZwC/QH/ZEjRwK6Bd3Ly4t69eqZdn7DMJNUAszixYtTvHjxRLszQUFBHteWHGXLlrW0ckFyOC5My5Ur59FzjRs3DtCN2y1atHD7HeOhwzCAN2zY0KOakmP//v2mJ8t/Ep5//nmn6hK7d++WJWDNJi4uTmqpWbMmzZs3t0SHOw4fPpyo+oonk2WfPXvWqTyqUaXSjJ3YlGK0x6NHj6hWrRorVqwgICBAfm783qpVK5o2bcqNGzecPn9aDM+bpzUuGIY3IcRTe/GklLi4uEQ5N0C/7mrUqEFoaCibN2+WD9Zr1641LVGo4eFsbGxs2bLFaYwbDyl2SVzqzoATGBhoquHL29ubI0eO0KdPH7766ivmz59PZGSkzP9itF9AQADFihXj9u3blC5d2skj+llJKqFxYGCg2zYyCA0NJTQ0lN69e7ut9OVJjDxWVrFq1SpAv7+5rqNc31euXJkuXbowffp0j67f7927J9cmu3btSrJSncG8efNo0KCBR3O9bN++XeYs0jSNcuXKPXEOLTO5e/cu48eP51//+hfTpk2zWo4kPj6ewYMHy/dWbbSkhEuXLqFpGj4+PtIu8bSkuhFn8uTJlCpVSr6vUaMGhQsXJnv27IC+COjcuTMhISEAVK9enYsXL5pSitawAD9uJ8jMDPh+fn7yBmBYzY2yuDdv3mTZsmX0798fgDJlytC5c2fTtAFOD22PHj3i1Vdf5caNG3JH0jDggJ4AWQjBd999Z8oNJCwsTHoEde/e3bIE0F26dAF0g9LRo0cpVaqU0w6I4S43d+7cJN3rPMnJkyfljcrLy4thw4ZZEqY0dOhQt67HERERbl2VrdAIevjEsWPHOHbsmOlhbwbGTf3Ro0eJ5sb79+9LQ2vWrFllKU9PYXi6jR07NskQKcMLxyglbMZ87kq+fPlMTZz6tAwfPtzJeyowMNCSxON//PEHNWrUkOVdZ82aZbqG5Lh//75TX77wwgtOIX2pjas7v+HVYnhJGV5KoD98uyMqKoovvviCLl26sH79egD+85//PNEuXXIYSfJnz55Nx44dk/SyMbwQvvzySxnOaxUlSpSQejRN87gXR+bMmVm+fLmsgFaiRAlq1qyJj48P2bJl4+zZs9StW5ddu3YBeiJRx771FOfPn5frIsdiCI4YG1HFihXjs88+M8372h1JGXCsCh9ynAuMawugatWqjBkzhtKlS5M1a1YePHhgiocxJK6qmRxJrX88wcOHD53aq0yZMgCmJv1/UoOV8TA7c+ZMj21WDx06lJ07d8r3xqam8dnzzz8vDcDGvOnp9UTJkiWdDEnHjh2jdevWtG7d2pLQt8dRpkwZrl69yqxZs5y8ia3m999/5/z583KsO1agththYWEIIZg4ceIzt2Gq3yH8/Pyk0cEdmTJlYsyYMdKIExUVxaZNm0zZATTyuBglyipVqkTTpk3Ztm2bpdUmDKOR6wSbPXt2evfuLRcjixcv9rgRJzY2VpYsdWXBggVcvXqVTz75xMl4A7qF+7333gP0Ur+e5tKlS7Ro0QIhBB999JFl4QiONGnShIkTJzoZcF544QX27dsHYFqeHkdu3LiRyK2wRo0apnqYBAUF0bt3b6cM7oaXjbtqDwZWubkn5V7raaKiovj444/ZtWsXN2/eBHQjTrdu3ejfv7/MjzV37lz5ELBw4UKPGkweV5YX/vHUAZL01DGDpk2bMnToUP7++2/LNDyO4OBgevbsyaNHj6R3glV6Dx48KD0FCxYsaElo59y5c53ygRQpUoSYmBj+85//JGqXcuXKcfLkSY/lxHE1ABpuzm3atElUKWjRokUsWbIk0d8wHgoMz0f4x8M3NTC8Cx4XUmYYW+2QV6V///7Ss+TIkSOm7N5WqFAhySpURYsWpVy5cvzxxx+AedffjBkzkjTeuDJ27FgKFCggjeKexggx2bZtW5IVmMCa/C8XL16kRo0aXLp0SRriHNdYGzZscLrGPLEJm9w6xRVHD2LH8KrQ0FCCg4NNCa1avHgxd+7cke9ffPFFwNzNFcNDePDgwTJX15AhQ/D19UUIIcPfHOfRefPmMWHCBI8ZcWrVquWU/3TkyJEMHDgQLy8vMmTIIDeh4R9vIU+vlV9++WUZHn/jxg2++eYbtm/fTkhICGXKlOGHH36wPAebweHDhzl//jyHDx82tQLw43jw4AGtWrUCkJXa7BaKZuSO3bt3L4sXL6Zo0aKpknbAEjN/4cKFZVlxowykp7l3757Tg/5zzz1HSEgIhQoVomvXrnTt2lW6lV+5csUpSZGVxMbGcvDgQUA3gHmaP/74g0uXLsmJy/CqiomJkWFD7tzUjh49yuXLl+nTp88zJ2p6HJcvX6Z69erSGGdm+FtyxMXFOT0E9OjRw5RY0eQ4e/YsMTEx8n3p0qVN9wbq1atXolhx10VRUFBQou/8L3Hnzh1ef/11jh07RoECBeR1t3PnTr788ktmz55Nq1at+Pjjj9myZYssfe5p43fOnDmpX78+GzduZNiwYTKppKNHjvEwBP88+Jqd1Ng4Z0BAALNnz2bQoEG2CoNJnz493bt3p0ePHsTFxdGvX79k87J5mr1799KuXTvu378P6K7lhresmVy5ckWGRcA/JXIdd0gNVq9eTWhoKF5eXmTKlImDBw8m2kx4Fo4cOeJ2k+TcuXOJckt4eXk5Le5z5szJiy++KHW3a9dO5tSx4lowwi0rVqxo+rkdadq0KU2bNpX3RbvM7465lsza0OjevfsTGbBGjBhB27ZtPbruM7xgH9cvxufNmjUzxZCjaRqHDx8G9BxZkZGRvPXWW0yfPp28efM6GW2OHj3q0fBZw8CVHMnlvnFsW7O8i10N0lZ6p44dO5axY8cm+fmuXbv4/fff5ftZs2YxaNAgj2hp2LCh3CCDxEbuu3fvOhVoyJ8/P2+99ZZHtDjimKy7QYMG3L59m6ioKJo2bUrhwoUZPXo0oOc6syrU8sCBA9SpU4c333zT0jxirsTHxzN+/HjCwsKoXr26JffblLB3715ANyQ2bNiQMWPG4Ovr+8x/1xIjjhDCVLc+0C2IRq4WIQShoaEUKlQIgBw5cvDFF1/IGM7x48czdepUU27uR48elYs9dx165MgRuVPk4+PDpUuXPGqV/e2335z+37/++iuvvvoq3t7eZM+eXSZjcuTSpUu8+eabFChQgPHjx3vcBbhSpUpcvXqVAgUKMGLECFOMW0kRHx/P4sWLAd3d3rHt6tevb5UsmdzL1XV9165dlliojYWfYdQyjDhBQUHUrl2b4sWLOxl2ksqf89/KgQMHOHbsGDlz5uTMmTOyj27dusW0adOYPHmyTMwJerUo8HzuBD8/P5YvX07Lli3ZuHGjNNAWLlxY5lVyTGhsLDY2bNjgUV1JERgYyIwZM5g/f36ihOxWYsT7x8fH8+OPP8oKGWZjeAVOmzaN+/fvkzdvXgYMGCDDdMymQ4cO0nCzceNGedzVSOJ6/N69e5w8eTJVjTjFihVzWwnx+vXrhIWFyY0ngF9++cXpO88//7wplSSvXbsG6N5vuXLlcsoB4EhYWBjp0qWTxl4reOWVV/jqq6/IlSuXfIhLLjG6WcTHx1sSvli0aFFpxLl+/Xoib6rGjRtz/Phx+T4yMpKLFy967IEpODj4iTxMQDdIREREePwhbufOnbz55pvy/euvv87KlSvJlCmTU7UlgNOnT3vUiPO4wgxBQUFOxpvkvJrMevh1TQjvyRDGuLg47ty5Q2xs7FM9mxhazTCqenl5Jeud2Lt3b2k8BH2etSIsJ1u2bGTLlo1Dhw5x4MABWQDA19eXzp07m27ICQsLo379+sTExLBmzRpLn7kciY+P54svvmD48OFkzpyZjRs3ejwdSlhYmIwkcqyKbIQMGxXP3nnnHendHBYWRrNmzQAoUKAAPXv2lGFfz4rX47+iUCgUCoVCoVAoFAqFQqGwGks8cTRNS3FssCfo0qVLomSgjrGiM2bMYOzYsani6pQct27donnz5rJigbvdhKVLl3pUgyulSpXC29tblm+8desWoCeJmjx5MjVr1qRr164yD8bdu3dp1KgR/v7+7Nu3z5S8Cn/99RdCCF577TXat2/v8fMlx5kzZ6QLvmume6PtzCYyMlLufBr5lAxu3ryJr6+vDFW4d+8ef/31F7lz5zal74wdK1e3Y8NKbcTe28Vd88KFC6YkzM6ZMycAHTt2dOqHHDlyMHLkSOLi4mRcOfxTeaJOnToez7Xk5+fHhg0b+Pbbb2Wp8QsXLjh54BgYu5BRUVGWJDeuXLky06dPJywszPRzJ0W3bt3kfPnFF184VUAyk+joaBl6aoTbBAYG0rdvX0v0ABQqVIjVq1cDesjusmXLmDNnjpNHgkGmTJmIiYnBx8eH5557zmO5cVzJmTMnZcuWle8/+ugjp8TUZnHv3j1Z0rVevXoyx58jRg6rmTNnMmDAgFTNx/OkrF+/nueff57Tp0/bwgPHYOLEiU6ego0aNTLlvN7e3slWnXHnIbBs2TKn/EqpQUpDqIxqSq4JjkuUKOHxBMeOIUyZMmVi9erVcvc/Y8aM9OrVy/JQdUciIiLYtm1bsiHhVnoXe8I7/sGDBwD06dOHtWvXEhUVRY0aNShQoAAdO3aUOamS8vw2cuI4hvsDMrfm0xIVFQXo990PPviAihUrpmhtu2nTJqf0Hq+99hqlS5d+Ji2pQeXKleV6pkqVKqxatYrZs2fLCA5PYoThLVu2jKJFi/L111/bxgsH9CTQw4cPJ0OGDBw4cMAUD6URI0awadMmSpQoQdWqVfHy8iIuLk56FIeHhzN9+nT8/f1Jnz49H3zwAWPGjJHXy7Jly6hTp06q6bHEiDN58mSZODR79uxyYeJJ8ufPLycy44HJEdeJxAxiY2OJjIyUD2RGtQSD9evXM3v2bGkYqFWrlscTXBUpUoSiRYvKG85nn31GQEAAL7/8MjVq1KBNmzYsXbqUTZs2yX+TP39+9u3b53F3csNgY+QdcFfRyGxcY30/+OADfH19GTduHIMHD06VxFUp5eeff2bBggWsWbOGGzduuP1OkSJFaNeunbwWLl++zJYtW2jUqBFDhgyhTJkyplV2MIiIiJALH6sqUjnyxhtvyN+PHz9uihGnZMmS+Pv7JwrJA7262IYNG8iaNSuNGjXi0KFDnDlzBtBzXhw6dMiUkvWtWrWSyePOnz/PhAkTpCGncOHCjB07VsZ2W2HAAT2x8tSpU1m5ciULFy40fSw7YoSyTJo0iSxZshAUFGSZAQf0ypCOpekDAwOdklJbhTEX+fn50aNHD3r06EHr1q1Zvnw5AC1btgT0SkuXL18mICCAbNmymTLmQa+S5Wh0tuq+c/r0aZkHrnHjxok+v3PnjgwhzJgxowxtNJssWbKwePFicuXKxbVr12jWrJnH11cVK1bEy8uLpUuXJvtw8+effzJkyBCEEDJcxw73HNAN8675NzyR0yglueccc6h8//33cpPFMUmvJ5kyZYrT+R3vJw8ePHCqTuVpevXqlWzY2eNC0gIDA/n8889tszGVWhiG9vnz58tjmzdvRgjB/PnzZRL2tm3bMnr0aK5fv87WrVvRNI1Vq1bJDUbH0MbSpUvj7+//TLqMe9x3333Hd999x9ChQxk1alSy/2bLli20bNmSu3fvyoTMGzdutCz/jCvGc9/Ro0cpU6YMrVq1Yvfu3R7XZ6QYmTlzpnw+sBNGXsHBgwc7VcX2BEZC7G3btlG7dm2WLVsmbQmPHj2SefMOHTrEvn37mD17NlevXmXw4MEIIZgxYwZAqhpwwANGnPDwcJo1a8a///1vQM88biykHz16REhICCNGjJDf/+mnn0xJRJQrVy4Z2zhx4kSioqJ46623KFKkCFu3bmXixIke1+BKnjx5KF++PKdOnQL0/DOlSpUiLCyMwYMHs3PnTqebaZs2bUzR9eOPP8od9wMHDtChQweEEPj4+Eg9xg5fp06dGD58OLly5fK4roCAAOCfvAhLly6lT58+BAQEJIqDvH79Ovfv3+fevXscPXoUQFrjU2vnLS4uTv5t0BMZT548WSZlMzuxaps2bVJUytZdVZV169axbt062rVr53RTNgMj5jwwMNCUyg2PI1u2bKafM3369GzatInu3bu7rbTWtm1bWbHk7t27Mp9CSEgIFSpUYPHixR7NC+BKkSJFnPKopKSKlRmkS5eO1q1bc+jQIXbs2GFKUkJ3BAQEyJt+lixZmDlzpqVG54iIiEQPPvny5bPUyJUcjg/ixr0mf/788sHATO7fv8/atWsBPbGkVQmzHXM1GJUuQF9XHTx4kEGDBvHzzz8D+rxgVXnqxYsX8/bbb6NpGm3btjXFA+Hw4cMIIRg8eDDz5s1L1EfR0dGsWbOG/v37Ex8fT44cOeSi2oox5cidO3f46aefnNbFBq4e489KREREskYHw+DgiuF1467suKdxzdf1xx9/OCXCNaMaqqZpREREyLwXKcXKkuyexjBKGR6S4Gz8u3TpEqBvdDqOKVePdeNYxowZ+fzzz5/5nuRaWXT+/PkEBgaSN29evL29pfE/Ojqa6OhoRo0axaJFi4iNjeXFF1+UXoN2MeA44u/vT0hICO+//z5du3blm2++8di5QkNDad26NaBXbB0+fLjHzvU0TJo0iZkzZ/Luu+/yySefePx8xpzz8OFD6tat6+QMki5dOpnnt0mTJuTKlYujR4+yZcsW+Z2BAwcCel6j1NzcT/W7/NmzZzl9+rR0lbt79y5TpkwhX758TJw4kdGjR+Pn5ycrLpllnc6cObPMDl2zZk2CgoKcSgE6EhAQYNoCaPHixTJZaJkyZahatSrHjh0jOjpaTnTGrptZ5XsLFy4sF60bNmygffv2aJpGbGwsAwcOxNvbW7rgG4YVMzAeggw38gkTJjBhwgTeeecd0qdP73Rj2LhxI1FRUfKG0bVrV49UsYqPj5c3ro8//ph06dLJygBmVwRIDTdHsxOcOi4q7bLYEUJQtmxZjh07Zup5y5Yty759+5we0AzSpUsnx7evr68Ms8yYMSOLFi3i9ddfJzY21rQkfFFRUcyZMwewR7JS0B+GYmJiZHjZkiVLLDHiPPfcc+zfv19eSxs3bmTIkCFER0ebrgXg1KlTVK5cmdjYWBmy0aFDB9v0W1Jomoafn99jy2l7mjNnziCEwM/PT25uWMG5c+fk7z169CBHjhyEhoYya9Ys6aptePOaFWrmyrBhw2Q1qmnTprF582ZTztupUycWLlzIDz/8wO7du+nTp48sgbtw4UJOnjwpvRdz5MjBnj17TAlHMIiLi6N9+/ZyDer6mTvP2fLly6d6EYKkkvQaa+HHbaJ8//33BAcHe9x7qUGDBtIIPnbsWObMmUNcXBwTJkyQmxzGw7pZRtXixYsTHh4u2zApjyajLY2CDf+tGPeSQYMGORkg3SUmdj3m7jtt27ZNlWIgb7/9NvDPZmVkZKQM7TISAwPs37+f/fv3Sy3NmjVjwYIFphlvbt++DTz5puHrr79OzZo1WbZsGU2aNJGeqqnJhQsXaN26tUwpMnr0aFPSLaQEI2xp+PDhNGrUiOXLlyeqZOkJjPuHpmluK/yePXsW0O+BRpnzHj16MHbsWP7880+5dujQoUOqRtWkuqWiTp06tGjRghUrVgB6nKHhlQN6FYe9e/daMrkZce1z586lc+fO3L171+lz44a5YsUK03Yo8+XLJxc6LVu2ZO/evU4TnI+PD2PGjAEwZaAaGFbGtm3b0rZtW9POmxzGzXrhwoV07NhRHl+9erVb6z7oF1TPnj3JkSOHRyYhxyoq//d//8fIkSPZuHEjQgh+++23VD9fcmzdupUJEybIxQ9A7ty5nUK+OnXqlOS/z507N927d/eoRlccvXDshGEMOXjwoHQlNwMvL68UXeeG51lQUBDHjx/n6tWrps4PwcHBXLhwgcKFC9vCewr06nADBw6kRYsWMu/aw4cPTfVIyJ49O+Hh4eTKlUt6IHzyySeWGXBAX2wblRmNh/2mTZva1gvHQAhBlixZLC+T/dJLL9GyZUsqVqwoHwasoFWrVjIv1tWrV528NHLnzs3SpUupWrWqJdoMD4XBgwfLDYzq1avTrVs3qlevnsiDITw8PFXXFcHBwdy9e5fvvvuOmzdvMmrUKLmJYtyfM2TIQOPGjVm0aJHpYz82NlauiVPKhg0bUn3N4s74YuS+SSlmzPctW7aU65hly5Zx6NAhuZ7y9vZm8eLFMhzOrBLxoBtyjLayy33PagYOHIgQgi+//PKp73OjRo1KtWqSRkj3wIEDmTt3LqB73Tx48IC7d+/KCnEG2bJl44033jB9Xti4cSMfffTRY6+91q1bO4UK3bhxgz///BMvLy9+/fXXVDfi3L17l88++4yiRYvKXDx2yYNz/vx5KlWqBOia5s+fb9q619gcFELw+eefM3LkSHLmzEloaChhYWEyPPDmzZtUqlSJwYMH07hxY9KlS0f27NllWw4fPpwrV67Y14iTMWNGli1bJnMnNGvWTLqvjhs3jjZt2li+eGzevDnVq1dny5YtfPPNN2zevJlPP/1UhiuZkQPDIH369PLG+vvvv9OlSxdWrlxJhQoVqFatGv379/d4Hpy0RuvWrcmbNy+hoaGyVG+WLFn49NNP5XfatGlD1qxZTU3sGBISIvM4AE7GFDPIkycPkydPZvLkyUl+x+pE0K4YXjihoaEIIeSDr5U7WF5eXsycOZM33niDrVu3WqYjJWTJksUppM8sWrRowZAhQ1i6dKll+W9c6devH0uWLGHlypUIIdi1axc3b940JdTTICAgAF9fX6Kjo2natCngnKTTChz//4bbsRkhzP8t+Pj4eNRtPaWULFlSesB6eXnx8OFDZqM1klQAAAVmSURBVM6cycqVK6lVq5ZpOYLcYczb48aNk7l4XnnlFcqXL48QQhpUvLy8iI+P5/r166l6/gwZMjBnzhz69+/PDz/8wNmzZ+U533vvPfLnz0+OHDl48cUXU/W8qY2x0bhz585kSyE/LcWLFycoKEjed5/UgGMWlStXlt7gISEhnD9/noCAAHr27EnXrl09nn/xvw3DM2DAgAGp/rczZszIsGHD6NevH0FBQYSEhHDixIlk/03btm156aWXAL2Yg5+fX6oZLPPkyQPoHsKGt+mePXsYOHCg9OIwmDdvHm3btvV4WWp3tGzZkpYtWzqFOW/evJlffvmFnTt3ynlz165dTgZpY8Pancf2sxIaGsqwYcPw8/Pj0KFDtvG+AT2xf4UKFaSm7du3m7q2M1Ku3L59mzVr1rBmzRon5wHD8WHEiBEMGDAgkUeXsTHsLl3Cs+CRLUovLy/p0mZUObIbuXPntpWXCeiufiEhIbIGvcI9Xl5e1K5dm9q1aycZEmcGGTNmZMOGDdIIZ7jbgb4jb9WuaFrFTgvKChUqMHXqVGkkVDhTpEgR08MFH4e3tzd79+5lwYIF9O/fH9ATFpoZAhMeHs7q1at59913yZ49u2nnTQ4fHx/b3ocVKSddunRMnTrV6ZjZGwWPY+zYsbLyU40aNShevDjdunWTc8WuXbs4ffq0RxJ8+/n5UaVKFapUqZLqf/tZ8fHxoXLlyhw4cCDRZ8WLF6d37960a9dOhi94cne5V69etvciEULQoEEDAPlTkXJcq5I+zqiSGvj6+jJo0CCZE9JOVK9enf3791stwy2O49v4fffu3RQuXFiGrDti5E719/dP9cIpEyZM4MUXX2T8+PG2MeBomsbZs2cJDAwkLi6OPXv2AJjuFWsYaUJCQpgwYQI3b96kU6dO0iPd2Bgz22vJPP97hUKhUCgUCoVCoVAoFArFU2NN+QKF4r+E/Pnzs2jRIkB3C82fPz+vvfYavXv3fuZSif8LGLlw7FiCs1OnTsnmEFLYD19fX3r27EnhwoWZPXt2qpdzTAmtW7eWVR0UT0edOnUQQtgijEnxZBw5csTpZ8+ePa2UYwt8fHzYt2+f1TIU/yNMmDCBDRs2yPLdjyuxrbAXNWrUAJChqWZhxzkqOjqa1157jdq1a7Nv3z7prWgVGTNmdKrUVbJkSQvVKCOOQvHMGO7bRkU2RcqxS0UqxX8P3t7eNGrUiEaNGlktRfGUVK1alapVq3qkoqBCoVD8N1OkSBFZ9luhSMv4+vpy7do1q2XYFhVOpVAoFAqFQqFQKBQKhUKRBhBPkpxSCHEHsKrURkFN0xKlohZCXAOigdQtd5Ay3GoC27aVHTVZ2X+g2iqlpLWxbru2UmPdLXZsKzXWU44dNYEa6ylFjfWUY0dNoMZ6SlFjPeXYUROosZ5S1FhPOWmqrVx5UiPOIU3TzE0JnQLsqEtpSjl21GVHTWBPXXbUBPbUpTSlHDvqsqMmsKcupSnl2FGXHTWBPXUpTSnHjrrsqAnsqUtpSjl21GVHTWBPXXbU5IoKp1IoFAqFQqFQKBQKhUKhSAMoI45CoVAoFAqFQqFQKBQKRRrgSY04czyi4tmxoy6lKeXYUZcdNYE9ddlRE9hTl9KUcuyoy46awJ66lKaUY0dddtQE9tSlNKUcO+qyoyawpy6lKeXYUZcdNYE9ddlRkxNPlBNHoVAoFAqFQqFQKBQKhUJhDSqcSqFQKBQKhUKhUCgUCoUiDaCMOAqFQqFQKBQKhUKhUCgUaQBlxFEoFAqFQqFQKBQKhUKhSAMoI45CoVAoFAqFQqFQKBQKRRpAGXEUCoVCoVAoFAqFQqFQKNIA/w8gmRX/LdJV5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x72 with 30 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "# find 30 random samples from test set\n",
    "n = 30\n",
    "sample_indices = np.random.permutation(X_test.shape[0])[0:n]\n",
    "\n",
    "test_samples = json.dumps({\"data\": X_test[sample_indices].tolist()})\n",
    "test_samples = bytes(test_samples, encoding='utf8')\n",
    "\n",
    "# predict using the deployed model\n",
    "result = service.run(input_data=test_samples)\n",
    "\n",
    "# compare actual value vs. the predicted values:\n",
    "i = 0\n",
    "plt.figure(figsize = (20, 1))\n",
    "\n",
    "for s in sample_indices:\n",
    "    plt.subplot(1, n, i + 1)\n",
    "    plt.axhline('')\n",
    "    plt.axvline('')\n",
    "    \n",
    "    # use different color for misclassified sample\n",
    "    font_color = 'red' if y_test[s] != result[i] else 'black'\n",
    "    clr_map = plt.cm.gray if y_test[s] != result[i] else plt.cm.Greys\n",
    "    \n",
    "    plt.text(x=10, y =-10, s=result[i], fontsize=18, color=font_color)\n",
    "    plt.imshow(X_test[s].reshape(28, 28), cmap=clr_map)\n",
    "    \n",
    "    i = i + 1\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also send raw HTTP request to test the web service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": [
     "score web service"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "POST to url http://23.101.137.78:80/score\n",
      "label: 2\n",
      "prediction: [2]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "# send a random row from the test set to score\n",
    "random_index = np.random.randint(0, len(X_test)-1)\n",
    "input_data = \"{\\\"data\\\": [\" + str(list(X_test[random_index])) + \"]}\"\n",
    "\n",
    "headers = {'Content-Type':'application/json'}\n",
    "\n",
    "# for AKS deployment you'd need to the service key in the header as well\n",
    "# api_key = service.get_key()\n",
    "# headers = {'Content-Type':'application/json',  'Authorization':('Bearer '+ api_key)} \n",
    "\n",
    "resp = requests.post(service.scoring_uri, input_data, headers=headers)\n",
    "\n",
    "print(\"POST to url\", service.scoring_uri)\n",
    "#print(\"input data:\", input_data)\n",
    "print(\"label:\", y_test[random_index])\n",
    "print(\"prediction:\", resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up resources\n",
    "\n",
    "To keep the resource group and workspace for other tutorials and exploration, you can delete only the ACI deployment using this API call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": [
     "delete web service"
    ]
   },
   "outputs": [],
   "source": [
    "service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If you're not going to use what you've created here, delete the resources you just created with this quickstart so you don't incur any charges. In the Azure portal, select and delete your resource group. You can also keep the resource group, but delete a single workspace by displaying the workspace properties and selecting the Delete button.\n",
    "\n",
    "\n",
    "## Next steps\n",
    "\n",
    "In this Azure Machine Learning tutorial, you used Python to:\n",
    "\n",
    "> * Set up your testing environment\n",
    "> * Retrieve the model from your workspace\n",
    "> * Test the model locally\n",
    "> * Deploy the model to ACI\n",
    "> * Test the deployed model\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "roastala"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "msauthor": "sgilley"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}