aml-pipelines-with-automated-machine-learning-step.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/NotebookVM/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure Machine Learning Pipeline with AutoMLStep (Udacity Course 2)\n",
    "This notebook demonstrates the use of AutoMLStep in Azure Machine Learning Pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this example we showcase how you can use AzureML Dataset to load data for AutoML via AML Pipeline. \n",
    "\n",
    "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you have executed the [configuration](https://aka.ms/pl-config) before running this notebook.\n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. Create an `Experiment` in an existing `Workspace`.\n",
    "2. Create or Attach existing AmlCompute to a workspace.\n",
    "3. Define data loading in a `TabularDataset`.\n",
    "4. Configure AutoML using `AutoMLConfig`.\n",
    "5. Use AutoMLStep\n",
    "6. Train the model using AmlCompute\n",
    "7. Explore the results.\n",
    "8. Test the best fitted model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Azure Machine Learning and Pipeline SDK-specific imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "We are currently using version 1.23.0 of the Azure ML SDK\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import os\n",
    "import csv\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "import pkg_resources\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig\n",
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "from azureml.pipeline.steps import AutoMLStep\n",
    "\n",
    "# Check core SDK version number\n",
    "print(\"We are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Workspace\n",
    "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "Note, we have launched a browser for you to login. For old experience with device code, use \"az login --use-device-code\"\n",
      "Performing interactive authentication. Please follow the instructions on the terminal.\n",
      "You have logged in. Now let us find all the subscriptions to which you have access...\n",
      "Interactive authentication successfully completed.\n",
      "quick-starts-ws-140144\n",
      "aml-quickstarts-140144\n",
      "southcentralus\n",
      "6b4af8be-9931-443e-90f6-c4c34a1f9737\n"
     ]
    }
   ],
   "source": [
    "ws = Workspace.from_config()\n",
    "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an Azure ML experiment\n",
    "Let's create an experiment named \"automlstep-classification\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.\n",
    "\n",
    "The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step.\n",
    "\n",
    "*Udacity Note:* There is no need to create an Azure ML experiment, this needs to re-use the experiment that was already created\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Experiment(Name: automl-bank-experiment,\n",
       "Workspace: quick-starts-ws-140144)"
      ],
      "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Workspace</th><th>Report Page</th><th>Docs Page</th></tr><tr><td>automl-bank-experiment</td><td>quick-starts-ws-140144</td><td><a href=\"https://ml.azure.com/experiments/automl-bank-experiment?wsid=/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourcegroups/aml-quickstarts-140144/workspaces/quick-starts-ws-140144\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.Experiment?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "source": [
    "# Choose a name for the run history container in the workspace.\n",
    "# NOTE: update these to match your existing exp name\n",
    "experiment_name = 'automl-bank-experiment'\n",
    "project_folder = './pipeline-bank-project'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                                                     \n",
       "Workspace name                 quick-starts-ws-140144\n",
       "Azure region                           southcentralus\n",
       "Subscription id  6b4af8be-9931-443e-90f6-c4c34a1f9737\n",
       "Resource group                 aml-quickstarts-140144\n",
       "Experiment Name                automl-bank-experiment"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>Workspace name</th>\n      <td>quick-starts-ws-140144</td>\n    </tr>\n    <tr>\n      <th>Azure region</th>\n      <td>southcentralus</td>\n    </tr>\n    <tr>\n      <th>Subscription id</th>\n      <td>6b4af8be-9931-443e-90f6-c4c34a1f9737</td>\n    </tr>\n    <tr>\n      <th>Resource group</th>\n      <td>aml-quickstarts-140144</td>\n    </tr>\n    <tr>\n      <th>Experiment Name</th>\n      <td>automl-bank-experiment</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "source": [
    "\n",
    "dic_data = {'Workspace name': ws.name,\n",
    "            'Azure region': ws.location,\n",
    "            'Subscription id': ws.subscription_id,\n",
    "            'Resource group': ws.resource_group,\n",
    "            'Experiment Name': experiment.name}\n",
    "\n",
    "df_data = pd.DataFrame.from_dict(data = dic_data, orient='index')\n",
    "\n",
    "df_data.rename(columns={0:''}, inplace = True)\n",
    "df_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or Attach an AmlCompute cluster\n",
    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource.\n",
    "\n",
    "**Udacity Note** There is no need to create a new compute target, it can re-use the previous cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Creating...\n",
      "SucceededProvisioning operation finished, operation \"Succeeded\"\n",
      "Succeeded......................\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n",
      "{'currentNodeCount': 1, 'targetNodeCount': 1, 'nodeStateCounts': {'preparingNodeCount': 1, 'runningNodeCount': 0, 'idleNodeCount': 0, 'unusableNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0}, 'allocationState': 'Steady', 'allocationStateTransitionTime': '2021-03-10T09:53:36.191000+00:00', 'errors': None, 'creationTime': '2021-03-10T09:51:27.453394+00:00', 'modifiedTime': '2021-03-10T09:51:42.881348+00:00', 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 1, 'maxNodeCount': 4, 'nodeIdleTimeBeforeScaleDown': 'PT120S'}, 'vmPriority': 'Dedicated', 'vmSize': 'STANDARD_DS12_V2'}\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import AmlCompute\n",
    "from azureml.core.compute import ComputeTarget\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# NOTE: update the cluster name to match the existing cluster\n",
    "# Choose a name for your CPU cluster\n",
    "amlcompute_cluster_name = \"cpu-cluster\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',# for GPU, use \"STANDARD_NC6\"\n",
    "                                                           #vm_priority = 'lowpriority', # optional\n",
    "                                                           min_nodes=1,\n",
    "                                                           max_nodes=4)\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "    \n",
    "    compute_target.wait_for_completion(show_output=True, min_node_count = 1, timeout_in_minutes = 10)\n",
    "# For a more detailed view of current AmlCompute status, use get_status().\n",
    "print(compute_target.get_status().serialize())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "cpu-cluster AmlCompute Succeeded\n"
     ]
    }
   ],
   "source": [
    "# Check details about compute_targets (i.e. compute_target)\n",
    "\n",
    "compute_targets = ws.compute_targets\n",
    "for name, ct in compute_targets.items():\n",
    "    print(name, ct.type, ct.provisioning_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "**Udacity note:** Make sure the `key` is the same name as the dataset that is uploaded, and that the description matches. If it is hard to find or unknown, loop over the `ws.datasets.keys()` and `print()` them.\n",
    "If it *isn't* found because it was deleted, it can be recreated with the link that has the CSV "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "32950 x 21 table of data:\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "   age          job  marital    education  default housing loan    contact  \\\n0   57   technician  married  high.school       no      no  yes   cellular   \n1   55      unknown  married      unknown  unknown     yes   no  telephone   \n2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n3   36       admin.  married  high.school       no      no   no  telephone   \n4   27    housemaid  married  high.school       no     yes   no   cellular   \n\n  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n0   may         mon  ...         1    999         1      failure         -1.8   \n1   may         thu  ...         2    999         0  nonexistent          1.1   \n2   may         fri  ...         1    999         1      failure         -1.8   \n3   jun         fri  ...         4    999         0  nonexistent          1.4   \n4   jul         fri  ...         2    999         0  nonexistent          1.4   \n\n   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n0          92.893          -46.2      1.299       5099.1  no  \n1          93.994          -36.4      4.860       5191.0  no  \n2          92.893          -46.2      1.313       5099.1  no  \n3          94.465          -41.8      4.967       5228.1  no  \n4          93.918          -42.7      4.963       5228.1  no  \n\n[5 rows x 21 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>job</th>\n      <th>marital</th>\n      <th>education</th>\n      <th>default</th>\n      <th>housing</th>\n      <th>loan</th>\n      <th>contact</th>\n      <th>month</th>\n      <th>day_of_week</th>\n      <th>...</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>poutcome</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n      <th>y</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>57</td>\n      <td>technician</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>mon</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.299</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>55</td>\n      <td>unknown</td>\n      <td>married</td>\n      <td>unknown</td>\n      <td>unknown</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>may</td>\n      <td>thu</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.1</td>\n      <td>93.994</td>\n      <td>-36.4</td>\n      <td>4.860</td>\n      <td>5191.0</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>33</td>\n      <td>blue-collar</td>\n      <td>married</td>\n      <td>basic.9y</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.313</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>36</td>\n      <td>admin.</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>jun</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>4</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>94.465</td>\n      <td>-41.8</td>\n      <td>4.967</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>27</td>\n      <td>housemaid</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>jul</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>93.918</td>\n      <td>-42.7</td>\n      <td>4.963</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 21 columns</p>\n</div>"
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "...\n"
     ]
    }
   ],
   "source": [
    "# Try to load the dataset from the Workspace. Otherwise, create it from the file\n",
    "# NOTE: update the key to match the dataset name\n",
    "found = False\n",
    "key = \"BankMarketing Dataset\"\n",
    "description_text = \"Bank Marketing DataSet for Udacity Course 2\"\n",
    "\n",
    "if key in ws.datasets.keys(): \n",
    "        found = True\n",
    "        dataset = ws.datasets[key] \n",
    "\n",
    "if not found:\n",
    "        # Create AML Dataset and register it into Workspace\n",
    "        example_data = 'https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv'\n",
    "        dataset = Dataset.Tabular.from_delimited_files(example_data)        \n",
    "        #Register Dataset in Workspace\n",
    "        dataset = dataset.register(workspace=ws,\n",
    "                                   name=key,\n",
    "                                   description=description_text)\n",
    "\n",
    "\n",
    "df = dataset.to_pandas_dataframe()\n",
    "\n",
    "# Describe dataset\n",
    "m, k = df.shape\n",
    "print(\"{} x {} table of data:\".format(m, k))\n",
    "display(df.head())\n",
    "print(\"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "age                 int64\n",
       "job                object\n",
       "marital            object\n",
       "education          object\n",
       "default            object\n",
       "housing            object\n",
       "loan               object\n",
       "contact            object\n",
       "month              object\n",
       "day_of_week        object\n",
       "duration            int64\n",
       "campaign            int64\n",
       "pdays               int64\n",
       "previous            int64\n",
       "poutcome           object\n",
       "emp.var.rate      float64\n",
       "cons.price.idx    float64\n",
       "cons.conf.idx     float64\n",
       "euribor3m         float64\n",
       "nr.employed       float64\n",
       "y                  object\n",
       "dtype: object"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further review of the Dataset"
   ]
  },
  {
   "source": [
    "#### Featurization\n",
    "\n",
    "The featurization customization is an advanced feature in AutoML which allows to change the default forecasting featurization behaviors and column types through `FeaturizationConfig`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "no     0.887951\n",
       "yes    0.112049\n",
       "Name: y, dtype: float64"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "df.y.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are facing a case of `Imbalanced data`: **88.80% - no** and **11.20% - yes**.\n",
    "\n",
    "**Automated ML** has built in capabilities to help deal with imbalanced data such as:\n",
    "\n",
    "* Apply a weight column to the dataset.\n",
    "* Run an experiment with sub-sampled data.\n",
    "* Review performance metric for imbalanced data such as AUC_weighted and F1-score.\n",
    "* Resampling to even the class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "This creates a general AutoML settings object.\n",
    "**Udacity notes:** These inputs must match what was used when training in the portal. `label_column_name` has to be `y` for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"experiment_timeout_minutes\": 60, # define the duration of the experiment (in minutes).\n",
    "    \"max_concurrent_iterations\": 5,\n",
    "    \"primary_metric\" : 'AUC_weighted'\n",
    "}\n",
    "automl_config = AutoMLConfig(compute_target=compute_target,\n",
    "                             task = \"classification\",\n",
    "                             training_data=dataset,\n",
    "                             label_column_name=\"y\",   \n",
    "                             path = project_folder,\n",
    "                             enable_early_stopping= True,\n",
    "                             featurization= 'auto',\n",
    "                             debug_log = \"automl_errors.log\",\n",
    "                             **automl_settings\n",
    "                            )"
   ]
  },
  {
   "source": [
    "**__NOTE__:** I am using the parameter `blocked_models=['XGBoostClassifier']` in my `AutoMLConfig` because:\n",
    "\n",
    "* AutoML uses `Python 3.6` & `py-xgboost==0.90`.\n",
    "* My Python dev environment uses `Python 3.8.5` which does not support `py-xgboost<=0.90`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Pipeline and AutoMLStep\n",
    "\n",
    "You can define outputs for the AutoMLStep using TrainingOutput."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
    "\n",
    "ds = ws.get_default_datastore()\n",
    "metrics_output_name = 'metrics_output'\n",
    "best_model_output_name = 'best_model_output'\n",
    "\n",
    "metrics_data = PipelineData(name='metrics_data',\n",
    "                           datastore=ds,\n",
    "                           pipeline_output_name=metrics_output_name,\n",
    "                           training_output=TrainingOutput(type='Metrics'))\n",
    "model_data = PipelineData(name='model_data',\n",
    "                           datastore=ds,\n",
    "                           pipeline_output_name=best_model_output_name,\n",
    "                           training_output=TrainingOutput(type='Model'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an `AutoMLStep`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "automlstep-remarks-sample1"
    ]
   },
   "outputs": [],
   "source": [
    "automl_step = AutoMLStep(\n",
    "    name='automl_module',\n",
    "    automl_config=automl_config,\n",
    "    outputs=[metrics_data, model_data],\n",
    "    allow_reuse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a pipeline using the `AutoMLStep` created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "automlstep-remarks-sample2"
    ]
   },
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "pipeline = Pipeline(\n",
    "    description=\"pipeline_with_automlstep\",\n",
    "    workspace=ws,    \n",
    "    steps=[automl_step])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Created step automl_module [55cf9d05][c9ae4379-43a8-407a-8e96-1314f09c1a4e], (This step will run and generate new outputs)\n",
      "Submitted PipelineRun f16a1ce1-6f4c-4839-a6eb-ca81e45a814c\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/automl-bank-experiment/runs/f16a1ce1-6f4c-4839-a6eb-ca81e45a814c?wsid=/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourcegroups/aml-quickstarts-140144/workspaces/quick-starts-ws-140144\n"
     ]
    }
   ],
   "source": [
    "pipeline_run = experiment.submit(pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "8a050b5708d34b9b9fedfb9c6de31371"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "application/aml.mini.widget.v1": "\"ServiceException:\\n\\tCode: 409\\n\\tMessage: (UserError) Workspace is in deleting state.Current Operation cannot be performed\\n\\tDetails:\\n\\n\\tHeaders: {\\n\\t    \\\"Date\\\": \\\"Wed, 10 Mar 2021 11:53:40 GMT\\\",\\n\\t    \\\"Content-Type\\\": \\\"application/json; charset=utf-8\\\",\\n\\t    \\\"Content-Length\\\": \\\"607\\\",\\n\\t    \\\"Connection\\\": \\\"keep-alive\\\",\\n\\t    \\\"Request-Context\\\": \\\"appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d\\\",\\n\\t    \\\"x-ms-response-type\\\": \\\"error\\\",\\n\\t    \\\"x-ms-client-request-id\\\": \\\"6ff8796b-5fb5-48a5-9d7d-d8c8825753ba\\\",\\n\\t    \\\"x-ms-client-session-id\\\": \\\"\\\",\\n\\t    \\\"X-Content-Type-Options\\\": \\\"nosniff\\\",\\n\\t    \\\"x-request-time\\\": \\\"0.076\\\",\\n\\t    \\\"Strict-Transport-Security\\\": \\\"max-age=15724800; includeSubDomains; preload\\\"\\n\\t}\\n\\tInnerException: {\\n    \\\"additional_properties\\\": {},\\n    \\\"error\\\": {\\n        \\\"additional_properties\\\": {\\n            \\\"debugInfo\\\": null\\n        },\\n        \\\"code\\\": \\\"UserError\\\",\\n        \\\"severity\\\": null,\\n        \\\"message\\\": \\\"Workspace is in deleting state.Current Operation cannot be performed\\\",\\n        \\\"message_format\\\": null,\\n        \\\"message_parameters\\\": null,\\n        \\\"reference_code\\\": null,\\n        \\\"details_uri\\\": null,\\n        \\\"target\\\": null,\\n        \\\"details\\\": [],\\n        \\\"inner_error\\\": null\\n    },\\n    \\\"correlation\\\": {\\n        \\\"operation\\\": \\\"6be4648dc72ca64cae9617c62f792142\\\",\\n        \\\"request\\\": \\\"dee385666526e14b\\\"\\n    },\\n    \\\"environment\\\": \\\"southcentralus\\\",\\n    \\\"location\\\": \\\"southcentralus\\\",\\n    \\\"time\\\": {},\\n    \\\"component_name\\\": \\\"run-history\\\"\\n}\""
     },
     "metadata": {}
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(pipeline_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "PipelineRunId: f16a1ce1-6f4c-4839-a6eb-ca81e45a814c\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/automl-bank-experiment/runs/f16a1ce1-6f4c-4839-a6eb-ca81e45a814c?wsid=/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourcegroups/aml-quickstarts-140144/workspaces/quick-starts-ws-140144\n",
      "PipelineRun Status: Running\n",
      "\n",
      "\n",
      "StepRunId: 1dce1f34-1509-49d0-9a73-79b3f2a03411\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/automl-bank-experiment/runs/1dce1f34-1509-49d0-9a73-79b3f2a03411?wsid=/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourcegroups/aml-quickstarts-140144/workspaces/quick-starts-ws-140144\n",
      "StepRun( automl_module ) Status: NotStarted\n",
      "StepRun( automl_module ) Status: Queued\n",
      "StepRun( automl_module ) Status: Running\n",
      "\n",
      "StepRun(automl_module) Execution Summary\n",
      "=========================================\n",
      "StepRun( automl_module ) Status: Finished\n",
      "{'runId': '1dce1f34-1509-49d0-9a73-79b3f2a03411', 'target': 'cpu-cluster', 'status': 'Completed', 'startTimeUtc': '2021-03-10T10:06:15.362761Z', 'endTimeUtc': '2021-03-10T10:36:33.815329Z', 'properties': {'ContentSnapshotId': '00000000-0000-0000-0000-000000000000', 'StepType': 'AutoMLStep', 'azureml.moduleid': 'c9ae4379-43a8-407a-8e96-1314f09c1a4e', 'azureml.runsource': 'azureml.StepRun', 'azureml.nodeid': '55cf9d05', 'azureml.pipelinerunid': 'f16a1ce1-6f4c-4839-a6eb-ca81e45a814c', 'num_iterations': '1000', 'training_type': 'TrainFull', 'acquisition_function': 'EI', 'metrics': 'accuracy', 'primary_metric': 'AUC_weighted', 'train_split': '0', 'MaxTimeSeconds': None, 'acquisition_parameter': '0', 'num_cross_validation': None, 'target': 'cpu-cluster', 'RawAMLSettingsString': None, 'AMLSettingsJsonString': '{\"path\": null, \"name\": \"placeholder\", \"subscription_id\": \"6b4af8be-9931-443e-90f6-c4c34a1f9737\", \"resource_group\": \"aml-quickstarts-140144\", \"workspace_name\": \"quick-starts-ws-140144\", \"region\": \"southcentralus\", \"compute_target\": \"cpu-cluster\", \"spark_service\": null, \"azure_service\": null, \"many_models\": false, \"pipeline_fetch_max_batch_size\": 1, \"iterations\": 1000, \"primary_metric\": \"AUC_weighted\", \"task_type\": \"classification\", \"data_script\": null, \"validation_size\": 0.0, \"n_cross_validations\": null, \"y_min\": null, \"y_max\": null, \"num_classes\": null, \"featurization\": \"auto\", \"_ignore_package_version_incompatibilities\": false, \"is_timeseries\": false, \"max_cores_per_iteration\": 1, \"max_concurrent_iterations\": 5, \"iteration_timeout_minutes\": null, \"mem_in_mb\": null, \"enforce_time_on_windows\": false, \"experiment_timeout_minutes\": 60, \"experiment_exit_score\": null, \"whitelist_models\": null, \"blacklist_algos\": null, \"supported_models\": [\"XGBoostClassifier\", \"LogisticRegression\", \"BernoulliNaiveBayes\", \"KNN\", \"DecisionTree\", \"TensorFlowDNN\", \"SVM\", \"SGD\", \"LinearSVM\", \"TensorFlowLinearClassifier\", \"AveragedPerceptronClassifier\", \"RandomForest\", \"ExtremeRandomTrees\", \"MultinomialNaiveBayes\", \"LightGBM\", \"GradientBoosting\"], \"private_models\": [], \"auto_blacklist\": true, \"blacklist_samples_reached\": false, \"exclude_nan_labels\": true, \"verbosity\": 20, \"_debug_log\": \"automl_errors.log\", \"show_warnings\": false, \"model_explainability\": true, \"service_url\": null, \"sdk_url\": null, \"sdk_packages\": null, \"enable_onnx_compatible_models\": false, \"enable_split_onnx_featurizer_estimator_models\": false, \"vm_type\": \"STANDARD_DS12_V2\", \"telemetry_verbosity\": 20, \"send_telemetry\": true, \"enable_dnn\": false, \"scenario\": \"SDK-1.13.0\", \"environment_label\": null, \"force_text_dnn\": false, \"enable_feature_sweeping\": true, \"enable_early_stopping\": true, \"early_stopping_n_iters\": 10, \"metrics\": null, \"enable_ensembling\": true, \"enable_stack_ensembling\": true, \"ensemble_iterations\": 15, \"enable_tf\": false, \"enable_subsampling\": null, \"subsample_seed\": null, \"enable_nimbusml\": false, \"enable_streaming\": false, \"force_streaming\": false, \"track_child_runs\": true, \"allowed_private_models\": [], \"label_column_name\": \"y\", \"weight_column_name\": null, \"cv_split_column_names\": null, \"enable_local_managed\": false, \"_local_managed_run_id\": null, \"cost_mode\": 1, \"lag_length\": 0, \"metric_operation\": \"maximize\", \"preprocess\": true}', 'DataPrepJsonString': '{\\\\\"training_data\\\\\": {\\\\\"datasetId\\\\\": \\\\\"3538ae08-65a9-4383-bd89-3fdc289d412f\\\\\"}, \\\\\"datasets\\\\\": 0}', 'EnableSubsampling': 'False', 'runTemplate': 'AutoML', 'Orchestrator': 'automl', 'ClientType': 'Others', '_aml_system_scenario_identification': 'Remote.Parent', 'root_attribution': 'azureml.StepRun', 'snapshotId': '00000000-0000-0000-0000-000000000000', 'SetupRunId': '1dce1f34-1509-49d0-9a73-79b3f2a03411_setup', 'SetupRunContainerId': 'dcid.1dce1f34-1509-49d0-9a73-79b3f2a03411_setup', 'ClientSdkVersion': '1.23.0', 'FeaturizationRunJsonPath': 'featurizer_container.json', 'FeaturizationRunId': '1dce1f34-1509-49d0-9a73-79b3f2a03411_featurize', 'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": true, \"subsampling\": false, \"dataset_classes\": 2, \"dataset_features\": 132, \"dataset_samples\": 29655, \"single_frequency_class_detected\": false}', 'ModelExplainRunId': '1dce1f34-1509-49d0-9a73-79b3f2a03411_ModelExplain'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.1dce1f34-1509-49d0-9a73-79b3f2a03411/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=wGUCgkx4TTjgkRbuSXzALTcZQCkSuaBvyd3hlJRWBaw%3D&st=2021-03-10T09%3A55%3A25Z&se=2021-03-10T18%3A05%3A25Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.1dce1f34-1509-49d0-9a73-79b3f2a03411/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=zZduusQPFxEdeHqjMaFQxQHLG7HllXqMHoILYt1vdQU%3D&st=2021-03-10T09%3A55%3A25Z&se=2021-03-10T18%3A05%3A25Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.1dce1f34-1509-49d0-9a73-79b3f2a03411/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=ubG8c8Ft2uWE0QqQQxbcR2K0YDCVhnoMO4q8f5%2F0Igc%3D&st=2021-03-10T09%3A55%3A25Z&se=2021-03-10T18%3A05%3A25Z&sp=r'}, 'submittedBy': 'ODL_User 140144'}\n",
      "\n",
      "\n",
      "\n",
      "PipelineRun Execution Summary\n",
      "==============================\n",
      "PipelineRun Status: Finished\n",
      "{'runId': 'f16a1ce1-6f4c-4839-a6eb-ca81e45a814c', 'status': 'Completed', 'startTimeUtc': '2021-03-10T10:05:10.796125Z', 'endTimeUtc': '2021-03-10T10:37:39.217116Z', 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{}'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.f16a1ce1-6f4c-4839-a6eb-ca81e45a814c/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=RbThzS9E4igZ%2BFcQjWmshvSjMrJGMom%2B12HHYcS6EOo%3D&st=2021-03-10T09%3A55%3A18Z&se=2021-03-10T18%3A05%3A18Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.f16a1ce1-6f4c-4839-a6eb-ca81e45a814c/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=PfK6F0pluRT8wB8YnsLux%2BQ2c5PfcpgvmacWIHsYqQc%3D&st=2021-03-10T09%3A55%3A18Z&se=2021-03-10T18%3A05%3A18Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://mlstrg140144.blob.core.windows.net/azureml/ExperimentRun/dcid.f16a1ce1-6f4c-4839-a6eb-ca81e45a814c/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=BbB7MTbA4SUet4l%2FtzfCcxRJQhKfUxBNw%2F09a6tSxLI%3D&st=2021-03-10T09%3A55%3A18Z&se=2021-03-10T18%3A05%3A18Z&sp=r'}, 'submittedBy': 'ODL_User 140144'}\n",
      "\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'Finished'"
      ]
     },
     "metadata": {},
     "execution_count": 16
    }
   ],
   "source": [
    "pipeline_run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine Results\n",
    "\n",
    "### Retrieve the metrics of all child runs\n",
    "Outputs of above run can be used as inputs of other steps in pipeline. In this tutorial, we will examine the outputs by retrieve output data and running some tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Downloading azureml/1dce1f34-1509-49d0-9a73-79b3f2a03411/metrics_data\nDownloaded azureml/1dce1f34-1509-49d0-9a73-79b3f2a03411/metrics_data, 1 files out of an estimated total of 1\n"
     ]
    }
   ],
   "source": [
    "metrics_output = pipeline_run.get_pipeline_output(metrics_output_name)\n",
    "num_file_downloaded = metrics_output.download('.', show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                  [0.9026893654928896]   \n",
       "average_precision_score_macro                       [0.8043768280782629]   \n",
       "AUC_macro                                           [0.9400441236128014]   \n",
       "average_precision_score_micro                       [0.9792529378468265]   \n",
       "recall_score_micro                                  [0.9092564491654022]   \n",
       "norm_macro_recall                                   [0.4644204746900509]   \n",
       "accuracy                                            [0.9092564491654022]   \n",
       "balanced_accuracy                                   [0.7322102373450254]   \n",
       "weighted_accuracy                                   [0.9532122345414047]   \n",
       "log_loss                                            [0.1858855432333583]   \n",
       "matthews_correlation                                [0.5075804332539731]   \n",
       "recall_score_weighted                               [0.9092564491654022]   \n",
       "recall_score_macro                                  [0.7322102373450254]   \n",
       "f1_score_micro                                       [0.909256449165402]   \n",
       "AUC_weighted                                        [0.9400441236128014]   \n",
       "f1_score_macro                                      [0.7519405667519964]   \n",
       "f1_score_weighted                                   [0.9052396183263401]   \n",
       "average_precision_score_weighted                    [0.9502272244802092]   \n",
       "precision_score_micro                               [0.9092564491654022]   \n",
       "precision_score_macro                               [0.7773756867569583]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_26  ...  \\\n",
       "AUC_micro                                           [0.9793693944704005]  ...   \n",
       "precision_score_weighted                            [0.9080335867085474]  ...   \n",
       "average_precision_score_macro                       [0.8085204474402641]  ...   \n",
       "AUC_macro                                           [0.9448491887516277]  ...   \n",
       "average_precision_score_micro                        [0.980352027134298]  ...   \n",
       "recall_score_micro                                  [0.9128983308042489]  ...   \n",
       "norm_macro_recall                                   [0.5016773270945287]  ...   \n",
       "accuracy                                            [0.9128983308042489]  ...   \n",
       "balanced_accuracy                                   [0.7508386635472644]  ...   \n",
       "weighted_accuracy                                   [0.9531333625443325]  ...   \n",
       "log_loss                                           [0.18115151864375498]  ...   \n",
       "matthews_correlation                                [0.5357619178216784]  ...   \n",
       "recall_score_weighted                               [0.9128983308042489]  ...   \n",
       "recall_score_macro                                  [0.7508386635472644]  ...   \n",
       "f1_score_micro                                      [0.9128983308042489]  ...   \n",
       "AUC_weighted                                        [0.9448491887516278]  ...   \n",
       "f1_score_macro                                      [0.7668111420906795]  ...   \n",
       "f1_score_weighted                                   [0.9100415644605403]  ...   \n",
       "average_precision_score_weighted                    [0.9516860732237478]  ...   \n",
       "precision_score_micro                               [0.9128983308042489]  ...   \n",
       "precision_score_macro                               [0.7860811293290488]  ...   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_18  \\\n",
       "AUC_micro                                           [0.8415028057870366]   \n",
       "precision_score_weighted                            [0.8966294781600979]   \n",
       "average_precision_score_macro                       [0.7192687147508846]   \n",
       "AUC_macro                                           [0.8734155232871537]   \n",
       "average_precision_score_micro                       [0.8477036469540248]   \n",
       "recall_score_micro                                  [0.7326251896813354]   \n",
       "norm_macro_recall                                   [0.5710201223680043]   \n",
       "accuracy                                            [0.7326251896813354]   \n",
       "balanced_accuracy                                   [0.7855100611840021]   \n",
       "weighted_accuracy                                   [0.7194953065987924]   \n",
       "log_loss                                            [0.5625868795604397]   \n",
       "matthews_correlation                                [0.3783961333211663]   \n",
       "recall_score_weighted                               [0.7326251896813354]   \n",
       "recall_score_macro                                  [0.7855100611840021]   \n",
       "f1_score_micro                                      [0.7326251896813354]   \n",
       "AUC_weighted                                        [0.8734155232871536]   \n",
       "f1_score_macro                                      [0.6217415399233475]   \n",
       "f1_score_weighted                                   [0.7806706276089838]   \n",
       "average_precision_score_weighted                    [0.9233758785468921]   \n",
       "precision_score_micro                               [0.7326251896813354]   \n",
       "precision_score_macro                               [0.6253752959866208]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_20  \\\n",
       "AUC_micro                                           [0.9763877305247063]   \n",
       "precision_score_weighted                             [0.895092517403297]   \n",
       "average_precision_score_macro                       [0.7953500733144905]   \n",
       "AUC_macro                                           [0.9329981457709313]   \n",
       "average_precision_score_micro                       [0.9775048906893984]   \n",
       "recall_score_micro                                  [0.9077389984825494]   \n",
       "norm_macro_recall                                   [0.3585080587647982]   \n",
       "accuracy                                            [0.9077389984825494]   \n",
       "balanced_accuracy                                   [0.6792540293823991]   \n",
       "weighted_accuracy                                   [0.9644656358962787]   \n",
       "log_loss                                           [0.19994256312838007]   \n",
       "matthews_correlation                                [0.4538726815046628]   \n",
       "recall_score_weighted                               [0.9077389984825494]   \n",
       "recall_score_macro                                  [0.6792540293823991]   \n",
       "f1_score_micro                                      [0.9077389984825494]   \n",
       "AUC_weighted                                        [0.9329981457709314]   \n",
       "f1_score_macro                                        [0.71617170693669]   \n",
       "f1_score_weighted                                   [0.8971238918213456]   \n",
       "average_precision_score_weighted                    [0.9474347924918001]   \n",
       "precision_score_micro                               [0.9077389984825494]   \n",
       "precision_score_macro                               [0.7873023436711378]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_24  \\\n",
       "AUC_micro                                           [0.9789654164009017]   \n",
       "precision_score_weighted                            [0.9113037932659365]   \n",
       "average_precision_score_macro                        [0.812573046671845]   \n",
       "AUC_macro                                           [0.9430264500867838]   \n",
       "average_precision_score_micro                       [0.9799669312828488]   \n",
       "recall_score_micro                                  [0.9165402124430956]   \n",
       "norm_macro_recall                                    [0.510515016291653]   \n",
       "accuracy                                            [0.9165402124430956]   \n",
       "balanced_accuracy                                   [0.7552575081458265]   \n",
       "weighted_accuracy                                   [0.9565823452967744]   \n",
       "log_loss                                           [0.18377453013344744]   \n",
       "matthews_correlation                                [0.5514496307758505]   \n",
       "recall_score_weighted                               [0.9165402124430956]   \n",
       "recall_score_macro                                  [0.7552575081458265]   \n",
       "f1_score_micro                                      [0.9165402124430956]   \n",
       "AUC_weighted                                        [0.9430264500867838]   \n",
       "f1_score_macro                                      [0.7742346026690958]   \n",
       "f1_score_weighted                                   [0.9133307204795383]   \n",
       "average_precision_score_weighted                     [0.952383837559749]   \n",
       "precision_score_micro                               [0.9165402124430956]   \n",
       "precision_score_macro                               [0.7978332522829199]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_22  \\\n",
       "AUC_micro                                           [0.9766515228619257]   \n",
       "precision_score_weighted                            [0.9000274768383943]   \n",
       "average_precision_score_macro                       [0.7998321444303222]   \n",
       "AUC_macro                                           [0.9342679499932389]   \n",
       "average_precision_score_micro                       [0.9777871805237555]   \n",
       "recall_score_micro                                  [0.9101669195751139]   \n",
       "norm_macro_recall                                   [0.4109757023749321]   \n",
       "accuracy                                            [0.9101669195751139]   \n",
       "balanced_accuracy                                    [0.705487851187466]   \n",
       "weighted_accuracy                                   [0.9609831957806473]   \n",
       "log_loss                                            [0.1921852994995217]   \n",
       "matthews_correlation                                [0.4867731611986173]   \n",
       "recall_score_weighted                               [0.9101669195751139]   \n",
       "recall_score_macro                                   [0.705487851187466]   \n",
       "f1_score_micro                                      [0.9101669195751139]   \n",
       "AUC_weighted                                        [0.9342679499932388]   \n",
       "f1_score_macro                                      [0.7372589501995638]   \n",
       "f1_score_weighted                                   [0.9026632441364442]   \n",
       "average_precision_score_weighted                    [0.9486146431518476]   \n",
       "precision_score_micro                               [0.9101669195751139]   \n",
       "precision_score_macro                               [0.7882750842617063]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_25  \\\n",
       "AUC_micro                                           [0.9733974085902907]   \n",
       "precision_score_weighted                            [0.8843704047191627]   \n",
       "average_precision_score_macro                       [0.7791553533474418]   \n",
       "AUC_macro                                           [0.9243114252741982]   \n",
       "average_precision_score_micro                       [0.9746121226103798]   \n",
       "recall_score_micro                                  [0.9004552352048558]   \n",
       "norm_macro_recall                                  [0.17742249192826853]   \n",
       "accuracy                                            [0.9004552352048558]   \n",
       "balanced_accuracy                                   [0.5887112459641343]   \n",
       "weighted_accuracy                                   [0.9778528352011013]   \n",
       "log_loss                                           [0.22929027929074844]   \n",
       "matthews_correlation                                [0.3310045181282741]   \n",
       "recall_score_weighted                               [0.9004552352048558]   \n",
       "recall_score_macro                                  [0.5887112459641343]   \n",
       "f1_score_micro                                      [0.9004552352048558]   \n",
       "AUC_weighted                                        [0.9243114252741981]   \n",
       "f1_score_macro                                      [0.6212887870981125]   \n",
       "f1_score_weighted                                   [0.8736142051995415]   \n",
       "average_precision_score_weighted                     [0.942782281265217]   \n",
       "precision_score_micro                               [0.9004552352048558]   \n",
       "precision_score_macro                               [0.8087657878957855]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_12  \\\n",
       "AUC_micro                                           [0.8268028304254619]   \n",
       "precision_score_weighted                            [0.8788181388329129]   \n",
       "average_precision_score_macro                       [0.7074092097970927]   \n",
       "AUC_macro                                            [0.850013985444024]   \n",
       "average_precision_score_micro                       [0.8083696028796385]   \n",
       "recall_score_micro                                  [0.7298937784522003]   \n",
       "norm_macro_recall                                   [0.4708454432459568]   \n",
       "accuracy                                            [0.7298937784522003]   \n",
       "balanced_accuracy                                   [0.7354227216229784]   \n",
       "weighted_accuracy                                   [0.7285210914182785]   \n",
       "log_loss                                            [0.5836886638828234]   \n",
       "matthews_correlation                                [0.3171591776578235]   \n",
       "recall_score_weighted                               [0.7298937784522003]   \n",
       "recall_score_macro                                  [0.7354227216229784]   \n",
       "f1_score_micro                                      [0.7298937784522003]   \n",
       "AUC_weighted                                        [0.8500139854440241]   \n",
       "f1_score_macro                                       [0.604168196425332]   \n",
       "f1_score_weighted                                   [0.7772862972560818]   \n",
       "average_precision_score_weighted                    [0.9170988852346857]   \n",
       "precision_score_micro                               [0.7298937784522003]   \n",
       "precision_score_macro                               [0.6068184320518542]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_16  \\\n",
       "AUC_micro                                           [0.8372877468735681]   \n",
       "precision_score_weighted                            [0.8914692469897325]   \n",
       "average_precision_score_macro                       [0.7118838873357954]   \n",
       "AUC_macro                                           [0.8529537072540925]   \n",
       "average_precision_score_micro                        [0.794703907496258]   \n",
       "recall_score_micro                                  [0.8169954476479514]   \n",
       "norm_macro_recall                                   [0.5547219860441941]   \n",
       "accuracy                                            [0.8169954476479514]   \n",
       "balanced_accuracy                                    [0.777360993022097]   \n",
       "weighted_accuracy                                   [0.8268356106376938]   \n",
       "log_loss                                            [0.5146885573540363]   \n",
       "matthews_correlation                                [0.4133801983987132]   \n",
       "recall_score_weighted                               [0.8169954476479514]   \n",
       "recall_score_macro                                   [0.777360993022097]   \n",
       "f1_score_micro                                      [0.8169954476479514]   \n",
       "AUC_weighted                                        [0.8529537072540926]   \n",
       "f1_score_macro                                      [0.6799831655281816]   \n",
       "f1_score_weighted                                   [0.8424787153334928]   \n",
       "average_precision_score_weighted                    [0.9177857992123136]   \n",
       "precision_score_micro                               [0.8169954476479514]   \n",
       "precision_score_macro                                [0.654025974025974]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_19  \\\n",
       "AUC_micro                                           [0.9746105401802059]   \n",
       "precision_score_weighted                            [0.8890546332831104]   \n",
       "average_precision_score_macro                       [0.7810523962199729]   \n",
       "AUC_macro                                           [0.9285931939975585]   \n",
       "average_precision_score_micro                       [0.9757189583187845]   \n",
       "recall_score_micro                                  [0.9025796661608497]   \n",
       "norm_macro_recall                                   [0.3455932884687698]   \n",
       "accuracy                                            [0.9025796661608497]   \n",
       "balanced_accuracy                                   [0.6727966442343849]   \n",
       "weighted_accuracy                                   [0.9596285749796182]   \n",
       "log_loss                                           [0.20678955773307725]   \n",
       "matthews_correlation                                [0.4276972780112856]   \n",
       "recall_score_weighted                               [0.9025796661608497]   \n",
       "recall_score_macro                                  [0.6727966442343849]   \n",
       "f1_score_micro                                      [0.9025796661608497]   \n",
       "AUC_weighted                                        [0.9285931939975586]   \n",
       "f1_score_macro                                      [0.7052632370927009]   \n",
       "f1_score_weighted                                    [0.892406452644354]   \n",
       "average_precision_score_weighted                    [0.9437150575561564]   \n",
       "precision_score_micro                               [0.9025796661608497]   \n",
       "precision_score_macro                               [0.7646535215263494]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_29  \\\n",
       "AUC_micro                                           [0.9795984627464707]   \n",
       "precision_score_weighted                            [0.9074329436294158]   \n",
       "average_precision_score_macro                       [0.8198704441305439]   \n",
       "AUC_macro                                           [0.9442592067752529]   \n",
       "average_precision_score_micro                       [0.9805583578526404]   \n",
       "recall_score_micro                                  [0.9132018209408195]   \n",
       "norm_macro_recall                                  [0.49017777259112316]   \n",
       "accuracy                                            [0.9132018209408195]   \n",
       "balanced_accuracy                                   [0.7450888862955616]   \n",
       "weighted_accuracy                                    [0.954939715235299]   \n",
       "log_loss                                           [0.18012736943904312]   \n",
       "matthews_correlation                                [0.5317952742313653]   \n",
       "recall_score_weighted                               [0.9132018209408195]   \n",
       "recall_score_macro                                  [0.7450888862955616]   \n",
       "f1_score_micro                                      [0.9132018209408195]   \n",
       "AUC_weighted                                         [0.944259206775253]   \n",
       "f1_score_macro                                      [0.7642816263884915]   \n",
       "f1_score_weighted                                   [0.9096764096913943]   \n",
       "average_precision_score_weighted                    [0.9541411484948241]   \n",
       "precision_score_micro                               [0.9132018209408195]   \n",
       "precision_score_macro                               [0.7884731106837775]   \n",
       "\n",
       "                                 1dce1f34-1509-49d0-9a73-79b3f2a03411_17  \n",
       "AUC_micro                                           [0.8300112599906512]  \n",
       "precision_score_weighted                             [0.880811359793208]  \n",
       "average_precision_score_macro                        [0.713816524369288]  \n",
       "AUC_macro                                           [0.8324821662433985]  \n",
       "average_precision_score_micro                       [0.8085191533067357]  \n",
       "recall_score_micro                                  [0.7614567526555387]  \n",
       "norm_macro_recall                                  [0.48981100200612415]  \n",
       "accuracy                                            [0.7614567526555387]  \n",
       "balanced_accuracy                                   [0.7449055010030621]  \n",
       "weighted_accuracy                                    [0.765565980737067]  \n",
       "log_loss                                            [0.5648245319007728]  \n",
       "matthews_correlation                               [0.34088651206507936]  \n",
       "recall_score_weighted                               [0.7614567526555387]  \n",
       "recall_score_macro                                  [0.7449055010030621]  \n",
       "f1_score_micro                                      [0.7614567526555387]  \n",
       "AUC_weighted                                        [0.8324821662433985]  \n",
       "f1_score_macro                                      [0.6276996722442643]  \n",
       "f1_score_weighted                                   [0.8008727631868612]  \n",
       "average_precision_score_weighted                     [0.914018220935529]  \n",
       "precision_score_micro                               [0.7614567526555387]  \n",
       "precision_score_macro                               [0.6186208697150116]  \n",
       "\n",
       "[21 rows x 38 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_7</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_0</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_8</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_14</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_10</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_27</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_21</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_13</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_31</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_26</th>\n      <th>...</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_18</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_20</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_24</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_22</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_25</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_12</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_16</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_19</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_29</th>\n      <th>1dce1f34-1509-49d0-9a73-79b3f2a03411_17</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>AUC_micro</th>\n      <td>[0.9591913991171614]</td>\n      <td>[0.979695082216353]</td>\n      <td>[0.9651563849212837]</td>\n      <td>[0.9627301217414531]</td>\n      <td>[0.8409726421372337]</td>\n      <td>[0.9792746171257781]</td>\n      <td>[0.9792565642982309]</td>\n      <td>[0.9758990146932517]</td>\n      <td>[0.9782191714581113]</td>\n      <td>[0.9793693944704005]</td>\n      <td>...</td>\n      <td>[0.8415028057870366]</td>\n      <td>[0.9763877305247063]</td>\n      <td>[0.9789654164009017]</td>\n      <td>[0.9766515228619257]</td>\n      <td>[0.9733974085902907]</td>\n      <td>[0.8268028304254619]</td>\n      <td>[0.8372877468735681]</td>\n      <td>[0.9746105401802059]</td>\n      <td>[0.9795984627464707]</td>\n      <td>[0.8300112599906512]</td>\n    </tr>\n    <tr>\n      <th>precision_score_weighted</th>\n      <td>[0.8447687581189142]</td>\n      <td>[0.9072720074188747]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.8762921187995815]</td>\n      <td>[0.9051091556982174]</td>\n      <td>[0.9062798949414683]</td>\n      <td>[0.8929725418691179]</td>\n      <td>[0.9026893654928896]</td>\n      <td>[0.9080335867085474]</td>\n      <td>...</td>\n      <td>[0.8966294781600979]</td>\n      <td>[0.895092517403297]</td>\n      <td>[0.9113037932659365]</td>\n      <td>[0.9000274768383943]</td>\n      <td>[0.8843704047191627]</td>\n      <td>[0.8788181388329129]</td>\n      <td>[0.8914692469897325]</td>\n      <td>[0.8890546332831104]</td>\n      <td>[0.9074329436294158]</td>\n      <td>[0.880811359793208]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_macro</th>\n      <td>[0.7171045179270072]</td>\n      <td>[0.8151093723721079]</td>\n      <td>[0.7382949076278038]</td>\n      <td>[0.7319151069033817]</td>\n      <td>[0.7242131699523829]</td>\n      <td>[0.8150378454592331]</td>\n      <td>[0.8126929119384294]</td>\n      <td>[0.7985126174047921]</td>\n      <td>[0.8043768280782629]</td>\n      <td>[0.8085204474402641]</td>\n      <td>...</td>\n      <td>[0.7192687147508846]</td>\n      <td>[0.7953500733144905]</td>\n      <td>[0.812573046671845]</td>\n      <td>[0.7998321444303222]</td>\n      <td>[0.7791553533474418]</td>\n      <td>[0.7074092097970927]</td>\n      <td>[0.7118838873357954]</td>\n      <td>[0.7810523962199729]</td>\n      <td>[0.8198704441305439]</td>\n      <td>[0.713816524369288]</td>\n    </tr>\n    <tr>\n      <th>AUC_macro</th>\n      <td>[0.8578750090303364]</td>\n      <td>[0.9450464668693166]</td>\n      <td>[0.887867766237471]</td>\n      <td>[0.8756689395328676]</td>\n      <td>[0.859009589754134]</td>\n      <td>[0.9438841004951403]</td>\n      <td>[0.943998021661693]</td>\n      <td>[0.9308878256246677]</td>\n      <td>[0.9400441236128014]</td>\n      <td>[0.9448491887516277]</td>\n      <td>...</td>\n      <td>[0.8734155232871537]</td>\n      <td>[0.9329981457709313]</td>\n      <td>[0.9430264500867838]</td>\n      <td>[0.9342679499932389]</td>\n      <td>[0.9243114252741982]</td>\n      <td>[0.850013985444024]</td>\n      <td>[0.8529537072540925]</td>\n      <td>[0.9285931939975585]</td>\n      <td>[0.9442592067752529]</td>\n      <td>[0.8324821662433985]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_micro</th>\n      <td>[0.9586854354179465]</td>\n      <td>[0.9806603102489483]</td>\n      <td>[0.9643687504090147]</td>\n      <td>[0.9638724327121927]</td>\n      <td>[0.8292355418289271]</td>\n      <td>[0.980258627086168]</td>\n      <td>[0.9802395848606664]</td>\n      <td>[0.9766643355999638]</td>\n      <td>[0.9792529378468265]</td>\n      <td>[0.980352027134298]</td>\n      <td>...</td>\n      <td>[0.8477036469540248]</td>\n      <td>[0.9775048906893984]</td>\n      <td>[0.9799669312828488]</td>\n      <td>[0.9777871805237555]</td>\n      <td>[0.9746121226103798]</td>\n      <td>[0.8083696028796385]</td>\n      <td>[0.794703907496258]</td>\n      <td>[0.9757189583187845]</td>\n      <td>[0.9805583578526404]</td>\n      <td>[0.8085191533067357]</td>\n    </tr>\n    <tr>\n      <th>recall_score_micro</th>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9116843702579667]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7089529590288316]</td>\n      <td>[0.9113808801213961]</td>\n      <td>[0.9125948406676783]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.9092564491654022]</td>\n      <td>[0.9128983308042489]</td>\n      <td>...</td>\n      <td>[0.7326251896813354]</td>\n      <td>[0.9077389984825494]</td>\n      <td>[0.9165402124430956]</td>\n      <td>[0.9101669195751139]</td>\n      <td>[0.9004552352048558]</td>\n      <td>[0.7298937784522003]</td>\n      <td>[0.8169954476479514]</td>\n      <td>[0.9025796661608497]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.7614567526555387]</td>\n    </tr>\n    <tr>\n      <th>norm_macro_recall</th>\n      <td>[0.002368263600612819]</td>\n      <td>[0.5026785366965085]</td>\n      <td>[0.0]</td>\n      <td>[0.0]</td>\n      <td>[0.4496320253701511]</td>\n      <td>[0.4762858735901099]</td>\n      <td>[0.4800211911893555]</td>\n      <td>[0.24549085203770704]</td>\n      <td>[0.4644204746900509]</td>\n      <td>[0.5016773270945287]</td>\n      <td>...</td>\n      <td>[0.5710201223680043]</td>\n      <td>[0.3585080587647982]</td>\n      <td>[0.510515016291653]</td>\n      <td>[0.4109757023749321]</td>\n      <td>[0.17742249192826853]</td>\n      <td>[0.4708454432459568]</td>\n      <td>[0.5547219860441941]</td>\n      <td>[0.3455932884687698]</td>\n      <td>[0.49017777259112316]</td>\n      <td>[0.48981100200612415]</td>\n    </tr>\n    <tr>\n      <th>accuracy</th>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9116843702579667]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7089529590288316]</td>\n      <td>[0.9113808801213961]</td>\n      <td>[0.9125948406676783]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.9092564491654022]</td>\n      <td>[0.9128983308042489]</td>\n      <td>...</td>\n      <td>[0.7326251896813354]</td>\n      <td>[0.9077389984825494]</td>\n      <td>[0.9165402124430956]</td>\n      <td>[0.9101669195751139]</td>\n      <td>[0.9004552352048558]</td>\n      <td>[0.7298937784522003]</td>\n      <td>[0.8169954476479514]</td>\n      <td>[0.9025796661608497]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.7614567526555387]</td>\n    </tr>\n    <tr>\n      <th>balanced_accuracy</th>\n      <td>[0.5011841318003064]</td>\n      <td>[0.7513392683482543]</td>\n      <td>[0.5]</td>\n      <td>[0.5]</td>\n      <td>[0.7248160126850756]</td>\n      <td>[0.7381429367950549]</td>\n      <td>[0.7400105955946777]</td>\n      <td>[0.6227454260188535]</td>\n      <td>[0.7322102373450254]</td>\n      <td>[0.7508386635472644]</td>\n      <td>...</td>\n      <td>[0.7855100611840021]</td>\n      <td>[0.6792540293823991]</td>\n      <td>[0.7552575081458265]</td>\n      <td>[0.705487851187466]</td>\n      <td>[0.5887112459641343]</td>\n      <td>[0.7354227216229784]</td>\n      <td>[0.777360993022097]</td>\n      <td>[0.6727966442343849]</td>\n      <td>[0.7450888862955616]</td>\n      <td>[0.7449055010030621]</td>\n    </tr>\n    <tr>\n      <th>weighted_accuracy</th>\n      <td>[0.9840510704229208]</td>\n      <td>[0.9514937218005303]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.7050145918943272]</td>\n      <td>[0.9543911754422495]</td>\n      <td>[0.9554428403944659]</td>\n      <td>[0.9766010009385309]</td>\n      <td>[0.9532122345414047]</td>\n      <td>[0.9531333625443325]</td>\n      <td>...</td>\n      <td>[0.7194953065987924]</td>\n      <td>[0.9644656358962787]</td>\n      <td>[0.9565823452967744]</td>\n      <td>[0.9609831957806473]</td>\n      <td>[0.9778528352011013]</td>\n      <td>[0.7285210914182785]</td>\n      <td>[0.8268356106376938]</td>\n      <td>[0.9596285749796182]</td>\n      <td>[0.954939715235299]</td>\n      <td>[0.765565980737067]</td>\n    </tr>\n    <tr>\n      <th>log_loss</th>\n      <td>[0.27471183914187036]</td>\n      <td>[0.17775706110025447]</td>\n      <td>[0.27277410497469884]</td>\n      <td>[0.26921680264223424]</td>\n      <td>[0.5369405537467462]</td>\n      <td>[0.1809974148381821]</td>\n      <td>[0.17981385781039308]</td>\n      <td>[0.33655623030329523]</td>\n      <td>[0.1858855432333583]</td>\n      <td>[0.18115151864375498]</td>\n      <td>...</td>\n      <td>[0.5625868795604397]</td>\n      <td>[0.19994256312838007]</td>\n      <td>[0.18377453013344744]</td>\n      <td>[0.1921852994995217]</td>\n      <td>[0.22929027929074844]</td>\n      <td>[0.5836886638828234]</td>\n      <td>[0.5146885573540363]</td>\n      <td>[0.20678955773307725]</td>\n      <td>[0.18012736943904312]</td>\n      <td>[0.5648245319007728]</td>\n    </tr>\n    <tr>\n      <th>matthews_correlation</th>\n      <td>[0.03032281531906968]</td>\n      <td>[0.5323740218566827]</td>\n      <td>[0.0]</td>\n      <td>[0.0]</td>\n      <td>[0.2980790940891629]</td>\n      <td>[0.5197756567915791]</td>\n      <td>[0.5254139610791995]</td>\n      <td>[0.3976739324324451]</td>\n      <td>[0.5075804332539731]</td>\n      <td>[0.5357619178216784]</td>\n      <td>...</td>\n      <td>[0.3783961333211663]</td>\n      <td>[0.4538726815046628]</td>\n      <td>[0.5514496307758505]</td>\n      <td>[0.4867731611986173]</td>\n      <td>[0.3310045181282741]</td>\n      <td>[0.3171591776578235]</td>\n      <td>[0.4133801983987132]</td>\n      <td>[0.4276972780112856]</td>\n      <td>[0.5317952742313653]</td>\n      <td>[0.34088651206507936]</td>\n    </tr>\n    <tr>\n      <th>recall_score_weighted</th>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9116843702579667]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7089529590288316]</td>\n      <td>[0.9113808801213961]</td>\n      <td>[0.9125948406676783]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.9092564491654022]</td>\n      <td>[0.9128983308042489]</td>\n      <td>...</td>\n      <td>[0.7326251896813354]</td>\n      <td>[0.9077389984825494]</td>\n      <td>[0.9165402124430956]</td>\n      <td>[0.9101669195751139]</td>\n      <td>[0.9004552352048558]</td>\n      <td>[0.7298937784522003]</td>\n      <td>[0.8169954476479514]</td>\n      <td>[0.9025796661608497]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.7614567526555387]</td>\n    </tr>\n    <tr>\n      <th>recall_score_macro</th>\n      <td>[0.5011841318003064]</td>\n      <td>[0.7513392683482543]</td>\n      <td>[0.5]</td>\n      <td>[0.5]</td>\n      <td>[0.7248160126850756]</td>\n      <td>[0.7381429367950549]</td>\n      <td>[0.7400105955946777]</td>\n      <td>[0.6227454260188535]</td>\n      <td>[0.7322102373450254]</td>\n      <td>[0.7508386635472644]</td>\n      <td>...</td>\n      <td>[0.7855100611840021]</td>\n      <td>[0.6792540293823991]</td>\n      <td>[0.7552575081458265]</td>\n      <td>[0.705487851187466]</td>\n      <td>[0.5887112459641343]</td>\n      <td>[0.7354227216229784]</td>\n      <td>[0.777360993022097]</td>\n      <td>[0.6727966442343849]</td>\n      <td>[0.7450888862955616]</td>\n      <td>[0.7449055010030621]</td>\n    </tr>\n    <tr>\n      <th>f1_score_micro</th>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9116843702579667]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7089529590288317]</td>\n      <td>[0.9113808801213961]</td>\n      <td>[0.9125948406676783]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.909256449165402]</td>\n      <td>[0.9128983308042489]</td>\n      <td>...</td>\n      <td>[0.7326251896813354]</td>\n      <td>[0.9077389984825494]</td>\n      <td>[0.9165402124430956]</td>\n      <td>[0.9101669195751139]</td>\n      <td>[0.9004552352048558]</td>\n      <td>[0.7298937784522003]</td>\n      <td>[0.8169954476479514]</td>\n      <td>[0.9025796661608497]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.7614567526555387]</td>\n    </tr>\n    <tr>\n      <th>AUC_weighted</th>\n      <td>[0.8578750090303364]</td>\n      <td>[0.9450464668693167]</td>\n      <td>[0.887867766237471]</td>\n      <td>[0.8756689395328677]</td>\n      <td>[0.8590095897541339]</td>\n      <td>[0.9438841004951403]</td>\n      <td>[0.943998021661693]</td>\n      <td>[0.9308878256246675]</td>\n      <td>[0.9400441236128014]</td>\n      <td>[0.9448491887516278]</td>\n      <td>...</td>\n      <td>[0.8734155232871536]</td>\n      <td>[0.9329981457709314]</td>\n      <td>[0.9430264500867838]</td>\n      <td>[0.9342679499932388]</td>\n      <td>[0.9243114252741981]</td>\n      <td>[0.8500139854440241]</td>\n      <td>[0.8529537072540926]</td>\n      <td>[0.9285931939975586]</td>\n      <td>[0.944259206775253]</td>\n      <td>[0.8324821662433985]</td>\n    </tr>\n    <tr>\n      <th>f1_score_macro</th>\n      <td>[0.4730282687304231]</td>\n      <td>[0.7653697272147331]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.5878687071952592]</td>\n      <td>[0.7580675984486392]</td>\n      <td>[0.7607503025413473]</td>\n      <td>[0.6650154684677185]</td>\n      <td>[0.7519405667519964]</td>\n      <td>[0.7668111420906795]</td>\n      <td>...</td>\n      <td>[0.6217415399233475]</td>\n      <td>[0.71617170693669]</td>\n      <td>[0.7742346026690958]</td>\n      <td>[0.7372589501995638]</td>\n      <td>[0.6212887870981125]</td>\n      <td>[0.604168196425332]</td>\n      <td>[0.6799831655281816]</td>\n      <td>[0.7052632370927009]</td>\n      <td>[0.7642816263884915]</td>\n      <td>[0.6276996722442643]</td>\n    </tr>\n    <tr>\n      <th>f1_score_weighted</th>\n      <td>[0.8359262707234042]</td>\n      <td>[0.9091539479147899]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.7612239322447801]</td>\n      <td>[0.9075232070026762]</td>\n      <td>[0.9086613440609772]</td>\n      <td>[0.885603431576398]</td>\n      <td>[0.9052396183263401]</td>\n      <td>[0.9100415644605403]</td>\n      <td>...</td>\n      <td>[0.7806706276089838]</td>\n      <td>[0.8971238918213456]</td>\n      <td>[0.9133307204795383]</td>\n      <td>[0.9026632441364442]</td>\n      <td>[0.8736142051995415]</td>\n      <td>[0.7772862972560818]</td>\n      <td>[0.8424787153334928]</td>\n      <td>[0.892406452644354]</td>\n      <td>[0.9096764096913943]</td>\n      <td>[0.8008727631868612]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_weighted</th>\n      <td>[0.9191154770723654]</td>\n      <td>[0.9531771295804466]</td>\n      <td>[0.927287743292874]</td>\n      <td>[0.9261939885813909]</td>\n      <td>[0.9218243986492881]</td>\n      <td>[0.9530284047649816]</td>\n      <td>[0.9525161907226625]</td>\n      <td>[0.947605275820125]</td>\n      <td>[0.9502272244802092]</td>\n      <td>[0.9516860732237478]</td>\n      <td>...</td>\n      <td>[0.9233758785468921]</td>\n      <td>[0.9474347924918001]</td>\n      <td>[0.952383837559749]</td>\n      <td>[0.9486146431518476]</td>\n      <td>[0.942782281265217]</td>\n      <td>[0.9170988852346857]</td>\n      <td>[0.9177857992123136]</td>\n      <td>[0.9437150575561564]</td>\n      <td>[0.9541411484948241]</td>\n      <td>[0.914018220935529]</td>\n    </tr>\n    <tr>\n      <th>precision_score_micro</th>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9116843702579667]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7089529590288316]</td>\n      <td>[0.9113808801213961]</td>\n      <td>[0.9125948406676783]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.9092564491654022]</td>\n      <td>[0.9128983308042489]</td>\n      <td>...</td>\n      <td>[0.7326251896813354]</td>\n      <td>[0.9077389984825494]</td>\n      <td>[0.9165402124430956]</td>\n      <td>[0.9101669195751139]</td>\n      <td>[0.9004552352048558]</td>\n      <td>[0.7298937784522003]</td>\n      <td>[0.8169954476479514]</td>\n      <td>[0.9025796661608497]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.7614567526555387]</td>\n    </tr>\n    <tr>\n      <th>precision_score_macro</th>\n      <td>[0.6941238991800789]</td>\n      <td>[0.7819118765348991]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.598804290308137]</td>\n      <td>[0.7836182515310355]</td>\n      <td>[0.78754962860383]</td>\n      <td>[0.822098675416211]</td>\n      <td>[0.7773756867569583]</td>\n      <td>[0.7860811293290488]</td>\n      <td>...</td>\n      <td>[0.6253752959866208]</td>\n      <td>[0.7873023436711378]</td>\n      <td>[0.7978332522829199]</td>\n      <td>[0.7882750842617063]</td>\n      <td>[0.8087657878957855]</td>\n      <td>[0.6068184320518542]</td>\n      <td>[0.654025974025974]</td>\n      <td>[0.7646535215263494]</td>\n      <td>[0.7884731106837775]</td>\n      <td>[0.6186208697150116]</td>\n    </tr>\n  </tbody>\n</table>\n<p>21 rows × 38 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "source": [
    "import json\n",
    "with open(metrics_output._path_on_datastore) as f:\n",
    "    metrics_output_result = f.read()\n",
    "    \n",
    "deserialized_metrics_output = json.loads(metrics_output_result)\n",
    "df = pd.DataFrame(deserialized_metrics_output)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the Best Model"
   ]
  },
  {
   "source": [
    "If you are a Mac user make sure you have:\n",
    "\n",
    "* **lightgbm** installed: `brew install lightgbm`\n",
    "* Confirm the version of **XGBoost** installed on your local environment matches with the installed on the AutoML training environment.\n",
    "\n",
    "Install the correct version as follows:\n",
    "```bash\n",
    "conda install -c anaconda py-xgboost==<the_specific_version>\n",
    "```\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Downloading azureml/1dce1f34-1509-49d0-9a73-79b3f2a03411/model_data\nDownloaded azureml/1dce1f34-1509-49d0-9a73-79b3f2a03411/model_data, 1 files out of an estimated total of 1\n"
     ]
    }
   ],
   "source": [
    "# Retrieve best model from Pipeline Run\n",
    "best_model_output = pipeline_run.get_pipeline_output(best_model_output_name)\n",
    "num_file_downloaded = best_model_output.download('.', show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "channels:\n- anaconda\n- conda-forge\n- pytorch\ndependencies:\n- python=3.6.2\n- pip=20.2.4\n- pip:\n  - azureml-core==1.23.0\n  - azureml-pipeline-core==1.23.0\n  - azureml-telemetry==1.23.0\n  - azureml-defaults==1.23.0\n  - azureml-interpret==1.23.0\n  - azureml-automl-core==1.23.0\n  - azureml-automl-runtime==1.23.0\n  - azureml-train-automl-client==1.23.0\n  - azureml-train-automl-runtime==1.23.0\n  - azureml-dataset-runtime==1.23.0\n  - azureml-mlflow==1.23.0\n  - inference-schema\n  - py-cpuinfo==5.0.0\n  - boto3==1.15.18\n  - botocore==1.18.18\n- numpy~=1.18.0\n- scikit-learn==0.22.1\n- pandas~=0.25.0\n- py-xgboost<=0.90\n- fbprophet==0.5\n- holidays==0.9.11\n- setuptools-git\n- psutil>5.0.0,<6.0.0\nname: azureml_661474bbe74e96b5d8added5888dfc85\n\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.environment import Environment\n",
    "\n",
    "# Retrieve AutoML training environment for the best model\n",
    "print(Environment.get(ws, \"AzureML-AutoML\").python.conda_dependencies.serialize_to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "PipelineWithYTransformations(Pipeline={'memory': None,\n",
       "                                       'steps': [('datatransformer',\n",
       "                                                  DataTransformer(enable_dnn=None,\n",
       "                                                                  enable_feature_sweeping=None,\n",
       "                                                                  feature_sweeping_config=None,\n",
       "                                                                  feature_sweeping_timeout=None,\n",
       "                                                                  featurization_config=None,\n",
       "                                                                  force_text_dnn=None,\n",
       "                                                                  is_cross_validation=None,\n",
       "                                                                  is_onnx_compatible=None,\n",
       "                                                                  logger=None,\n",
       "                                                                  observer=None,\n",
       "                                                                  task=None,\n",
       "                                                                  working_dir=None))...\n",
       "                                                                                                                                   min_weight_fraction_leaf=0.0,\n",
       "                                                                                                                                   n_estimators=25,\n",
       "                                                                                                                                   n_jobs=1,\n",
       "                                                                                                                                   oob_score=True,\n",
       "                                                                                                                                   random_state=None,\n",
       "                                                                                                                                   verbose=0,\n",
       "                                                                                                                                   warm_start=False))],\n",
       "                                                                                                      verbose=False))],\n",
       "                                                                                flatten_transform=None,\n",
       "                                                                                weights=[0.3333333333333333,\n",
       "                                                                                         0.26666666666666666,\n",
       "                                                                                         0.06666666666666667,\n",
       "                                                                                         0.06666666666666667,\n",
       "                                                                                         0.06666666666666667,\n",
       "                                                                                         0.13333333333333333,\n",
       "                                                                                         0.06666666666666667]))],\n",
       "                                       'verbose': False},\n",
       "                             y_transformer={},\n",
       "                             y_transformer_name='LabelEncoder')"
      ]
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "source": [
    "import pickle\n",
    "\n",
    "with open(best_model_output._path_on_datastore, \"rb\" ) as f:\n",
    "    best_model = pickle.load(f)\n",
    "best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[('datatransformer',\n",
       "  DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n",
       "                  feature_sweeping_config=None, feature_sweeping_timeout=None,\n",
       "                  featurization_config=None, force_text_dnn=None,\n",
       "                  is_cross_validation=None, is_onnx_compatible=None, logger=None,\n",
       "                  observer=None, task=None, working_dir=None)),\n",
       " ('prefittedsoftvotingclassifier',\n",
       "  PreFittedSoftVotingClassifier(classification_labels=None,\n",
       "                                estimators=[('0',\n",
       "                                             Pipeline(memory=None,\n",
       "                                                      steps=[('maxabsscaler',\n",
       "                                                              MaxAbsScaler(copy=True)),\n",
       "                                                             ('lightgbmclassifier',\n",
       "                                                              LightGBMClassifier(boosting_type='gbdt',\n",
       "                                                                                 class_weight=None,\n",
       "                                                                                 colsample_bytree=1.0,\n",
       "                                                                                 importance_type='split',\n",
       "                                                                                 learning_rate=0.1,\n",
       "                                                                                 max_depth=-1,\n",
       "                                                                                 min_child_samples=20,\n",
       "                                                                                 min_child_weight=0.001,\n",
       "                                                                                 min_spl...\n",
       "                                                                                   min_samples_leaf=0.01,\n",
       "                                                                                   min_samples_split=0.056842105263157895,\n",
       "                                                                                   min_weight_fraction_leaf=0.0,\n",
       "                                                                                   n_estimators=25,\n",
       "                                                                                   n_jobs=1,\n",
       "                                                                                   oob_score=True,\n",
       "                                                                                   random_state=None,\n",
       "                                                                                   verbose=0,\n",
       "                                                                                   warm_start=False))],\n",
       "                                                      verbose=False))],\n",
       "                                flatten_transform=None,\n",
       "                                weights=[0.3333333333333333, 0.26666666666666666,\n",
       "                                         0.06666666666666667, 0.06666666666666667,\n",
       "                                         0.06666666666666667, 0.13333333333333333,\n",
       "                                         0.06666666666666667]))]"
      ]
     },
     "metadata": {},
     "execution_count": 21
    }
   ],
   "source": [
    "best_model.steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model\n",
    "#### Load Test Data\n",
    "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "32950 x 20 table of data:\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "   age          job  marital    education  default housing loan    contact  \\\n0   57   technician  married  high.school       no      no  yes   cellular   \n1   55      unknown  married      unknown  unknown     yes   no  telephone   \n2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n3   36       admin.  married  high.school       no      no   no  telephone   \n4   27    housemaid  married  high.school       no     yes   no   cellular   \n\n  month day_of_week  duration  campaign  pdays  previous     poutcome  \\\n0   may         mon       371         1    999         1      failure   \n1   may         thu       285         2    999         0  nonexistent   \n2   may         fri        52         1    999         1      failure   \n3   jun         fri       355         4    999         0  nonexistent   \n4   jul         fri       189         2    999         0  nonexistent   \n\n   emp.var.rate  cons.price.idx  cons.conf.idx  euribor3m  nr.employed  \n0         -1.80           92.89         -46.20       1.30      5099.10  \n1          1.10           93.99         -36.40       4.86      5191.00  \n2         -1.80           92.89         -46.20       1.31      5099.10  \n3          1.40           94.47         -41.80       4.97      5228.10  \n4          1.40           93.92         -42.70       4.96      5228.10  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>job</th>\n      <th>marital</th>\n      <th>education</th>\n      <th>default</th>\n      <th>housing</th>\n      <th>loan</th>\n      <th>contact</th>\n      <th>month</th>\n      <th>day_of_week</th>\n      <th>duration</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>poutcome</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>57</td>\n      <td>technician</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>mon</td>\n      <td>371</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.80</td>\n      <td>92.89</td>\n      <td>-46.20</td>\n      <td>1.30</td>\n      <td>5099.10</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>55</td>\n      <td>unknown</td>\n      <td>married</td>\n      <td>unknown</td>\n      <td>unknown</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>may</td>\n      <td>thu</td>\n      <td>285</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.10</td>\n      <td>93.99</td>\n      <td>-36.40</td>\n      <td>4.86</td>\n      <td>5191.00</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>33</td>\n      <td>blue-collar</td>\n      <td>married</td>\n      <td>basic.9y</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>fri</td>\n      <td>52</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.80</td>\n      <td>92.89</td>\n      <td>-46.20</td>\n      <td>1.31</td>\n      <td>5099.10</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>36</td>\n      <td>admin.</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>jun</td>\n      <td>fri</td>\n      <td>355</td>\n      <td>4</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.40</td>\n      <td>94.47</td>\n      <td>-41.80</td>\n      <td>4.97</td>\n      <td>5228.10</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>27</td>\n      <td>housemaid</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>jul</td>\n      <td>fri</td>\n      <td>189</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.40</td>\n      <td>93.92</td>\n      <td>-42.70</td>\n      <td>4.96</td>\n      <td>5228.10</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "...\n"
     ]
    }
   ],
   "source": [
    "dataset_test = Dataset.Tabular.from_delimited_files(path='https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv')\n",
    "df_test = dataset_test.to_pandas_dataframe()\n",
    "df_test = df_test[pd.notnull(df_test['y'])]\n",
    "\n",
    "y_test = df_test['y']\n",
    "X_test = df_test.drop(['y'], axis=1)\n",
    "\n",
    "# Describe dataset\n",
    "m, k = X_test.shape\n",
    "print(\"{} x {} table of data:\".format(m, k))\n",
    "display(X_test.head())\n",
    "print(\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing Our Best Fitted Model\n",
    "\n",
    "We will use confusion matrix to see how our model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "ypred = best_model.predict(X_test)\n",
    "cm = confusion_matrix(y_test, ypred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fbb62f60ef0>"
      ],
      "text/html": "<style  type=\"text/css\" >\n    #T_9975de62_818e_11eb_af9b_acde48001122row0_col0 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }    #T_9975de62_818e_11eb_af9b_acde48001122row0_col1 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_9975de62_818e_11eb_af9b_acde48001122row1_col0 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_9975de62_818e_11eb_af9b_acde48001122row1_col1 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }</style><table id=\"T_9975de62_818e_11eb_af9b_acde48001122\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >0</th>        <th class=\"col_heading level0 col1\" >1</th>    </tr></thead><tbody>\n                <tr>\n                        <th id=\"T_9975de62_818e_11eb_af9b_acde48001122level0_row0\" class=\"row_heading level0 row0\" >0</th>\n                        <td id=\"T_9975de62_818e_11eb_af9b_acde48001122row0_col0\" class=\"data row0 col0\" >28729</td>\n                        <td id=\"T_9975de62_818e_11eb_af9b_acde48001122row0_col1\" class=\"data row0 col1\" >529</td>\n            </tr>\n            <tr>\n                        <th id=\"T_9975de62_818e_11eb_af9b_acde48001122level0_row1\" class=\"row_heading level0 row1\" >1</th>\n                        <td id=\"T_9975de62_818e_11eb_af9b_acde48001122row1_col0\" class=\"data row1 col0\" >1190</td>\n                        <td id=\"T_9975de62_818e_11eb_af9b_acde48001122row1_col1\" class=\"data row1 col1\" >2502</td>\n            </tr>\n    </tbody></table>"
     },
     "metadata": {},
     "execution_count": 24
    }
   ],
   "source": [
    "# Visualize the confusion matrix\n",
    "pd.DataFrame(cm).style.background_gradient(cmap='Blues', low=0, high=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publish and run from REST endpoint\n",
    "\n",
    "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
    "\n",
    "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Pipeline(Name: Bankmarketing Train,\n",
       "Id: 3f067809-5669-45c7-9f05-da3bcb9604e5,\n",
       "Status: Active,\n",
       "Endpoint: https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourceGroups/aml-quickstarts-140144/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-140144/PipelineRuns/PipelineSubmit/3f067809-5669-45c7-9f05-da3bcb9604e5)"
      ],
      "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Id</th><th>Status</th><th>Endpoint</th></tr><tr><td>Bankmarketing Train</td><td><a href=\"https://ml.azure.com/pipelines/3f067809-5669-45c7-9f05-da3bcb9604e5?wsid=/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourcegroups/aml-quickstarts-140144/workspaces/quick-starts-ws-140144\" target=\"_blank\" rel=\"noopener\">3f067809-5669-45c7-9f05-da3bcb9604e5</a></td><td>Active</td><td><a href=\"https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/6b4af8be-9931-443e-90f6-c4c34a1f9737/resourceGroups/aml-quickstarts-140144/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-140144/PipelineRuns/PipelineSubmit/3f067809-5669-45c7-9f05-da3bcb9604e5\" target=\"_blank\" rel=\"noopener\">REST Endpoint</a></td></tr></table>"
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "source": [
    "published_pipeline = pipeline_run.publish_pipeline(\n",
    "    name=\"Bankmarketing Train\", \n",
    "    description=\"Training bankmarketing pipeline\", \n",
    "    version=\"1.0\")\n",
    "\n",
    "published_pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Authenticate once again, to retrieve the `auth_header` so that the endpoint can be used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "\n",
    "interactive_auth = InteractiveLoginAuthentication()\n",
    "auth_header = interactive_auth.get_authentication_header()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the REST url from the endpoint property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the process_count_per_node is passed through to ParallelRunStep because you defined it is defined as a PipelineParameter object in the step configuration.\n",
    "\n",
    "Make the request to trigger the run. Access the Id key from the response dict to get the value of the run id.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "rest_endpoint = published_pipeline.endpoint\n",
    "response = requests.post(rest_endpoint, \n",
    "                         headers=auth_header, \n",
    "                         json={\"ExperimentName\": \"pipeline-rest-endpoint\"}\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Submitted pipeline run:  1087f5e9-2faf-4c57-890f-d1514ec5a736\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    response.raise_for_status()\n",
    "except Exception:    \n",
    "    raise Exception(\"Received bad response from the endpoint: {}\\n\"\n",
    "                    \"Response Code: {}\\n\"\n",
    "                    \"Headers: {}\\n\"\n",
    "                    \"Content: {}\".format(rest_endpoint, response.status_code, response.headers, response.content))\n",
    "\n",
    "run_id = response.json().get('Id')\n",
    "print('Submitted pipeline run: ', run_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "d9bdf2f3676e44d4a165e8f1e86bb8e5"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "application/aml.mini.widget.v1": "\"ServiceException:\\n\\tCode: 409\\n\\tMessage: (UserError) Workspace is in deleting state.Current Operation cannot be performed\\n\\tDetails:\\n\\n\\tHeaders: {\\n\\t    \\\"Date\\\": \\\"Wed, 10 Mar 2021 11:53:06 GMT\\\",\\n\\t    \\\"Content-Type\\\": \\\"application/json; charset=utf-8\\\",\\n\\t    \\\"Content-Length\\\": \\\"607\\\",\\n\\t    \\\"Connection\\\": \\\"keep-alive\\\",\\n\\t    \\\"Request-Context\\\": \\\"appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d\\\",\\n\\t    \\\"x-ms-response-type\\\": \\\"error\\\",\\n\\t    \\\"x-ms-client-request-id\\\": \\\"c900a01d-2f73-4955-9336-331c5142bb00\\\",\\n\\t    \\\"x-ms-client-session-id\\\": \\\"\\\",\\n\\t    \\\"X-Content-Type-Options\\\": \\\"nosniff\\\",\\n\\t    \\\"x-request-time\\\": \\\"0.064\\\",\\n\\t    \\\"Strict-Transport-Security\\\": \\\"max-age=15724800; includeSubDomains; preload\\\"\\n\\t}\\n\\tInnerException: {\\n    \\\"additional_properties\\\": {},\\n    \\\"error\\\": {\\n        \\\"additional_properties\\\": {\\n            \\\"debugInfo\\\": null\\n        },\\n        \\\"code\\\": \\\"UserError\\\",\\n        \\\"severity\\\": null,\\n        \\\"message\\\": \\\"Workspace is in deleting state.Current Operation cannot be performed\\\",\\n        \\\"message_format\\\": null,\\n        \\\"message_parameters\\\": null,\\n        \\\"reference_code\\\": null,\\n        \\\"details_uri\\\": null,\\n        \\\"target\\\": null,\\n        \\\"details\\\": [],\\n        \\\"inner_error\\\": null\\n    },\\n    \\\"correlation\\\": {\\n        \\\"operation\\\": \\\"e4ffa9b5f492d142b8f09b1eeabcf6db\\\",\\n        \\\"request\\\": \\\"f621ca049401a344\\\"\\n    },\\n    \\\"environment\\\": \\\"southcentralus\\\",\\n    \\\"location\\\": \\\"southcentralus\\\",\\n    \\\"time\\\": {},\\n    \\\"component_name\\\": \\\"run-history\\\"\\n}\""
     },
     "metadata": {}
    }
   ],
   "source": [
    "from azureml.pipeline.core.run import PipelineRun\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "published_pipeline_run = PipelineRun(ws.experiments[\"pipeline-rest-endpoint\"], run_id)\n",
    "RunDetails(published_pipeline_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "sanpil"
   }
  ],
  "categories": [
   "how-to-use-azureml",
   "machine-learning-pipelines",
   "intro-to-pipelines"
  ],
  "category": "tutorial",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "Custom"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "Automated Machine Learning"
  ],
  "friendly_name": "How to use AutoMLStep with AML Pipelines",
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "UdacityEnv",
   "language": "python",
   "name": "udacityenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13-final"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "order_index": 11,
  "star_tag": [
   "featured"
  ],
  "tags": [
   "None"
  ],
  "task": "Demonstrates the use of AutoMLStep"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}