aml-pipelines-with-automated-machine-learning-step.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
        "Licensed under the MIT License."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Azure Machine Learning Pipeline with AutoMLStep (Udacity Course 2)\n",
        "This notebook demonstrates the use of AutoMLStep in Azure Machine Learning Pipeline."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Introduction\n",
        "In this example we showcase how you can use AzureML Dataset to load data for AutoML via AML Pipeline. \n",
        "\n",
        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you have executed the [configuration](https://aka.ms/pl-config) before running this notebook.\n",
        "\n",
        "In this notebook you will learn how to:\n",
        "1. Create an `Experiment` in an existing `Workspace`.\n",
        "2. Create or Attach existing AmlCompute to a workspace.\n",
        "3. Define data loading in a `TabularDataset`.\n",
        "4. Configure AutoML using `AutoMLConfig`.\n",
        "5. Use AutoMLStep\n",
        "6. Train the model using AmlCompute\n",
        "7. Explore the results.\n",
        "8. Test the best fitted model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Azure Machine Learning and Pipeline SDK-specific imports"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import os\n",
        "import csv\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import datasets\n",
        "import pkg_resources\n",
        "\n",
        "import azureml.core\n",
        "from azureml.core.experiment import Experiment\n",
        "from azureml.core.workspace import Workspace\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "from azureml.core.dataset import Dataset\n",
        "\n",
        "from azureml.pipeline.steps import AutoMLStep\n",
        "\n",
        "# Check core SDK version number\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "SDK version: 1.26.0\n"
          ]
        }
      ],
      "execution_count": 1,
      "metadata": {
        "gather": {
          "logged": 1618851921424
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initialize Workspace\n",
        "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ws_udacity_azureml_p2\n",
            "UdacityAzureMLP2\n",
            "eastus2\n",
            "ca1598e0-85dc-47d5-b06d-41b5342b4989\n"
          ]
        }
      ],
      "execution_count": 2,
      "metadata": {
        "gather": {
          "logged": 1618851924666
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create an Azure ML experiment\n",
        "Let's create an experiment named \"automlstep-classification\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.\n",
        "\n",
        "The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step.\n",
        "\n",
        "*Udacity Note:* There is no need to create an Azure ML experiment, this needs to re-use the experiment that was already created\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Choose a name for the run history container in the workspace.\n",
        "# NOTE: update these to match your existing experiment name\n",
        "experiment_name = 'p2'\n",
        "project_folder = './pipeline-project'\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "experiment"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 3,
          "data": {
            "text/plain": "Experiment(Name: p2,\nWorkspace: ws_udacity_azureml_p2)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Workspace</th><th>Report Page</th><th>Docs Page</th></tr><tr><td>p2</td><td>ws_udacity_azureml_p2</td><td><a href=\"https://ml.azure.com/experiments/id/2cc14f36-5b12-471b-bc39-ee801e541d29?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&amp;tid=3814e260-63cb-43a8-82ce-e862c309e004\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.Experiment?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 3,
      "metadata": {
        "gather": {
          "logged": 1618851924832
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create or Attach an AmlCompute cluster\n",
        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource.\n",
        "\n",
        "**Udacity Note** There is no need to create a new compute target, it can re-use the previous cluster"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.compute import AmlCompute\n",
        "from azureml.core.compute import ComputeTarget\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# NOTE: update the cluster name to match the existing cluster\n",
        "# Choose a name for your CPU cluster\n",
        "amlcompute_cluster_name = \"MLAzureP2Compute\"\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print('Found existing cluster, use it.')\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',# for GPU, use \"STANDARD_NC6\"\n",
        "                                                           #vm_priority = 'lowpriority', # optional\n",
        "                                                           max_nodes=4)\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True, min_node_count = 1, timeout_in_minutes = 10)\n",
        "# For a more detailed view of current AmlCompute status, use get_status()."
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating...\n",
            "SucceededProvisioning operation finished, operation \"Succeeded\"\n",
            "Succeeded...................................................................................................................\n",
            "AmlCompute wait for completion finished\n",
            "\n",
            "Wait timeout has been reached\n",
            "Current provisioning state of AmlCompute is \"Succeeded\" and current node count is \"0\"\n"
          ]
        }
      ],
      "execution_count": 4,
      "metadata": {
        "gather": {
          "logged": 1618852544146
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data\n",
        "\n",
        "**Udacity note:** Make sure the `key` is the same name as the dataset that is uploaded, and that the description matches. If it is hard to find or unknown, loop over the `ws.datasets.keys()` and `print()` them.\n",
        "If it *isn't* found because it was deleted, it can be recreated with the link that has the CSV "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Try to load the dataset from the Workspace. Otherwise, create it from the file\n",
        "# NOTE: update the key to match the dataset name\n",
        "found = False\n",
        "key = \"bankmarketing_train\"\n",
        "description_text = \"Bank Marketing DataSet for Udacity Course 2\"\n",
        "\n",
        "if key in ws.datasets.keys(): \n",
        "        found = True\n",
        "        dataset = ws.datasets[key] \n",
        "\n",
        "if not found:\n",
        "        # Create AML Dataset and register it into Workspace\n",
        "        example_data = 'https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv'\n",
        "        dataset = Dataset.Tabular.from_delimited_files(example_data)        \n",
        "        #Register Dataset in Workspace\n",
        "        dataset = dataset.register(workspace=ws,\n",
        "                                   name=key,\n",
        "                                   description=description_text)\n",
        "\n",
        "\n",
        "df = dataset.to_pandas_dataframe()\n",
        "df.describe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 5,
          "data": {
            "text/plain": "                age      duration      campaign         pdays      previous  \\\ncount  32950.000000  32950.000000  32950.000000  32950.000000  32950.000000   \nmean      40.040212    257.335205      2.561730    962.174780      0.174780   \nstd       10.432313    257.331700      2.763646    187.646785      0.496503   \nmin       17.000000      0.000000      1.000000      0.000000      0.000000   \n25%       32.000000    102.000000      1.000000    999.000000      0.000000   \n50%       38.000000    179.000000      2.000000    999.000000      0.000000   \n75%       47.000000    318.000000      3.000000    999.000000      0.000000   \nmax       98.000000   4918.000000     56.000000    999.000000      7.000000   \n\n       emp.var.rate  cons.price.idx  cons.conf.idx     euribor3m   nr.employed  \ncount  32950.000000    32950.000000   32950.000000  32950.000000  32950.000000  \nmean       0.076228       93.574243     -40.518680      3.615654   5166.859608  \nstd        1.572242        0.578636       4.623004      1.735748     72.208448  \nmin       -3.400000       92.201000     -50.800000      0.634000   4963.600000  \n25%       -1.800000       93.075000     -42.700000      1.344000   5099.100000  \n50%        1.100000       93.749000     -41.800000      4.857000   5191.000000  \n75%        1.400000       93.994000     -36.400000      4.961000   5228.100000  \nmax        1.400000       94.767000     -26.900000      5.045000   5228.100000  ",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>duration</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>40.040212</td>\n      <td>257.335205</td>\n      <td>2.561730</td>\n      <td>962.174780</td>\n      <td>0.174780</td>\n      <td>0.076228</td>\n      <td>93.574243</td>\n      <td>-40.518680</td>\n      <td>3.615654</td>\n      <td>5166.859608</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>10.432313</td>\n      <td>257.331700</td>\n      <td>2.763646</td>\n      <td>187.646785</td>\n      <td>0.496503</td>\n      <td>1.572242</td>\n      <td>0.578636</td>\n      <td>4.623004</td>\n      <td>1.735748</td>\n      <td>72.208448</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>17.000000</td>\n      <td>0.000000</td>\n      <td>1.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>-3.400000</td>\n      <td>92.201000</td>\n      <td>-50.800000</td>\n      <td>0.634000</td>\n      <td>4963.600000</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>32.000000</td>\n      <td>102.000000</td>\n      <td>1.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>-1.800000</td>\n      <td>93.075000</td>\n      <td>-42.700000</td>\n      <td>1.344000</td>\n      <td>5099.100000</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>38.000000</td>\n      <td>179.000000</td>\n      <td>2.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.100000</td>\n      <td>93.749000</td>\n      <td>-41.800000</td>\n      <td>4.857000</td>\n      <td>5191.000000</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>47.000000</td>\n      <td>318.000000</td>\n      <td>3.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.400000</td>\n      <td>93.994000</td>\n      <td>-36.400000</td>\n      <td>4.961000</td>\n      <td>5228.100000</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>98.000000</td>\n      <td>4918.000000</td>\n      <td>56.000000</td>\n      <td>999.000000</td>\n      <td>7.000000</td>\n      <td>1.400000</td>\n      <td>94.767000</td>\n      <td>-26.900000</td>\n      <td>5.045000</td>\n      <td>5228.100000</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 5,
      "metadata": {
        "gather": {
          "logged": 1618852549480
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Review the Dataset Result\n",
        "\n",
        "You can peek the result of a TabularDataset at any range using `skip(i)` and `take(j).to_pandas_dataframe()`. Doing so evaluates only `j` records for all the steps in the TabularDataset, which makes it fast even against large datasets.\n",
        "\n",
        "`TabularDataset` objects are composed of a list of transformation steps (optional)."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset.take(5).to_pandas_dataframe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": "   age          job  marital    education  default housing loan    contact  \\\n0   57   technician  married  high.school       no      no  yes   cellular   \n1   55      unknown  married      unknown  unknown     yes   no  telephone   \n2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n3   36       admin.  married  high.school       no      no   no  telephone   \n4   27    housemaid  married  high.school       no     yes   no   cellular   \n\n  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n0   may         mon  ...         1    999         1      failure         -1.8   \n1   may         thu  ...         2    999         0  nonexistent          1.1   \n2   may         fri  ...         1    999         1      failure         -1.8   \n3   jun         fri  ...         4    999         0  nonexistent          1.4   \n4   jul         fri  ...         2    999         0  nonexistent          1.4   \n\n   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n0          92.893          -46.2      1.299       5099.1  no  \n1          93.994          -36.4      4.860       5191.0  no  \n2          92.893          -46.2      1.313       5099.1  no  \n3          94.465          -41.8      4.967       5228.1  no  \n4          93.918          -42.7      4.963       5228.1  no  \n\n[5 rows x 21 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>job</th>\n      <th>marital</th>\n      <th>education</th>\n      <th>default</th>\n      <th>housing</th>\n      <th>loan</th>\n      <th>contact</th>\n      <th>month</th>\n      <th>day_of_week</th>\n      <th>...</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>poutcome</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n      <th>y</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>57</td>\n      <td>technician</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>mon</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.299</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>55</td>\n      <td>unknown</td>\n      <td>married</td>\n      <td>unknown</td>\n      <td>unknown</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>may</td>\n      <td>thu</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.1</td>\n      <td>93.994</td>\n      <td>-36.4</td>\n      <td>4.860</td>\n      <td>5191.0</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>33</td>\n      <td>blue-collar</td>\n      <td>married</td>\n      <td>basic.9y</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.313</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>36</td>\n      <td>admin.</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>jun</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>4</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>94.465</td>\n      <td>-41.8</td>\n      <td>4.967</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>27</td>\n      <td>housemaid</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>jul</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>93.918</td>\n      <td>-42.7</td>\n      <td>4.963</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 21 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 6,
      "metadata": {
        "gather": {
          "logged": 1618852549764
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train\n",
        "This creates a general AutoML settings object.\n",
        "**Udacity notes:** These inputs must match what was used when training in the portal. `label_column_name` has to be `y` for example."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_settings = {\n",
        "    \"experiment_timeout_minutes\": 20,\n",
        "    \"max_concurrent_iterations\": 2,\n",
        "    \"primary_metric\" : 'accuracy'\n",
        "}\n",
        "automl_config = AutoMLConfig(compute_target=compute_target,\n",
        "                             task = \"classification\",\n",
        "                             training_data=dataset,\n",
        "                             label_column_name=\"y\",   \n",
        "                             path = project_folder,\n",
        "                             enable_early_stopping= True,\n",
        "                             featurization= 'auto',\n",
        "                             debug_log = \"automl_errors.log\",\n",
        "                             **automl_settings\n",
        "                            )"
      ],
      "outputs": [],
      "execution_count": 7,
      "metadata": {
        "gather": {
          "logged": 1618852549932
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Create Pipeline and AutoMLStep\n",
        "\n",
        "You can define outputs for the AutoMLStep using TrainingOutput."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
        "\n",
        "ds = ws.get_default_datastore()\n",
        "metrics_output_name = 'metrics_output'\n",
        "best_model_output_name = 'best_model_output'\n",
        "\n",
        "metrics_data = PipelineData(name='metrics_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=metrics_output_name,\n",
        "                           training_output=TrainingOutput(type='Metrics'))\n",
        "model_data = PipelineData(name='model_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=best_model_output_name,\n",
        "                           training_output=TrainingOutput(type='Model'))"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {
        "gather": {
          "logged": 1618852550756
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create an AutoMLStep."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_step = AutoMLStep(\n",
        "    name='automl_module',\n",
        "    automl_config=automl_config,\n",
        "    outputs=[metrics_data, model_data],\n",
        "    allow_reuse=True)"
      ],
      "outputs": [],
      "execution_count": 9,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample1"
        ],
        "gather": {
          "logged": 1618852550894
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import Pipeline\n",
        "pipeline = Pipeline(\n",
        "    description=\"pipeline_with_automlstep_1\",\n",
        "    workspace=ws,    \n",
        "    steps=[automl_step])"
      ],
      "outputs": [],
      "execution_count": 10,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample2"
        ],
        "gather": {
          "logged": 1618852552605
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run = experiment.submit(pipeline)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Created step automl_module [cd06412c][9261aaf1-229f-4bda-8580-8c3ed55f6932], (This step will run and generate new outputs)\n",
            "Submitted PipelineRun 77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b\n",
            "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&tid=3814e260-63cb-43a8-82ce-e862c309e004\n"
          ]
        }
      ],
      "execution_count": 11,
      "metadata": {
        "gather": {
          "logged": 1618852555533
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4c157298bd9942b1841ebb62ffaafde5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&tid=3814e260-63cb-43a8-82ce-e862c309e004\", \"run_id\": \"77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b\", \"run_properties\": {\"run_id\": \"77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b\", \"created_utc\": \"2021-04-19T17:15:53.966006Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"SDK\", \"runType\": \"SDK\", \"azureml.parameters\": \"{}\"}, \"tags\": {\"azureml.pipelineComponent\": \"pipelinerun\"}, \"end_time_utc\": \"2021-04-19T17:48:39.470111Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=6yGpJnjnXtRz8QZ58ePcoAZPsEmkWytJPQHqP%2BHOMME%3D&st=2021-04-19T18%3A06%3A29Z&se=2021-04-20T02%3A16%3A29Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=wCKC7xXLe7zxJgZM4TB3LCxgDQwuUaanzih7VTP8kK0%3D&st=2021-04-19T18%3A06%3A29Z&se=2021-04-20T02%3A16%3A29Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=jm8BcyWeTwF%2Fcp8fpM0ts2plL2Q84e8naH3JrSUiyTM%3D&st=2021-04-19T18%3A06%3A30Z&se=2021-04-20T02%3A16%3A30Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:32:45\", \"run_number\": \"75\", \"run_queued_details\": {\"status\": \"Finished\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"4a11362c-6199-4ec8-90cf-8f94516c3773\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-04-19T17:16:23.752721Z\", \"created_time\": \"2021-04-19T17:16:03.617642Z\", \"end_time\": \"2021-04-19T17:47:56.291337Z\", \"duration\": \"0:31:52\", \"run_number\": 76, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-04-19T17:16:03.617642Z\", \"is_reused\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2021-04-19 17:16:03Z] Submitting 1 runs, first five are: cd06412c:4a11362c-6199-4ec8-90cf-8f94516c3773\\n[2021-04-19 17:48:38Z] Completing processing run id 4a11362c-6199-4ec8-90cf-8f94516c3773.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"8340727b\": {\"node_id\": \"8340727b\", \"name\": \"bankmarketing_train\"}}, \"module_nodes\": {\"cd06412c\": {\"node_id\": \"cd06412c\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": false, \"run_id\": \"4a11362c-6199-4ec8-90cf-8f94516c3773\"}}, \"edges\": [{\"source_node_id\": \"8340727b\", \"source_node_name\": \"bankmarketing_train\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"cd06412c\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"4a11362c-6199-4ec8-90cf-8f94516c3773\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-04-19T17:16:23.752721Z\", \"created_time\": \"2021-04-19T17:16:03.617642Z\", \"end_time\": \"2021-04-19T17:47:56.291337Z\", \"duration\": \"0:31:52\", \"run_number\": 76, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-04-19T17:16:03.617642Z\", \"is_reused\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.26.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 12,
      "metadata": {
        "gather": {
          "logged": 1618852556374
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run.wait_for_completion()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineRunId: 77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b\n",
            "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&tid=3814e260-63cb-43a8-82ce-e862c309e004\n",
            "PipelineRun Status: NotStarted\n",
            "PipelineRun Status: Running\n",
            "\n",
            "\n",
            "StepRunId: 4a11362c-6199-4ec8-90cf-8f94516c3773\n",
            "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/4a11362c-6199-4ec8-90cf-8f94516c3773?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&tid=3814e260-63cb-43a8-82ce-e862c309e004\n",
            "StepRun( automl_module ) Status: NotStarted\n",
            "StepRun( automl_module ) Status: Running\n",
            "\n",
            "StepRun(automl_module) Execution Summary\n",
            "=========================================\n",
            "StepRun( automl_module ) Status: Finished\n",
            "{'runId': '4a11362c-6199-4ec8-90cf-8f94516c3773', 'target': 'MLAzureP2Compute', 'status': 'Completed', 'startTimeUtc': '2021-04-19T17:16:23.752721Z', 'endTimeUtc': '2021-04-19T17:47:56.291337Z', 'properties': {'ContentSnapshotId': '8782f2c7-a6ba-471d-bbda-ad14e818824c', 'StepType': 'AutoMLStep', 'azureml.moduleid': '9261aaf1-229f-4bda-8580-8c3ed55f6932', 'azureml.runsource': 'azureml.StepRun', 'azureml.nodeid': 'cd06412c', 'azureml.pipelinerunid': '77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b', 'num_iterations': '1000', 'training_type': 'TrainFull', 'acquisition_function': 'EI', 'metrics': 'accuracy', 'primary_metric': 'accuracy', 'train_split': '0', 'MaxTimeSeconds': None, 'acquisition_parameter': '0', 'num_cross_validation': None, 'target': 'MLAzureP2Compute', 'RawAMLSettingsString': None, 'AMLSettingsJsonString': '{\"path\": null, \"name\": \"placeholder\", \"subscription_id\": \"ca1598e0-85dc-47d5-b06d-41b5342b4989\", \"resource_group\": \"UdacityAzureMLP2\", \"workspace_name\": \"ws_udacity_azureml_p2\", \"region\": \"eastus2\", \"compute_target\": \"MLAzureP2Compute\", \"spark_service\": null, \"azure_service\": null, \"many_models\": false, \"pipeline_fetch_max_batch_size\": 1, \"enable_batch_run\": false, \"enable_run_restructure\": false, \"iterations\": 1000, \"primary_metric\": \"accuracy\", \"task_type\": \"classification\", \"data_script\": null, \"test_size\": 0.0, \"validation_size\": 0.0, \"n_cross_validations\": null, \"y_min\": null, \"y_max\": null, \"num_classes\": null, \"featurization\": \"auto\", \"_ignore_package_version_incompatibilities\": false, \"is_timeseries\": false, \"max_cores_per_iteration\": 1, \"max_concurrent_iterations\": 2, \"iteration_timeout_minutes\": null, \"mem_in_mb\": null, \"enforce_time_on_windows\": false, \"experiment_timeout_minutes\": 20, \"experiment_exit_score\": null, \"whitelist_models\": null, \"blacklist_algos\": null, \"supported_models\": [\"KNN\", \"MultinomialNaiveBayes\", \"ExtremeRandomTrees\", \"LightGBM\", \"AveragedPerceptronClassifier\", \"DecisionTree\", \"SGD\", \"RandomForest\", \"XGBoostClassifier\", \"LinearSVM\", \"TensorFlowLinearClassifier\", \"SVM\", \"TensorFlowDNN\", \"GradientBoosting\", \"LogisticRegression\", \"BernoulliNaiveBayes\"], \"private_models\": [], \"auto_blacklist\": true, \"blacklist_samples_reached\": false, \"exclude_nan_labels\": true, \"verbosity\": 20, \"_debug_log\": \"automl_errors.log\", \"show_warnings\": false, \"model_explainability\": true, \"service_url\": null, \"sdk_url\": null, \"sdk_packages\": null, \"enable_onnx_compatible_models\": false, \"enable_split_onnx_featurizer_estimator_models\": false, \"vm_type\": \"STANDARD_D2_V2\", \"telemetry_verbosity\": 20, \"send_telemetry\": true, \"enable_dnn\": false, \"scenario\": \"SDK-1.13.0\", \"environment_label\": null, \"save_mlflow\": false, \"force_text_dnn\": false, \"enable_feature_sweeping\": true, \"enable_early_stopping\": true, \"early_stopping_n_iters\": 10, \"metrics\": null, \"enable_metric_confidence\": false, \"enable_ensembling\": true, \"enable_stack_ensembling\": true, \"ensemble_iterations\": 15, \"enable_tf\": false, \"enable_subsampling\": null, \"subsample_seed\": null, \"enable_nimbusml\": false, \"enable_streaming\": false, \"force_streaming\": false, \"track_child_runs\": true, \"allowed_private_models\": [], \"label_column_name\": \"y\", \"weight_column_name\": null, \"cv_split_column_names\": null, \"enable_local_managed\": false, \"_local_managed_run_id\": null, \"cost_mode\": 1, \"lag_length\": 0, \"metric_operation\": \"maximize\", \"preprocess\": true}', 'DataPrepJsonString': '{\\\\\"training_data\\\\\": {\\\\\"datasetId\\\\\": \\\\\"805df0cb-1844-4d72-b4b7-37067c2edef5\\\\\"}, \\\\\"datasets\\\\\": 0}', 'EnableSubsampling': 'False', 'runTemplate': 'AutoML', 'Orchestrator': 'automl', 'ClientType': 'Others', '_aml_system_scenario_identification': 'Remote.Parent', 'root_attribution': 'azureml.StepRun', 'snapshotId': '8782f2c7-a6ba-471d-bbda-ad14e818824c', 'SetupRunId': '4a11362c-6199-4ec8-90cf-8f94516c3773_setup', 'SetupRunContainerId': 'dcid.4a11362c-6199-4ec8-90cf-8f94516c3773_setup', 'ClientSdkVersion': '1.26.0', 'FeaturizationRunJsonPath': 'featurizer_container.json', 'FeaturizationRunId': '4a11362c-6199-4ec8-90cf-8f94516c3773_featurize', 'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": true, \"subsampling\": false, \"dataset_classes\": 2, \"dataset_features\": 132, \"dataset_samples\": 29655, \"single_frequency_class_detected\": false}', 'ModelExplainRunId': '4a11362c-6199-4ec8-90cf-8f94516c3773_ModelExplain'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.4a11362c-6199-4ec8-90cf-8f94516c3773/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=uMysCPIPIsxTY%2BrjvRcLFv92zs1nDvCzolSDdIRQvLc%3D&st=2021-04-19T17%3A06%3A08Z&se=2021-04-20T01%3A16%3A08Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.4a11362c-6199-4ec8-90cf-8f94516c3773/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=zVXgipE3aqRCT0fkZwsNhqJQmTiusLHbC%2BLt0fmSy8s%3D&st=2021-04-19T17%3A06%3A08Z&se=2021-04-20T01%3A16%3A08Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.4a11362c-6199-4ec8-90cf-8f94516c3773/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=3ITjq6%2BSz3NEI9SNVacTnyuX0e7v%2FQeiMorkdkpLPP0%3D&st=2021-04-19T17%3A06%3A08Z&se=2021-04-20T01%3A16%3A08Z&sp=r'}, 'submittedBy': 'Abhi Ojha'}\n",
            "\n",
            "\n",
            "\n",
            "PipelineRun Execution Summary\n",
            "==============================\n",
            "PipelineRun Status: Finished\n",
            "{'runId': '77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b', 'status': 'Completed', 'startTimeUtc': '2021-04-19T17:15:57.096817Z', 'endTimeUtc': '2021-04-19T17:48:39.470111Z', 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{}'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=Ltx8EWsR97dGJcTa7li9sNs4HiTilnfwfLgtpoPIMNg%3D&st=2021-04-19T17%3A06%3A17Z&se=2021-04-20T01%3A16%3A17Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=6ZzaJ33aq4Lf58PMeixauV7%2Btoty9zA%2BS9AiTX4RaL0%3D&st=2021-04-19T17%3A06%3A17Z&se=2021-04-20T01%3A16%3A17Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.77ade7b3-6fa8-4eca-9f55-c1b1e3a8fe3b/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=tQP8v%2BlGIqdbPEayfRLYBSDO%2Fh7A1nCvibrKo2kNAl4%3D&st=2021-04-19T17%3A06%3A17Z&se=2021-04-20T01%3A16%3A17Z&sp=r'}, 'submittedBy': 'Abhi Ojha'}\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 13,
          "data": {
            "text/plain": "'Finished'"
          },
          "metadata": {}
        }
      ],
      "execution_count": 13,
      "metadata": {
        "gather": {
          "logged": 1618854524770
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Examine Results\n",
        "\n",
        "### Retrieve the metrics of all child runs\n",
        "Outputs of above run can be used as inputs of other steps in pipeline. In this tutorial, we will examine the outputs by retrieve output data and running some tests."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "metrics_output = pipeline_run.get_pipeline_output(metrics_output_name)\n",
        "num_file_downloaded = metrics_output.download('.', show_progress=True)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading azureml/4a11362c-6199-4ec8-90cf-8f94516c3773/metrics_data\n",
            "Downloaded azureml/4a11362c-6199-4ec8-90cf-8f94516c3773/metrics_data, 1 files out of an estimated total of 1\n"
          ]
        }
      ],
      "execution_count": 14,
      "metadata": {
        "gather": {
          "logged": 1618854528306
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "with open(metrics_output._path_on_datastore) as f:\n",
        "    metrics_output_result = f.read()\n",
        "    \n",
        "deserialized_metrics_output = json.loads(metrics_output_result)\n",
        "df = pd.DataFrame(deserialized_metrics_output)\n",
        "df"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 15,
          "data": {
            "text/plain": "                                 4a11362c-6199-4ec8-90cf-8f94516c3773_7  \\\nAUC_micro                                          [0.9781896974539526]   \nnorm_macro_recall                                 [0.41910578367574525]   \nf1_score_weighted                                  [0.9038810538459678]   \nprecision_score_weighted                            [0.901311576372359]   \naverage_precision_score_weighted                   [0.9498586901696163]   \nrecall_score_weighted                              [0.9110773899848255]   \nprecision_score_macro                              [0.7908196055445871]   \naccuracy                                           [0.9110773899848255]   \nf1_score_micro                                     [0.9110773899848255]   \naverage_precision_score_macro                      [0.8022857542994071]   \nweighted_accuracy                                  [0.9611104717292754]   \nAUC_weighted                                       [0.9406868983248957]   \nmatthews_correlation                               [0.4937290323447371]   \nbalanced_accuracy                                  [0.7095528918378726]   \nrecall_score_macro                                 [0.7095528918378726]   \nAUC_macro                                          [0.9406868983248957]   \naverage_precision_score_micro                      [0.9792469103727843]   \nf1_score_macro                                     [0.7410400670578374]   \nprecision_score_micro                              [0.9110773899848255]   \nrecall_score_micro                                 [0.9110773899848255]   \nlog_loss                                          [0.18485719214536023]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_10  \\\nAUC_micro                                           [0.9771292780480841]   \nnorm_macro_recall                                   [0.3327507608637261]   \nf1_score_weighted                                   [0.8922458323878717]   \nprecision_score_weighted                            [0.8895916674605656]   \naverage_precision_score_weighted                    [0.9492320854883415]   \nrecall_score_weighted                                [0.903793626707132]   \nprecision_score_macro                                [0.772448410062275]   \naccuracy                                             [0.903793626707132]   \nf1_score_micro                                       [0.903793626707132]   \naverage_precision_score_macro                       [0.8015553269193132]   \nweighted_accuracy                                   [0.9627381552023843]   \nAUC_weighted                                        [0.9366556635491167]   \nmatthews_correlation                               [0.42581079306268055]   \nbalanced_accuracy                                   [0.6663753804318631]   \nrecall_score_macro                                  [0.6663753804318631]   \nAUC_macro                                           [0.9366556635491168]   \naverage_precision_score_micro                       [0.9782235801974871]   \nf1_score_macro                                      [0.7017444770694911]   \nprecision_score_micro                                [0.903793626707132]   \nrecall_score_micro                                   [0.903793626707132]   \nlog_loss                                           [0.19892711190385837]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_21  \\\nAUC_micro                                           [0.9534762054982835]   \nnorm_macro_recall                                                  [0.0]   \nf1_score_weighted                                   [0.8353395018439429]   \nprecision_score_weighted                             [0.788565560086672]   \naverage_precision_score_weighted                    [0.9066762570000854]   \nrecall_score_weighted                               [0.8880121396054628]   \nprecision_score_macro                               [0.4440060698027314]   \naccuracy                                            [0.8880121396054628]   \nf1_score_micro                                      [0.8880121396054628]   \naverage_precision_score_macro                       [0.6840348429031319]   \nweighted_accuracy                                   [0.9843450583187134]   \nAUC_weighted                                        [0.8291418679737037]   \nmatthews_correlation                                               [0.0]   \nbalanced_accuracy                                                  [0.5]   \nrecall_score_macro                                                 [0.5]   \nAUC_macro                                           [0.8291418679737037]   \naverage_precision_score_micro                       [0.9514423062659481]   \nf1_score_macro                                      [0.4703423886834914]   \nprecision_score_micro                               [0.8880121396054628]   \nrecall_score_micro                                  [0.8880121396054628]   \nlog_loss                                            [0.2789041773663072]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_15  \\\nAUC_micro                                           [0.9715234145633819]   \nnorm_macro_recall                                  [0.08869920551563681]   \nf1_score_weighted                                    [0.855910589385397]   \nprecision_score_weighted                            [0.8704808674095915]   \naverage_precision_score_weighted                    [0.9377366764156898]   \nrecall_score_weighted                               [0.8931714719271624]   \nprecision_score_macro                               [0.7786772665370782]   \naccuracy                                            [0.8931714719271624]   \nf1_score_micro                                      [0.8931714719271624]   \naverage_precision_score_macro                       [0.7600198800090151]   \nweighted_accuracy                                    [0.979774506570003]   \nAUC_weighted                                        [0.9184773648830132]   \nmatthews_correlation                               [0.22234411229941844]   \nbalanced_accuracy                                   [0.5443496027578184]   \nrecall_score_macro                                  [0.5443496027578184]   \nAUC_macro                                           [0.9184773648830131]   \naverage_precision_score_micro                       [0.9727129482425465]   \nf1_score_macro                                      [0.5544040176787701]   \nprecision_score_micro                               [0.8931714719271624]   \nrecall_score_micro                                  [0.8931714719271624]   \nlog_loss                                           [0.23663712579666943]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_17  \\\nAUC_micro                                           [0.9793942631614094]   \nnorm_macro_recall                                   [0.5064138542957541]   \nf1_score_weighted                                   [0.9102831069309071]   \nprecision_score_weighted                            [0.9083837833780429]   \naverage_precision_score_weighted                    [0.9545391828359017]   \nrecall_score_weighted                               [0.9128983308042489]   \nprecision_score_macro                               [0.7855784879651856]   \naccuracy                                            [0.9128983308042489]   \nf1_score_micro                                      [0.9128983308042489]   \naverage_precision_score_macro                       [0.8219228160768492]   \nweighted_accuracy                                   [0.9525453867527467]   \nAUC_weighted                                        [0.9439961692850012]   \nmatthews_correlation                                [0.5378120541497806]   \nbalanced_accuracy                                   [0.7532069271478771]   \nrecall_score_macro                                  [0.7532069271478771]   \nAUC_macro                                           [0.9439961692850012]   \naverage_precision_score_micro                       [0.9803460970885172]   \nf1_score_macro                                      [0.7680037996089013]   \nprecision_score_micro                               [0.9128983308042489]   \nrecall_score_micro                                  [0.9128983308042489]   \nlog_loss                                           [0.18546137380446998]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_14  \\\nAUC_micro                                           [0.9723749369647763]   \nnorm_macro_recall                                  [0.36964176887155076]   \nf1_score_weighted                                   [0.8959712268844915]   \nprecision_score_weighted                            [0.8928321441255288]   \naverage_precision_score_weighted                    [0.9378486130491581]   \nrecall_score_weighted                               [0.9050075872534142]   \nprecision_score_macro                               [0.7721631205673759]   \naccuracy                                            [0.9050075872534142]   \nf1_score_micro                                      [0.9050075872534142]   \naverage_precision_score_macro                       [0.7595699066838024]   \nweighted_accuracy                                   [0.9596739896135009]   \nAUC_weighted                                        [0.9190117755586305]   \nmatthews_correlation                                 [0.448559599848506]   \nbalanced_accuracy                                   [0.6848208844357754]   \nrecall_score_macro                                  [0.6848208844357754]   \nAUC_macro                                           [0.9190117755586305]   \naverage_precision_score_micro                       [0.9736438309245249]   \nf1_score_macro                                      [0.7168551535252986]   \nprecision_score_micro                               [0.9050075872534142]   \nrecall_score_micro                                  [0.9050075872534142]   \nlog_loss                                           [0.21392914688531073]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_19  \\\nAUC_micro                                            [0.978919639588193]   \nnorm_macro_recall                                   [0.4959155093943284]   \nf1_score_weighted                                   [0.9090397106606033]   \nprecision_score_weighted                            [0.9069697699068349]   \naverage_precision_score_weighted                    [0.9526661376923169]   \nrecall_score_weighted                               [0.9119878603945372]   \nprecision_score_macro                               [0.7835922951274648]   \naccuracy                                            [0.9119878603945372]   \nf1_score_micro                                      [0.9119878603945372]   \naverage_precision_score_macro                       [0.8144743900739468]   \nweighted_accuracy                                   [0.9527120986999115]   \nAUC_weighted                                        [0.9419780048791601]   \nmatthews_correlation                                [0.5303542542460528]   \nbalanced_accuracy                                   [0.7479577546971642]   \nrecall_score_macro                                  [0.7479577546971642]   \nAUC_macro                                           [0.9419780048791602]   \naverage_precision_score_micro                       [0.9799005886582354]   \nf1_score_macro                                      [0.7640701170131234]   \nprecision_score_micro                               [0.9119878603945372]   \nrecall_score_micro                                  [0.9119878603945372]   \nlog_loss                                           [0.18558180944526662]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_18  \\\nAUC_micro                                           [0.9788298359817722]   \nnorm_macro_recall                                  [0.49215611089808786]   \nf1_score_weighted                                   [0.9062804699821092]   \nprecision_score_weighted                             [0.904434899217646]   \naverage_precision_score_weighted                    [0.9523973441226696]   \nrecall_score_weighted                                [0.908649468892261]   \nprecision_score_macro                               [0.7730122814372734]   \naccuracy                                             [0.908649468892261]   \nf1_score_micro                                       [0.908649468892261]   \naverage_precision_score_macro                       [0.8128587183770495]   \nweighted_accuracy                                   [0.9490115533678862]   \nAUC_weighted                                        [0.9426115177077949]   \nmatthews_correlation                                [0.5183910930168123]   \nbalanced_accuracy                                   [0.7460780554490439]   \nrecall_score_macro                                  [0.7460780554490439]   \nAUC_macro                                           [0.9426115177077949]   \naverage_precision_score_micro                       [0.9798331125567491]   \nf1_score_macro                                      [0.7585358306343567]   \nprecision_score_micro                                [0.908649468892261]   \nrecall_score_micro                                   [0.908649468892261]   \nlog_loss                                           [0.18276368510928603]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_16  \\\nAUC_micro                                           [0.9778602333512174]   \nnorm_macro_recall                                   [0.4853930835959077]   \nf1_score_weighted                                    [0.906156466274463]   \nprecision_score_weighted                            [0.9040871676607377]   \naverage_precision_score_weighted                    [0.9508113391401188]   \nrecall_score_weighted                               [0.9089529590288316]   \nprecision_score_macro                               [0.7744847871584888]   \naccuracy                                            [0.9089529590288316]   \nf1_score_micro                                      [0.9089529590288316]   \naverage_precision_score_macro                       [0.8088969274697876]   \nweighted_accuracy                                    [0.950229930267267]   \nAUC_weighted                                        [0.9370849518474679]   \nmatthews_correlation                                [0.5162034816601402]   \nbalanced_accuracy                                   [0.7426965417979539]   \nrecall_score_macro                                  [0.7426965417979539]   \nAUC_macro                                           [0.9370849518474678]   \naverage_precision_score_micro                       [0.9788087720258263]   \nf1_score_macro                                      [0.7571849668386146]   \nprecision_score_micro                               [0.9089529590288316]   \nrecall_score_micro                                  [0.9089529590288316]   \nlog_loss                                            [0.3214779057369575]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_28  ...  \\\nAUC_micro                                           [0.9768338932626571]  ...   \nnorm_macro_recall                                  [0.49215611089808786]  ...   \nf1_score_weighted                                   [0.9062804699821092]  ...   \nprecision_score_weighted                             [0.904434899217646]  ...   \naverage_precision_score_weighted                    [0.9486736347163315]  ...   \nrecall_score_weighted                                [0.908649468892261]  ...   \nprecision_score_macro                               [0.7730122814372734]  ...   \naccuracy                                             [0.908649468892261]  ...   \nf1_score_micro                                       [0.908649468892261]  ...   \naverage_precision_score_macro                       [0.7991745048777775]  ...   \nweighted_accuracy                                   [0.9490115533678862]  ...   \nAUC_weighted                                         [0.937106254179425]  ...   \nmatthews_correlation                                [0.5183910930168123]  ...   \nbalanced_accuracy                                   [0.7460780554490439]  ...   \nrecall_score_macro                                  [0.7460780554490439]  ...   \nAUC_macro                                            [0.937106254179425]  ...   \naverage_precision_score_micro                       [0.9778754906463094]  ...   \nf1_score_macro                                      [0.7585358306343567]  ...   \nprecision_score_micro                                [0.908649468892261]  ...   \nrecall_score_micro                                   [0.908649468892261]  ...   \nlog_loss                                           [0.22829568238730566]  ...   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_9  \\\nAUC_micro                                          [0.9712773066286575]   \nnorm_macro_recall                                  [0.1401656395237909]   \nf1_score_weighted                                  [0.8662136392619358]   \nprecision_score_weighted                           [0.8772966484411009]   \naverage_precision_score_weighted                   [0.9378060982987899]   \nrecall_score_weighted                              [0.8968133535660091]   \nprecision_score_macro                              [0.7906573862807009]   \naccuracy                                           [0.8968133535660091]   \nf1_score_micro                                     [0.8968133535660091]   \naverage_precision_score_macro                      [0.7620204361207507]   \nweighted_accuracy                                  [0.9779317071981736]   \nAUC_weighted                                       [0.9154468766150411]   \nmatthews_correlation                               [0.2854476429412161]   \nbalanced_accuracy                                  [0.5700828197618955]   \nrecall_score_macro                                 [0.5700828197618955]   \nAUC_macro                                           [0.915446876615041]   \naverage_precision_score_micro                      [0.9724336547737189]   \nf1_score_macro                                     [0.5945349258052841]   \nprecision_score_micro                              [0.8968133535660091]   \nrecall_score_micro                                 [0.8968133535660091]   \nlog_loss                                          [0.23235315960994582]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_5  \\\nAUC_micro                                          [0.8284765854366182]   \nnorm_macro_recall                                  [0.4458485459769155]   \nf1_score_weighted                                    [0.75701585495587]   \nprecision_score_weighted                           [0.8759759678358785]   \naverage_precision_score_weighted                    [0.917298616342847]   \nrecall_score_weighted                              [0.7034901365705615]   \nprecision_score_macro                              [0.5971856577566018]   \naccuracy                                           [0.7034901365705615]   \nf1_score_micro                                     [0.7034901365705615]   \naverage_precision_score_macro                      [0.7023261438469145]   \nweighted_accuracy                                  [0.6986651661824931]   \nAUC_weighted                                       [0.8549964156511013]   \nmatthews_correlation                              [0.29438099191554834]   \nbalanced_accuracy                                  [0.7229242729884577]   \nrecall_score_macro                                 [0.7229242729884577]   \nAUC_macro                                          [0.8549964156511012]   \naverage_precision_score_micro                      [0.8216086940681235]   \nf1_score_macro                                     [0.5839877109467246]   \nprecision_score_micro                              [0.7034901365705615]   \nrecall_score_micro                                 [0.7034901365705615]   \nlog_loss                                           [0.5425420944004252]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_12  \\\nAUC_micro                                           [0.9700581881316473]   \nnorm_macro_recall                                   [0.3286977606618171]   \nf1_score_weighted                                    [0.892342935520704]   \nprecision_score_weighted                            [0.8900794345597077]   \naverage_precision_score_weighted                    [0.9385179673164679]   \nrecall_score_weighted                               [0.9044006069802731]   \nprecision_score_macro                                [0.776355093319379]   \naccuracy                                            [0.9044006069802731]   \nf1_score_micro                                      [0.9044006069802731]   \naverage_precision_score_macro                       [0.7630054059702478]   \nweighted_accuracy                                   [0.9639989574179747]   \nAUC_weighted                                        [0.9200699457438869]   \nmatthews_correlation                               [0.42623303560744186]   \nbalanced_accuracy                                   [0.6643488803309086]   \nrecall_score_macro                                  [0.6643488803309086]   \nAUC_macro                                           [0.9200699457438868]   \naverage_precision_score_micro                       [0.9679693407446847]   \nf1_score_macro                                      [0.7008431471845663]   \nprecision_score_micro                               [0.9044006069802731]   \nrecall_score_micro                                  [0.9044006069802731]   \nlog_loss                                           [0.22539140754429898]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_11  \\\nAUC_micro                                           [0.9709334739488947]   \nnorm_macro_recall                                  [0.14731859211961917]   \nf1_score_weighted                                   [0.8695917447568857]   \nprecision_score_weighted                            [0.8927421895096666]   \naverage_precision_score_weighted                    [0.9389105423108886]   \nrecall_score_weighted                               [0.9010622154779969]   \nprecision_score_macro                               [0.8572850121746318]   \naccuracy                                            [0.9010622154779969]   \nf1_score_micro                                      [0.9010622154779969]   \naverage_precision_score_macro                       [0.7661826412910435]   \nweighted_accuracy                                   [0.9823475042704128]   \nAUC_weighted                                         [0.915315820964088]   \nmatthews_correlation                                [0.3244525388373706]   \nbalanced_accuracy                                   [0.5736592960598096]   \nrecall_score_macro                                  [0.5736592960598096]   \nAUC_macro                                            [0.915315820964088]   \naverage_precision_score_micro                       [0.9722607266220833]   \nf1_score_macro                                      [0.6013584326439484]   \nprecision_score_micro                               [0.9010622154779969]   \nrecall_score_micro                                  [0.9010622154779969]   \nlog_loss                                           [0.22149830847512975]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_24  \\\nAUC_micro                                            [0.978035787888487]   \nnorm_macro_recall                                   [0.3977432494762405]   \nf1_score_weighted                                   [0.8992883396952636]   \nprecision_score_weighted                             [0.896203970678224]   \naverage_precision_score_weighted                    [0.9502830355518578]   \nrecall_score_weighted                               [0.9068285280728376]   \nprecision_score_macro                               [0.7759960359081934]   \naccuracy                                            [0.9068285280728376]   \nf1_score_micro                                      [0.9068285280728376]   \naverage_precision_score_macro                       [0.8045843908248679]   \nweighted_accuracy                                   [0.9584586020317934]   \nAUC_weighted                                        [0.9399149203385405]   \nmatthews_correlation                               [0.46856282431427704]   \nbalanced_accuracy                                   [0.6988716247381203]   \nrecall_score_macro                                  [0.6988716247381203]   \nAUC_macro                                           [0.9399149203385404]   \naverage_precision_score_micro                       [0.9790986967072279]   \nf1_score_macro                                      [0.7286665549036044]   \nprecision_score_micro                               [0.9068285280728376]   \nrecall_score_micro                                  [0.9068285280728376]   \nlog_loss                                            [0.1855209800247038]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_13  \\\nAUC_micro                                           [0.9781379337341491]   \nnorm_macro_recall                                   [0.4796794276896974]   \nf1_score_weighted                                   [0.9084088618605763]   \nprecision_score_weighted                            [0.9060269508317579]   \naverage_precision_score_weighted                     [0.950292900508557]   \nrecall_score_weighted                               [0.9122913505311078]   \nprecision_score_macro                               [0.7864891147777485]   \naccuracy                                            [0.9122913505311078]   \nf1_score_micro                                      [0.9122913505311078]   \naverage_precision_score_macro                       [0.8050360880499143]   \nweighted_accuracy                                   [0.9551064271824634]   \nAUC_weighted                                        [0.9393707846852903]   \nmatthews_correlation                                [0.5242574455664286]   \nbalanced_accuracy                                   [0.7398397138448487]   \nrecall_score_macro                                  [0.7398397138448487]   \nAUC_macro                                           [0.9393707846852904]   \naverage_precision_score_micro                       [0.9791629630104502]   \nf1_score_macro                                      [0.7602368688673142]   \nprecision_score_micro                               [0.9122913505311078]   \nrecall_score_micro                                  [0.9122913505311078]   \nlog_loss                                           [0.21102079316404485]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_27  \\\nAUC_micro                                           [0.9693777991668988]   \nnorm_macro_recall                                  [0.04128577170939174]   \nf1_score_weighted                                   [0.8450250587736351]   \nprecision_score_weighted                            [0.8536588221856857]   \naverage_precision_score_weighted                     [0.933128227372767]   \nrecall_score_weighted                               [0.8889226100151745]   \nprecision_score_macro                               [0.7202719797596457]   \naccuracy                                            [0.8889226100151745]   \nf1_score_micro                                      [0.8889226100151746]   \naverage_precision_score_macro                       [0.7444849772802506]   \nweighted_accuracy                                    [0.980356503726242]   \nAUC_weighted                                        [0.9088260192239652]   \nmatthews_correlation                               [0.13486362497228446]   \nbalanced_accuracy                                   [0.5206428858546959]   \nrecall_score_macro                                  [0.5206428858546959]   \nAUC_macro                                           [0.9088260192239652]   \naverage_precision_score_micro                       [0.9703871559841004]   \nf1_score_macro                                      [0.5129361873990307]   \nprecision_score_micro                               [0.8889226100151745]   \nrecall_score_micro                                  [0.8889226100151745]   \nlog_loss                                           [0.24793059362231587]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_25  \\\nAUC_micro                                           [0.8880121396054628]   \nnorm_macro_recall                                                  [0.0]   \nf1_score_weighted                                   [0.8353395018439429]   \nprecision_score_weighted                             [0.788565560086672]   \naverage_precision_score_weighted                    [0.8011068409624182]   \nrecall_score_weighted                               [0.8880121396054628]   \nprecision_score_macro                               [0.4440060698027314]   \naccuracy                                            [0.8880121396054628]   \nf1_score_micro                                      [0.8880121396054628]   \naverage_precision_score_macro                                      [0.5]   \nweighted_accuracy                                   [0.9843450583187134]   \nAUC_weighted                                                       [0.5]   \nmatthews_correlation                                               [0.0]   \nbalanced_accuracy                                                  [0.5]   \nrecall_score_macro                                                 [0.5]   \nAUC_macro                                                          [0.5]   \naverage_precision_score_micro                       [0.8445594902839405]   \nf1_score_macro                                      [0.4703423886834914]   \nprecision_score_micro                               [0.8880121396054628]   \nrecall_score_micro                                  [0.8880121396054628]   \nlog_loss                                           [0.35067519027071736]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_26  \\\nAUC_micro                                           [0.9778390489107284]   \nnorm_macro_recall                                   [0.4837083469946113]   \nf1_score_weighted                                   [0.9065304612995165]   \nprecision_score_weighted                            [0.9043639107609646]   \naverage_precision_score_weighted                    [0.9499840099845974]   \nrecall_score_weighted                               [0.9095599393019727]   \nprecision_score_macro                               [0.7766115551902172]   \naccuracy                                            [0.9095599393019727]   \nf1_score_micro                                      [0.9095599393019727]   \naverage_precision_score_macro                       [0.8041712317790284]   \nweighted_accuracy                                   [0.9511967445870643]   \nAUC_weighted                                        [0.9386214983134111]   \nmatthews_correlation                                [0.5172993681045217]   \nbalanced_accuracy                                   [0.7418541734973056]   \nrecall_score_macro                                  [0.7418541734973056]   \nAUC_macro                                           [0.9386214983134111]   \naverage_precision_score_micro                       [0.9788608451294213]   \nf1_score_macro                                      [0.7575617064479683]   \nprecision_score_micro                               [0.9095599393019727]   \nrecall_score_micro                                  [0.9095599393019727]   \nlog_loss                                            [0.1876484785138208]   \n\n                                 4a11362c-6199-4ec8-90cf-8f94516c3773_4  \nAUC_micro                                          [0.8724818262829827]  \nnorm_macro_recall                                  [0.5880027118794771]  \nf1_score_weighted                                  [0.8297554387078409]  \nprecision_score_weighted                           [0.8964714404557308]  \naverage_precision_score_weighted                   [0.9323956343903792]  \nrecall_score_weighted                              [0.7981790591805766]  \nprecision_score_macro                              [0.6495818344433837]  \naccuracy                                           [0.7981790591805766]  \nf1_score_micro                                     [0.7981790591805767]  \naverage_precision_score_macro                      [0.7520694885582426]  \nweighted_accuracy                                  [0.7992162698903164]  \nAUC_weighted                                       [0.8885022052544517]  \nmatthews_correlation                               [0.4194151268149887]  \nbalanced_accuracy                                  [0.7940013559397385]  \nrecall_score_macro                                 [0.7940013559397385]  \nAUC_macro                                          [0.8885022052544518]  \naverage_precision_score_micro                      [0.8577423292287286]  \nf1_score_macro                                     [0.6711291077624292]  \nprecision_score_micro                              [0.7981790591805766]  \nrecall_score_micro                                 [0.7981790591805766]  \nlog_loss                                           [0.5089305587772139]  \n\n[21 rows x 33 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_7</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_10</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_21</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_15</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_17</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_14</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_19</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_18</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_16</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_28</th>\n      <th>...</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_9</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_5</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_12</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_11</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_24</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_13</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_27</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_25</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_26</th>\n      <th>4a11362c-6199-4ec8-90cf-8f94516c3773_4</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>AUC_micro</th>\n      <td>[0.9781896974539526]</td>\n      <td>[0.9771292780480841]</td>\n      <td>[0.9534762054982835]</td>\n      <td>[0.9715234145633819]</td>\n      <td>[0.9793942631614094]</td>\n      <td>[0.9723749369647763]</td>\n      <td>[0.978919639588193]</td>\n      <td>[0.9788298359817722]</td>\n      <td>[0.9778602333512174]</td>\n      <td>[0.9768338932626571]</td>\n      <td>...</td>\n      <td>[0.9712773066286575]</td>\n      <td>[0.8284765854366182]</td>\n      <td>[0.9700581881316473]</td>\n      <td>[0.9709334739488947]</td>\n      <td>[0.978035787888487]</td>\n      <td>[0.9781379337341491]</td>\n      <td>[0.9693777991668988]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9778390489107284]</td>\n      <td>[0.8724818262829827]</td>\n    </tr>\n    <tr>\n      <th>norm_macro_recall</th>\n      <td>[0.41910578367574525]</td>\n      <td>[0.3327507608637261]</td>\n      <td>[0.0]</td>\n      <td>[0.08869920551563681]</td>\n      <td>[0.5064138542957541]</td>\n      <td>[0.36964176887155076]</td>\n      <td>[0.4959155093943284]</td>\n      <td>[0.49215611089808786]</td>\n      <td>[0.4853930835959077]</td>\n      <td>[0.49215611089808786]</td>\n      <td>...</td>\n      <td>[0.1401656395237909]</td>\n      <td>[0.4458485459769155]</td>\n      <td>[0.3286977606618171]</td>\n      <td>[0.14731859211961917]</td>\n      <td>[0.3977432494762405]</td>\n      <td>[0.4796794276896974]</td>\n      <td>[0.04128577170939174]</td>\n      <td>[0.0]</td>\n      <td>[0.4837083469946113]</td>\n      <td>[0.5880027118794771]</td>\n    </tr>\n    <tr>\n      <th>f1_score_weighted</th>\n      <td>[0.9038810538459678]</td>\n      <td>[0.8922458323878717]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.855910589385397]</td>\n      <td>[0.9102831069309071]</td>\n      <td>[0.8959712268844915]</td>\n      <td>[0.9090397106606033]</td>\n      <td>[0.9062804699821092]</td>\n      <td>[0.906156466274463]</td>\n      <td>[0.9062804699821092]</td>\n      <td>...</td>\n      <td>[0.8662136392619358]</td>\n      <td>[0.75701585495587]</td>\n      <td>[0.892342935520704]</td>\n      <td>[0.8695917447568857]</td>\n      <td>[0.8992883396952636]</td>\n      <td>[0.9084088618605763]</td>\n      <td>[0.8450250587736351]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.9065304612995165]</td>\n      <td>[0.8297554387078409]</td>\n    </tr>\n    <tr>\n      <th>precision_score_weighted</th>\n      <td>[0.901311576372359]</td>\n      <td>[0.8895916674605656]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.8704808674095915]</td>\n      <td>[0.9083837833780429]</td>\n      <td>[0.8928321441255288]</td>\n      <td>[0.9069697699068349]</td>\n      <td>[0.904434899217646]</td>\n      <td>[0.9040871676607377]</td>\n      <td>[0.904434899217646]</td>\n      <td>...</td>\n      <td>[0.8772966484411009]</td>\n      <td>[0.8759759678358785]</td>\n      <td>[0.8900794345597077]</td>\n      <td>[0.8927421895096666]</td>\n      <td>[0.896203970678224]</td>\n      <td>[0.9060269508317579]</td>\n      <td>[0.8536588221856857]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.9043639107609646]</td>\n      <td>[0.8964714404557308]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_weighted</th>\n      <td>[0.9498586901696163]</td>\n      <td>[0.9492320854883415]</td>\n      <td>[0.9066762570000854]</td>\n      <td>[0.9377366764156898]</td>\n      <td>[0.9545391828359017]</td>\n      <td>[0.9378486130491581]</td>\n      <td>[0.9526661376923169]</td>\n      <td>[0.9523973441226696]</td>\n      <td>[0.9508113391401188]</td>\n      <td>[0.9486736347163315]</td>\n      <td>...</td>\n      <td>[0.9378060982987899]</td>\n      <td>[0.917298616342847]</td>\n      <td>[0.9385179673164679]</td>\n      <td>[0.9389105423108886]</td>\n      <td>[0.9502830355518578]</td>\n      <td>[0.950292900508557]</td>\n      <td>[0.933128227372767]</td>\n      <td>[0.8011068409624182]</td>\n      <td>[0.9499840099845974]</td>\n      <td>[0.9323956343903792]</td>\n    </tr>\n    <tr>\n      <th>recall_score_weighted</th>\n      <td>[0.9110773899848255]</td>\n      <td>[0.903793626707132]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8931714719271624]</td>\n      <td>[0.9128983308042489]</td>\n      <td>[0.9050075872534142]</td>\n      <td>[0.9119878603945372]</td>\n      <td>[0.908649468892261]</td>\n      <td>[0.9089529590288316]</td>\n      <td>[0.908649468892261]</td>\n      <td>...</td>\n      <td>[0.8968133535660091]</td>\n      <td>[0.7034901365705615]</td>\n      <td>[0.9044006069802731]</td>\n      <td>[0.9010622154779969]</td>\n      <td>[0.9068285280728376]</td>\n      <td>[0.9122913505311078]</td>\n      <td>[0.8889226100151745]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9095599393019727]</td>\n      <td>[0.7981790591805766]</td>\n    </tr>\n    <tr>\n      <th>precision_score_macro</th>\n      <td>[0.7908196055445871]</td>\n      <td>[0.772448410062275]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.7786772665370782]</td>\n      <td>[0.7855784879651856]</td>\n      <td>[0.7721631205673759]</td>\n      <td>[0.7835922951274648]</td>\n      <td>[0.7730122814372734]</td>\n      <td>[0.7744847871584888]</td>\n      <td>[0.7730122814372734]</td>\n      <td>...</td>\n      <td>[0.7906573862807009]</td>\n      <td>[0.5971856577566018]</td>\n      <td>[0.776355093319379]</td>\n      <td>[0.8572850121746318]</td>\n      <td>[0.7759960359081934]</td>\n      <td>[0.7864891147777485]</td>\n      <td>[0.7202719797596457]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.7766115551902172]</td>\n      <td>[0.6495818344433837]</td>\n    </tr>\n    <tr>\n      <th>accuracy</th>\n      <td>[0.9110773899848255]</td>\n      <td>[0.903793626707132]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8931714719271624]</td>\n      <td>[0.9128983308042489]</td>\n      <td>[0.9050075872534142]</td>\n      <td>[0.9119878603945372]</td>\n      <td>[0.908649468892261]</td>\n      <td>[0.9089529590288316]</td>\n      <td>[0.908649468892261]</td>\n      <td>...</td>\n      <td>[0.8968133535660091]</td>\n      <td>[0.7034901365705615]</td>\n      <td>[0.9044006069802731]</td>\n      <td>[0.9010622154779969]</td>\n      <td>[0.9068285280728376]</td>\n      <td>[0.9122913505311078]</td>\n      <td>[0.8889226100151745]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9095599393019727]</td>\n      <td>[0.7981790591805766]</td>\n    </tr>\n    <tr>\n      <th>f1_score_micro</th>\n      <td>[0.9110773899848255]</td>\n      <td>[0.903793626707132]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8931714719271624]</td>\n      <td>[0.9128983308042489]</td>\n      <td>[0.9050075872534142]</td>\n      <td>[0.9119878603945372]</td>\n      <td>[0.908649468892261]</td>\n      <td>[0.9089529590288316]</td>\n      <td>[0.908649468892261]</td>\n      <td>...</td>\n      <td>[0.8968133535660091]</td>\n      <td>[0.7034901365705615]</td>\n      <td>[0.9044006069802731]</td>\n      <td>[0.9010622154779969]</td>\n      <td>[0.9068285280728376]</td>\n      <td>[0.9122913505311078]</td>\n      <td>[0.8889226100151746]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9095599393019727]</td>\n      <td>[0.7981790591805767]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_macro</th>\n      <td>[0.8022857542994071]</td>\n      <td>[0.8015553269193132]</td>\n      <td>[0.6840348429031319]</td>\n      <td>[0.7600198800090151]</td>\n      <td>[0.8219228160768492]</td>\n      <td>[0.7595699066838024]</td>\n      <td>[0.8144743900739468]</td>\n      <td>[0.8128587183770495]</td>\n      <td>[0.8088969274697876]</td>\n      <td>[0.7991745048777775]</td>\n      <td>...</td>\n      <td>[0.7620204361207507]</td>\n      <td>[0.7023261438469145]</td>\n      <td>[0.7630054059702478]</td>\n      <td>[0.7661826412910435]</td>\n      <td>[0.8045843908248679]</td>\n      <td>[0.8050360880499143]</td>\n      <td>[0.7444849772802506]</td>\n      <td>[0.5]</td>\n      <td>[0.8041712317790284]</td>\n      <td>[0.7520694885582426]</td>\n    </tr>\n    <tr>\n      <th>weighted_accuracy</th>\n      <td>[0.9611104717292754]</td>\n      <td>[0.9627381552023843]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.979774506570003]</td>\n      <td>[0.9525453867527467]</td>\n      <td>[0.9596739896135009]</td>\n      <td>[0.9527120986999115]</td>\n      <td>[0.9490115533678862]</td>\n      <td>[0.950229930267267]</td>\n      <td>[0.9490115533678862]</td>\n      <td>...</td>\n      <td>[0.9779317071981736]</td>\n      <td>[0.6986651661824931]</td>\n      <td>[0.9639989574179747]</td>\n      <td>[0.9823475042704128]</td>\n      <td>[0.9584586020317934]</td>\n      <td>[0.9551064271824634]</td>\n      <td>[0.980356503726242]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.9511967445870643]</td>\n      <td>[0.7992162698903164]</td>\n    </tr>\n    <tr>\n      <th>AUC_weighted</th>\n      <td>[0.9406868983248957]</td>\n      <td>[0.9366556635491167]</td>\n      <td>[0.8291418679737037]</td>\n      <td>[0.9184773648830132]</td>\n      <td>[0.9439961692850012]</td>\n      <td>[0.9190117755586305]</td>\n      <td>[0.9419780048791601]</td>\n      <td>[0.9426115177077949]</td>\n      <td>[0.9370849518474679]</td>\n      <td>[0.937106254179425]</td>\n      <td>...</td>\n      <td>[0.9154468766150411]</td>\n      <td>[0.8549964156511013]</td>\n      <td>[0.9200699457438869]</td>\n      <td>[0.915315820964088]</td>\n      <td>[0.9399149203385405]</td>\n      <td>[0.9393707846852903]</td>\n      <td>[0.9088260192239652]</td>\n      <td>[0.5]</td>\n      <td>[0.9386214983134111]</td>\n      <td>[0.8885022052544517]</td>\n    </tr>\n    <tr>\n      <th>matthews_correlation</th>\n      <td>[0.4937290323447371]</td>\n      <td>[0.42581079306268055]</td>\n      <td>[0.0]</td>\n      <td>[0.22234411229941844]</td>\n      <td>[0.5378120541497806]</td>\n      <td>[0.448559599848506]</td>\n      <td>[0.5303542542460528]</td>\n      <td>[0.5183910930168123]</td>\n      <td>[0.5162034816601402]</td>\n      <td>[0.5183910930168123]</td>\n      <td>...</td>\n      <td>[0.2854476429412161]</td>\n      <td>[0.29438099191554834]</td>\n      <td>[0.42623303560744186]</td>\n      <td>[0.3244525388373706]</td>\n      <td>[0.46856282431427704]</td>\n      <td>[0.5242574455664286]</td>\n      <td>[0.13486362497228446]</td>\n      <td>[0.0]</td>\n      <td>[0.5172993681045217]</td>\n      <td>[0.4194151268149887]</td>\n    </tr>\n    <tr>\n      <th>balanced_accuracy</th>\n      <td>[0.7095528918378726]</td>\n      <td>[0.6663753804318631]</td>\n      <td>[0.5]</td>\n      <td>[0.5443496027578184]</td>\n      <td>[0.7532069271478771]</td>\n      <td>[0.6848208844357754]</td>\n      <td>[0.7479577546971642]</td>\n      <td>[0.7460780554490439]</td>\n      <td>[0.7426965417979539]</td>\n      <td>[0.7460780554490439]</td>\n      <td>...</td>\n      <td>[0.5700828197618955]</td>\n      <td>[0.7229242729884577]</td>\n      <td>[0.6643488803309086]</td>\n      <td>[0.5736592960598096]</td>\n      <td>[0.6988716247381203]</td>\n      <td>[0.7398397138448487]</td>\n      <td>[0.5206428858546959]</td>\n      <td>[0.5]</td>\n      <td>[0.7418541734973056]</td>\n      <td>[0.7940013559397385]</td>\n    </tr>\n    <tr>\n      <th>recall_score_macro</th>\n      <td>[0.7095528918378726]</td>\n      <td>[0.6663753804318631]</td>\n      <td>[0.5]</td>\n      <td>[0.5443496027578184]</td>\n      <td>[0.7532069271478771]</td>\n      <td>[0.6848208844357754]</td>\n      <td>[0.7479577546971642]</td>\n      <td>[0.7460780554490439]</td>\n      <td>[0.7426965417979539]</td>\n      <td>[0.7460780554490439]</td>\n      <td>...</td>\n      <td>[0.5700828197618955]</td>\n      <td>[0.7229242729884577]</td>\n      <td>[0.6643488803309086]</td>\n      <td>[0.5736592960598096]</td>\n      <td>[0.6988716247381203]</td>\n      <td>[0.7398397138448487]</td>\n      <td>[0.5206428858546959]</td>\n      <td>[0.5]</td>\n      <td>[0.7418541734973056]</td>\n      <td>[0.7940013559397385]</td>\n    </tr>\n    <tr>\n      <th>AUC_macro</th>\n      <td>[0.9406868983248957]</td>\n      <td>[0.9366556635491168]</td>\n      <td>[0.8291418679737037]</td>\n      <td>[0.9184773648830131]</td>\n      <td>[0.9439961692850012]</td>\n      <td>[0.9190117755586305]</td>\n      <td>[0.9419780048791602]</td>\n      <td>[0.9426115177077949]</td>\n      <td>[0.9370849518474678]</td>\n      <td>[0.937106254179425]</td>\n      <td>...</td>\n      <td>[0.915446876615041]</td>\n      <td>[0.8549964156511012]</td>\n      <td>[0.9200699457438868]</td>\n      <td>[0.915315820964088]</td>\n      <td>[0.9399149203385404]</td>\n      <td>[0.9393707846852904]</td>\n      <td>[0.9088260192239652]</td>\n      <td>[0.5]</td>\n      <td>[0.9386214983134111]</td>\n      <td>[0.8885022052544518]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_micro</th>\n      <td>[0.9792469103727843]</td>\n      <td>[0.9782235801974871]</td>\n      <td>[0.9514423062659481]</td>\n      <td>[0.9727129482425465]</td>\n      <td>[0.9803460970885172]</td>\n      <td>[0.9736438309245249]</td>\n      <td>[0.9799005886582354]</td>\n      <td>[0.9798331125567491]</td>\n      <td>[0.9788087720258263]</td>\n      <td>[0.9778754906463094]</td>\n      <td>...</td>\n      <td>[0.9724336547737189]</td>\n      <td>[0.8216086940681235]</td>\n      <td>[0.9679693407446847]</td>\n      <td>[0.9722607266220833]</td>\n      <td>[0.9790986967072279]</td>\n      <td>[0.9791629630104502]</td>\n      <td>[0.9703871559841004]</td>\n      <td>[0.8445594902839405]</td>\n      <td>[0.9788608451294213]</td>\n      <td>[0.8577423292287286]</td>\n    </tr>\n    <tr>\n      <th>f1_score_macro</th>\n      <td>[0.7410400670578374]</td>\n      <td>[0.7017444770694911]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.5544040176787701]</td>\n      <td>[0.7680037996089013]</td>\n      <td>[0.7168551535252986]</td>\n      <td>[0.7640701170131234]</td>\n      <td>[0.7585358306343567]</td>\n      <td>[0.7571849668386146]</td>\n      <td>[0.7585358306343567]</td>\n      <td>...</td>\n      <td>[0.5945349258052841]</td>\n      <td>[0.5839877109467246]</td>\n      <td>[0.7008431471845663]</td>\n      <td>[0.6013584326439484]</td>\n      <td>[0.7286665549036044]</td>\n      <td>[0.7602368688673142]</td>\n      <td>[0.5129361873990307]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.7575617064479683]</td>\n      <td>[0.6711291077624292]</td>\n    </tr>\n    <tr>\n      <th>precision_score_micro</th>\n      <td>[0.9110773899848255]</td>\n      <td>[0.903793626707132]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8931714719271624]</td>\n      <td>[0.9128983308042489]</td>\n      <td>[0.9050075872534142]</td>\n      <td>[0.9119878603945372]</td>\n      <td>[0.908649468892261]</td>\n      <td>[0.9089529590288316]</td>\n      <td>[0.908649468892261]</td>\n      <td>...</td>\n      <td>[0.8968133535660091]</td>\n      <td>[0.7034901365705615]</td>\n      <td>[0.9044006069802731]</td>\n      <td>[0.9010622154779969]</td>\n      <td>[0.9068285280728376]</td>\n      <td>[0.9122913505311078]</td>\n      <td>[0.8889226100151745]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9095599393019727]</td>\n      <td>[0.7981790591805766]</td>\n    </tr>\n    <tr>\n      <th>recall_score_micro</th>\n      <td>[0.9110773899848255]</td>\n      <td>[0.903793626707132]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8931714719271624]</td>\n      <td>[0.9128983308042489]</td>\n      <td>[0.9050075872534142]</td>\n      <td>[0.9119878603945372]</td>\n      <td>[0.908649468892261]</td>\n      <td>[0.9089529590288316]</td>\n      <td>[0.908649468892261]</td>\n      <td>...</td>\n      <td>[0.8968133535660091]</td>\n      <td>[0.7034901365705615]</td>\n      <td>[0.9044006069802731]</td>\n      <td>[0.9010622154779969]</td>\n      <td>[0.9068285280728376]</td>\n      <td>[0.9122913505311078]</td>\n      <td>[0.8889226100151745]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9095599393019727]</td>\n      <td>[0.7981790591805766]</td>\n    </tr>\n    <tr>\n      <th>log_loss</th>\n      <td>[0.18485719214536023]</td>\n      <td>[0.19892711190385837]</td>\n      <td>[0.2789041773663072]</td>\n      <td>[0.23663712579666943]</td>\n      <td>[0.18546137380446998]</td>\n      <td>[0.21392914688531073]</td>\n      <td>[0.18558180944526662]</td>\n      <td>[0.18276368510928603]</td>\n      <td>[0.3214779057369575]</td>\n      <td>[0.22829568238730566]</td>\n      <td>...</td>\n      <td>[0.23235315960994582]</td>\n      <td>[0.5425420944004252]</td>\n      <td>[0.22539140754429898]</td>\n      <td>[0.22149830847512975]</td>\n      <td>[0.1855209800247038]</td>\n      <td>[0.21102079316404485]</td>\n      <td>[0.24793059362231587]</td>\n      <td>[0.35067519027071736]</td>\n      <td>[0.1876484785138208]</td>\n      <td>[0.5089305587772139]</td>\n    </tr>\n  </tbody>\n</table>\n<p>21 rows × 33 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 15,
      "metadata": {
        "gather": {
          "logged": 1618854528750
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrieve the Best Model"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Retrieve best model from Pipeline Run\n",
        "best_model_output = pipeline_run.get_pipeline_output(best_model_output_name)\n",
        "num_file_downloaded = best_model_output.download('.', show_progress=True)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading azureml/4a11362c-6199-4ec8-90cf-8f94516c3773/model_data\n",
            "Downloaded azureml/4a11362c-6199-4ec8-90cf-8f94516c3773/model_data, 1 files out of an estimated total of 1\n"
          ]
        }
      ],
      "execution_count": 16,
      "metadata": {
        "gather": {
          "logged": 1618854532609
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "with open(best_model_output._path_on_datastore, \"rb\" ) as f:\n",
        "    best_model = pickle.load(f)\n",
        "best_model"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 17,
          "data": {
            "text/plain": "PipelineWithYTransformations(Pipeline={'memory': None,\n                                       'steps': [('datatransformer',\n                                                  DataTransformer(enable_dnn=None,\n                                                                  enable_feature_sweeping=None,\n                                                                  feature_sweeping_config=None,\n                                                                  feature_sweeping_timeout=None,\n                                                                  featurization_config=None,\n                                                                  force_text_dnn=None,\n                                                                  is_cross_validation=None,\n                                                                  is_onnx_compatible=None,\n                                                                  logger=None,\n                                                                  observer=None,\n                                                                  task=None,\n                                                                  working_dir=None))...\n                                                                                                                                     oob_score=True,\n                                                                                                                                     random_state=None,\n                                                                                                                                     verbose=0,\n                                                                                                                                     warm_start=False))],\n                                                                                                      verbose=False))],\n                                                                                flatten_transform=None,\n                                                                                weights=[0.15384615384615385,\n                                                                                         0.07692307692307693,\n                                                                                         0.15384615384615385,\n                                                                                         0.15384615384615385,\n                                                                                         0.15384615384615385,\n                                                                                         0.07692307692307693,\n                                                                                         0.07692307692307693,\n                                                                                         0.07692307692307693,\n                                                                                         0.07692307692307693]))],\n                                       'verbose': False},\n                             y_transformer={},\n                             y_transformer_name='LabelEncoder')"
          },
          "metadata": {}
        }
      ],
      "execution_count": 17,
      "metadata": {
        "gather": {
          "logged": 1618854546464
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "best_model.steps"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 18,
          "data": {
            "text/plain": "[('datatransformer',\n  DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n                  feature_sweeping_config=None, feature_sweeping_timeout=None,\n                  featurization_config=None, force_text_dnn=None,\n                  is_cross_validation=None, is_onnx_compatible=None, logger=None,\n                  observer=None, task=None, working_dir=None)),\n ('prefittedsoftvotingclassifier',\n  PreFittedSoftVotingClassifier(classification_labels=None,\n                                estimators=[('17',\n                                             Pipeline(memory=None,\n                                                      steps=[('sparsenormalizer',\n                                                              <azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f3b96d289b0>),\n                                                             ('xgboostclassifier',\n                                                              XGBoostClassifier(base_score=0.5,\n                                                                                booster='gbtree',\n                                                                                colsample_bylevel=1,\n                                                                                colsample_bynode=1,\n                                                                                colsample_bytree=0.7,\n                                                                                eta=0.3,\n                                                                                g...\n                                                                                     min_samples_split=0.01,\n                                                                                     min_weight_fraction_leaf=0.0,\n                                                                                     n_estimators=25,\n                                                                                     n_jobs=1,\n                                                                                     oob_score=True,\n                                                                                     random_state=None,\n                                                                                     verbose=0,\n                                                                                     warm_start=False))],\n                                                      verbose=False))],\n                                flatten_transform=None,\n                                weights=[0.15384615384615385, 0.07692307692307693,\n                                         0.15384615384615385, 0.15384615384615385,\n                                         0.15384615384615385, 0.07692307692307693,\n                                         0.07692307692307693, 0.07692307692307693,\n                                         0.07692307692307693]))]"
          },
          "metadata": {}
        }
      ],
      "execution_count": 18,
      "metadata": {
        "gather": {
          "logged": 1618854546611
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Test the Model\n",
        "#### Load Test Data\n",
        "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_test = Dataset.Tabular.from_delimited_files(path='https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv')\n",
        "df_test = dataset_test.to_pandas_dataframe()\n",
        "df_test = df_test[pd.notnull(df_test['y'])]\n",
        "\n",
        "y_test = df_test['y']\n",
        "X_test = df_test.drop(['y'], axis=1)"
      ],
      "outputs": [],
      "execution_count": 19,
      "metadata": {
        "gather": {
          "logged": 1618854550390
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Testing Our Best Fitted Model\n",
        "\n",
        "We will use confusion matrix to see how our model works."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "ypred = best_model.predict(X_test)\n",
        "cm = confusion_matrix(y_test, ypred)"
      ],
      "outputs": [],
      "execution_count": 20,
      "metadata": {
        "gather": {
          "logged": 1618854555540
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the confusion matrix\n",
        "pd.DataFrame(cm).style.background_gradient(cmap='Blues', low=0, high=0.9)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 21,
          "data": {
            "text/plain": "<pandas.io.formats.style.Styler at 0x7f3b8cf799e8>",
            "text/html": "<style  type=\"text/css\" >\n    #T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row0_col0 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }    #T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row0_col1 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row1_col0 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row1_col1 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }</style><table id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >0</th>        <th class=\"col_heading level0 col1\" >1</th>    </tr></thead><tbody>\n                <tr>\n                        <th id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174level0_row0\" class=\"row_heading level0 row0\" >0</th>\n                        <td id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row0_col0\" class=\"data row0 col0\" >28719</td>\n                        <td id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row0_col1\" class=\"data row0 col1\" >539</td>\n            </tr>\n            <tr>\n                        <th id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174level0_row1\" class=\"row_heading level0 row1\" >1</th>\n                        <td id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row1_col0\" class=\"data row1 col0\" >1174</td>\n                        <td id=\"T_8ec5f89c_a137_11eb_bb7c_dd8b463ef174row1_col1\" class=\"data row1 col1\" >2518</td>\n            </tr>\n    </tbody></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 21,
      "metadata": {
        "gather": {
          "logged": 1618854555721
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Publish and run from REST endpoint\n",
        "\n",
        "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
        "\n",
        "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "published_pipeline = pipeline_run.publish_pipeline(\n",
        "    name=\"Bankmarketing Train\", description=\"Training bankmarketing pipeline\", version=\"1.0\")\n",
        "\n",
        "published_pipeline\n"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 22,
          "data": {
            "text/plain": "Pipeline(Name: Bankmarketing Train,\nId: 969cd5e5-605d-46cb-b721-7a5b662e3f74,\nStatus: Active,\nEndpoint: https://eastus2.api.azureml.ms/pipelines/v1.0/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourceGroups/UdacityAzureMLP2/providers/Microsoft.MachineLearningServices/workspaces/ws_udacity_azureml_p2/PipelineRuns/PipelineSubmit/969cd5e5-605d-46cb-b721-7a5b662e3f74)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Id</th><th>Status</th><th>Endpoint</th></tr><tr><td>Bankmarketing Train</td><td><a href=\"https://ml.azure.com/pipelines/969cd5e5-605d-46cb-b721-7a5b662e3f74?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2\" target=\"_blank\" rel=\"noopener\">969cd5e5-605d-46cb-b721-7a5b662e3f74</a></td><td>Active</td><td><a href=\"https://eastus2.api.azureml.ms/pipelines/v1.0/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourceGroups/UdacityAzureMLP2/providers/Microsoft.MachineLearningServices/workspaces/ws_udacity_azureml_p2/PipelineRuns/PipelineSubmit/969cd5e5-605d-46cb-b721-7a5b662e3f74\" target=\"_blank\" rel=\"noopener\">REST Endpoint</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 22,
      "metadata": {
        "gather": {
          "logged": 1618854558780
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Authenticate once again, to retrieve the `auth_header` so that the endpoint can be used"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
        "\n",
        "interactive_auth = InteractiveLoginAuthentication()\n",
        "auth_header = interactive_auth.get_authentication_header()\n",
        "\n"
      ],
      "outputs": [],
      "execution_count": 23,
      "metadata": {
        "gather": {
          "logged": 1618854562172
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get the REST url from the endpoint property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the process_count_per_node is passed through to ParallelRunStep because you defined it is defined as a PipelineParameter object in the step configuration.\n",
        "\n",
        "Make the request to trigger the run. Access the Id key from the response dict to get the value of the run id.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "rest_endpoint = published_pipeline.endpoint\n",
        "response = requests.post(rest_endpoint, \n",
        "                         headers=auth_header, \n",
        "                         json={\"ExperimentName\": \"pipeline-rest-endpoint\"}\n",
        "                        )"
      ],
      "outputs": [],
      "execution_count": 24,
      "metadata": {
        "gather": {
          "logged": 1618854565982
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    response.raise_for_status()\n",
        "except Exception:    \n",
        "    raise Exception(\"Received bad response from the endpoint: {}\\n\"\n",
        "                    \"Response Code: {}\\n\"\n",
        "                    \"Headers: {}\\n\"\n",
        "                    \"Content: {}\".format(rest_endpoint, response.status_code, response.headers, response.content))\n",
        "\n",
        "run_id = response.json().get('Id')\n",
        "print('Submitted pipeline run: ', run_id)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Submitted pipeline run:  465325f0-a7df-40b2-993b-763a595c6b7b\n"
          ]
        }
      ],
      "execution_count": 25,
      "metadata": {
        "gather": {
          "logged": 1618854566077
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core.run import PipelineRun\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "published_pipeline_run = PipelineRun(ws.experiments[\"pipeline-rest-endpoint\"], run_id)\n",
        "RunDetails(published_pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cf3b5e65a7504437a38eb5847b96e69b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/465325f0-a7df-40b2-993b-763a595c6b7b?wsid=/subscriptions/ca1598e0-85dc-47d5-b06d-41b5342b4989/resourcegroups/UdacityAzureMLP2/workspaces/ws_udacity_azureml_p2&tid=3814e260-63cb-43a8-82ce-e862c309e004\", \"run_id\": \"465325f0-a7df-40b2-993b-763a595c6b7b\", \"run_properties\": {\"run_id\": \"465325f0-a7df-40b2-993b-763a595c6b7b\", \"created_utc\": \"2021-04-19T17:49:22.509922Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"Unavailable\", \"runType\": \"HTTP\", \"azureml.parameters\": \"{}\", \"azureml.pipelineid\": \"969cd5e5-605d-46cb-b721-7a5b662e3f74\"}, \"tags\": {\"azureml.pipelineid\": \"969cd5e5-605d-46cb-b721-7a5b662e3f74\", \"azureml.pipelineComponent\": \"pipelinerun\"}, \"end_time_utc\": \"2021-04-19T17:49:29.300402Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.465325f0-a7df-40b2-993b-763a595c6b7b/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=3eobmNt4BQODf0bb8gHF5%2BebctxdZz5N21c9g0ek5RE%3D&st=2021-04-19T17%3A39%3A47Z&se=2021-04-20T01%3A49%3A47Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.465325f0-a7df-40b2-993b-763a595c6b7b/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=AKxZo6VHTSuoXMgwqgqyRcfDbTOooAtTRj77IT6qDNA%3D&st=2021-04-19T17%3A39%3A47Z&se=2021-04-20T01%3A49%3A47Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://wsudacityazure9537564226.blob.core.windows.net/azureml/ExperimentRun/dcid.465325f0-a7df-40b2-993b-763a595c6b7b/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=m8zMda8c3M3iXaWefb%2Fip51RsVxWM8MKdvtfaZ6QtFI%3D&st=2021-04-19T17%3A39%3A47Z&se=2021-04-20T01%3A49%3A47Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:00:06\", \"run_number\": \"1\", \"run_queued_details\": {\"status\": \"Finished\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"45ef33e2-7a10-4b8d-9b5d-fdfa8672b1a0\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-04-19T17:49:28.949087Z\", \"created_time\": \"2021-04-19T17:49:28.949087Z\", \"end_time\": \"2021-04-19T17:49:29.011825Z\", \"duration\": \"0:00:00\", \"run_number\": 2, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-04-19T17:49:28.949087Z\", \"is_reused\": \"Yes\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2021-04-19 17:49:29Z] Completing processing run id 45ef33e2-7a10-4b8d-9b5d-fdfa8672b1a0.\\n[2021-04-19 17:49:29Z] Finishing experiment: no runs left and nothing to schedule.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"8340727b\": {\"node_id\": \"8340727b\", \"name\": \"bankmarketing_train\"}}, \"module_nodes\": {\"cd06412c\": {\"node_id\": \"cd06412c\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": true, \"run_id\": \"45ef33e2-7a10-4b8d-9b5d-fdfa8672b1a0\"}}, \"edges\": [{\"source_node_id\": \"8340727b\", \"source_node_name\": \"bankmarketing_train\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"cd06412c\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"45ef33e2-7a10-4b8d-9b5d-fdfa8672b1a0\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-04-19T17:49:28.949087Z\", \"created_time\": \"2021-04-19T17:49:28.949087Z\", \"end_time\": \"2021-04-19T17:49:29.011825Z\", \"duration\": \"0:00:00\", \"run_number\": 2, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-04-19T17:49:28.949087Z\", \"is_reused\": \"Yes\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.26.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 26,
      "metadata": {
        "gather": {
          "logged": 1618854566626
        }
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    }
  ],
  "metadata": {
    "order_index": 11,
    "exclude_from_index": false,
    "task": "Demonstrates the use of AutoMLStep",
    "deployment": [
      "None"
    ],
    "authors": [
      {
        "name": "sanpil"
      }
    ],
    "star_tag": [
      "featured"
    ],
    "metadata": {
      "interpreter": {
        "hash": "2be2c946c6ea3fae536cfdd279941e292763a35926b8ae2b3cb39ac4fc53c4bc"
      }
    },
    "kernel_info": {
      "name": "python3-azureml"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.9",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "compute": [
      "AML Compute"
    ],
    "kernelspec": {
      "name": "python3-azureml",
      "language": "python",
      "display_name": "Python 3.6 - AzureML"
    },
    "tags": [
      "None"
    ],
    "datasets": [
      "Custom"
    ],
    "categories": [
      "how-to-use-azureml",
      "machine-learning-pipelines",
      "intro-to-pipelines"
    ],
    "category": "tutorial",
    "framework": [
      "Automated Machine Learning"
    ],
    "friendly_name": "How to use AutoMLStep with AML Pipelines",
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}