examples/pzmm_h2o_model_import.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "Copyright © 2021, SAS Institute Inc., Cary, NC, USA.  All Rights Reserved.\n",
    "SPDX-License-Identifier: Apache-2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "# HMEQ Dataset : Build and Import Trained H2O.ai Models into SAS Model Manager\n",
    "\n",
    "This notebook provides an example of how to build and train a Python model and then import the model into SAS Model Manager using the HMEQ data set. Lines of code that must be modified by the user, such as directory paths are noted with the comment \"_Changes required by user._\".\n",
    "\n",
    "_**Note:** If you download only this notebook and not the rest of the repository, you must also download the hmeq.csv file from the data folder in the examples directory. These files are used when executing this notebook example._\n",
    "\n",
    "Here are the steps shown in this notebook:\n",
    "\n",
    "1. Import and review data and preprocess for model training.\n",
    "2. Build, train, and access an H2O.ai generalized linear estimator model.\n",
    "3. Serialize the model into pickle or MOJO files.\n",
    "4. Write the metadata JSON files needed for importing into SAS Model Manager.\n",
    "4. Write a score code Python file for model scoring.\n",
    "5. Zip the model, JSON, and score code files into an archive file.\n",
    "6. Import the ZIP archive file to SAS Model Manager via the Session object and relevant function call."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Python Package Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "# Standard Library\n",
    "from pathlib import Path\n",
    "import warnings\n",
    "\n",
    "# Third Party\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import h2o\n",
    "import pprint as pp\n",
    "from h2o.estimators.glm import H2OGeneralizedLinearEstimator\n",
    "\n",
    "# Application Specific\n",
    "import sasctl.pzmm as pzmm\n",
    "from sasctl import Session, publish_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Global Package Options\n",
    "pd.options.mode.chained_assignment = None  # default=\"warn\"\n",
    "plt.rc(\"font\", size=14)\n",
    "# Ignore warnings from pandas about SWAT using a feature that will be depreciated soon\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.40.0.3'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On SAS Viya, models created in H2O versions 3.24 and under are only compatible in the binary model format. For H2O versions 3.26+, models can be in the MOJO or binary model format. If using a binary model, the H2O version on the SAS Viya server must match the exact version of H2O used to create the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321..... not found.\n",
      "Attempting to start a local H2O server...\n",
      "  Java Version: openjdk version \"11.0.13\" 2021-10-19; OpenJDK Runtime Environment JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21); OpenJDK 64-Bit Server VM JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21, mixed mode)\n",
      "  Starting server from /Users/dalmoo/opt/anaconda3/envs/yeehaw/lib/python3.8/site-packages/h2o/backend/bin/h2o.jar\n",
      "  Ice root: /var/folders/vs/np2dp7cs1y7ggk5pl92q_rb40000gn/T/tmp0_92875_\n",
      "  JVM stdout: /var/folders/vs/np2dp7cs1y7ggk5pl92q_rb40000gn/T/tmp0_92875_/h2o_dalmoo_started_from_python.out\n",
      "  JVM stderr: /var/folders/vs/np2dp7cs1y7ggk5pl92q_rb40000gn/T/tmp0_92875_/h2o_dalmoo_started_from_python.err\n",
      "  Server is running at http://127.0.0.1:54321\n",
      "Connecting to H2O server at http://127.0.0.1:54321 ... successful.\n",
      "Warning: Your H2O cluster version is (6 months and 1 day) old.  There may be a newer version available.\n",
      "Please download and install the latest version from: https://h2o-release.s3.amazonaws.com/h2o/latest_stable.html\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-1.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-1 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-1 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-1 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table th,\n",
       "#h2o-table-1 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-1 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-1\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption></caption>\n",
       "    <thead></thead>\n",
       "    <tbody><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>02 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/Chicago</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.40.0.3</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>6 months and 1 day</td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_dalmoo_6awy1u</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>4 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>10</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>10</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://127.0.0.1:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.8.16 final</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "--------------------------  -----------------------------\n",
       "H2O_cluster_uptime:         02 secs\n",
       "H2O_cluster_timezone:       America/Chicago\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.40.0.3\n",
       "H2O_cluster_version_age:    6 months and 1 day\n",
       "H2O_cluster_name:           H2O_from_python_dalmoo_6awy1u\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    4 Gb\n",
       "H2O_cluster_total_cores:    10\n",
       "H2O_cluster_allowed_cores:  10\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://127.0.0.1:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.8.16 final\n",
       "--------------------------  -----------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Import and Review Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(5960, 13)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hmeq_data = h2o.import_file(\"data/hmeq.csv\", sep= \",\")\n",
    "hmeq_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Preprocess Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "hmeq_data[\"BAD\"] = hmeq_data[\"BAD\"].asfactor()\n",
    "\n",
    "train, validation, test = hmeq_data.split_frame(ratios=[.6, .2], seed=42)\n",
    "\n",
    "y = \"BAD\"\n",
    "x = list(hmeq_data.columns)\n",
    "x.remove(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Create, Train, and Assess Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glm Model Build progress: |██████████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style='margin: 1em 0 1em 0;'>Model Details\n",
       "=============\n",
       "H2OGeneralizedLinearEstimator : Generalized Linear Modeling\n",
       "Model Key: glmfit\n",
       "</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-2.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-2 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-2 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-2 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-2 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-2 .h2o-table th,\n",
       "#h2o-table-2 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-2 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-2\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>GLM Model: summary</caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>family</th>\n",
       "<th>link</th>\n",
       "<th>regularization</th>\n",
       "<th>lambda_search</th>\n",
       "<th>number_of_predictors_total</th>\n",
       "<th>number_of_active_predictors</th>\n",
       "<th>number_of_iterations</th>\n",
       "<th>training_frame</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>binomial</td>\n",
       "<td>logit</td>\n",
       "<td>Elastic Net (alpha = 0.5, lambda = 9.244E-4 )</td>\n",
       "<td>nlambda = 100, lambda.max = 0.2455, lambda.min = 9.244E-4, lambda.1se = -1.0</td>\n",
       "<td>18</td>\n",
       "<td>17</td>\n",
       "<td>93</td>\n",
       "<td>py_3_sid_a269</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialGLM: glm\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.12410463474559494\n",
       "RMSE: 0.3522848772592927\n",
       "LogLoss: 0.40208197287665776\n",
       "AUC: 0.7849415147958837\n",
       "AUCPR: 0.5691248967099318\n",
       "Gini: 0.5698830295917674\n",
       "Null degrees of freedom: 3580\n",
       "Residual degrees of freedom: 3563\n",
       "Null deviance: 3597.150438148379\n",
       "Residual deviance: 2879.711089742623\n",
       "AIC: 2915.711089742623</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-3.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-3 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-3 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-3 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-3 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-3 .h2o-table th,\n",
       "#h2o-table-3 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-3 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-3\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2574171649920362</caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>0</th>\n",
       "<th>1</th>\n",
       "<th>Error</th>\n",
       "<th>Rate</th></tr></thead>\n",
       "    <tbody><tr><td>0</td>\n",
       "<td>2446.0</td>\n",
       "<td>414.0</td>\n",
       "<td>0.1448</td>\n",
       "<td> (414.0/2860.0)</td></tr>\n",
       "<tr><td>1</td>\n",
       "<td>314.0</td>\n",
       "<td>407.0</td>\n",
       "<td>0.4355</td>\n",
       "<td> (314.0/721.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>2760.0</td>\n",
       "<td>821.0</td>\n",
       "<td>0.2033</td>\n",
       "<td> (728.0/3581.0)</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-4.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-4 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-4 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-4 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-4 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-4 .h2o-table th,\n",
       "#h2o-table-4 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-4 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-4\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Maximum Metrics: Maximum metrics at their respective thresholds</caption>\n",
       "    <thead><tr><th>metric</th>\n",
       "<th>threshold</th>\n",
       "<th>value</th>\n",
       "<th>idx</th></tr></thead>\n",
       "    <tbody><tr><td>max f1</td>\n",
       "<td>0.2574172</td>\n",
       "<td>0.5278859</td>\n",
       "<td>207.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1522967</td>\n",
       "<td>0.6261121</td>\n",
       "<td>279.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.3886058</td>\n",
       "<td>0.5571256</td>\n",
       "<td>144.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5672592</td>\n",
       "<td>0.8352416</td>\n",
       "<td>90.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9988092</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0012377</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9988092</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_mcc</td>\n",
       "<td>0.2661852</td>\n",
       "<td>0.4020977</td>\n",
       "<td>203.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.1782796</td>\n",
       "<td>0.7066434</td>\n",
       "<td>258.0</td></tr>\n",
       "<tr><td>max mean_per_class_accuracy</td>\n",
       "<td>0.2143151</td>\n",
       "<td>0.7195026</td>\n",
       "<td>234.0</td></tr>\n",
       "<tr><td>max tns</td>\n",
       "<td>0.9988092</td>\n",
       "<td>2860.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fns</td>\n",
       "<td>0.9988092</td>\n",
       "<td>717.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fps</td>\n",
       "<td>0.0062627</td>\n",
       "<td>2860.0</td>\n",
       "<td>398.0</td></tr>\n",
       "<tr><td>max tps</td>\n",
       "<td>0.0012377</td>\n",
       "<td>721.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tnr</td>\n",
       "<td>0.9988092</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fnr</td>\n",
       "<td>0.9988092</td>\n",
       "<td>0.9944521</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fpr</td>\n",
       "<td>0.0062627</td>\n",
       "<td>1.0</td>\n",
       "<td>398.0</td></tr>\n",
       "<tr><td>max tpr</td>\n",
       "<td>0.0012377</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-5.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-5 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-5 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-5 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-5 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-5 .h2o-table th,\n",
       "#h2o-table-5 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-5 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-5\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Gains/Lift Table: Avg response rate: 20.13 %, avg score: 20.13 %</caption>\n",
       "    <thead><tr><th>group</th>\n",
       "<th>cumulative_data_fraction</th>\n",
       "<th>lower_threshold</th>\n",
       "<th>lift</th>\n",
       "<th>cumulative_lift</th>\n",
       "<th>response_rate</th>\n",
       "<th>score</th>\n",
       "<th>cumulative_response_rate</th>\n",
       "<th>cumulative_score</th>\n",
       "<th>capture_rate</th>\n",
       "<th>cumulative_capture_rate</th>\n",
       "<th>gain</th>\n",
       "<th>cumulative_gain</th>\n",
       "<th>kolmogorov_smirnov</th></tr></thead>\n",
       "    <tbody><tr><td>1</td>\n",
       "<td>0.0100531</td>\n",
       "<td>0.9466824</td>\n",
       "<td>4.9667129</td>\n",
       "<td>4.9667129</td>\n",
       "<td>1.0</td>\n",
       "<td>0.9805069</td>\n",
       "<td>1.0</td>\n",
       "<td>0.9805069</td>\n",
       "<td>0.0499307</td>\n",
       "<td>0.0499307</td>\n",
       "<td>396.6712899</td>\n",
       "<td>396.6712899</td>\n",
       "<td>0.0499307</td></tr>\n",
       "<tr><td>2</td>\n",
       "<td>0.0201061</td>\n",
       "<td>0.8523402</td>\n",
       "<td>4.8287487</td>\n",
       "<td>4.8977308</td>\n",
       "<td>0.9722222</td>\n",
       "<td>0.8987941</td>\n",
       "<td>0.9861111</td>\n",
       "<td>0.9396505</td>\n",
       "<td>0.0485437</td>\n",
       "<td>0.0984743</td>\n",
       "<td>382.8748652</td>\n",
       "<td>389.7730775</td>\n",
       "<td>0.0981247</td></tr>\n",
       "<tr><td>3</td>\n",
       "<td>0.0301592</td>\n",
       "<td>0.7479558</td>\n",
       "<td>3.8629989</td>\n",
       "<td>4.5528202</td>\n",
       "<td>0.7777778</td>\n",
       "<td>0.7984072</td>\n",
       "<td>0.9166667</td>\n",
       "<td>0.8925694</td>\n",
       "<td>0.0388350</td>\n",
       "<td>0.1373093</td>\n",
       "<td>286.2998921</td>\n",
       "<td>355.2820157</td>\n",
       "<td>0.1341624</td></tr>\n",
       "<tr><td>4</td>\n",
       "<td>0.0402122</td>\n",
       "<td>0.6912340</td>\n",
       "<td>3.1731777</td>\n",
       "<td>4.2079095</td>\n",
       "<td>0.6388889</td>\n",
       "<td>0.7152449</td>\n",
       "<td>0.8472222</td>\n",
       "<td>0.8482383</td>\n",
       "<td>0.0319001</td>\n",
       "<td>0.1692094</td>\n",
       "<td>217.3177685</td>\n",
       "<td>320.7909539</td>\n",
       "<td>0.1615171</td></tr>\n",
       "<tr><td>5</td>\n",
       "<td>0.0502653</td>\n",
       "<td>0.6267302</td>\n",
       "<td>3.5870704</td>\n",
       "<td>4.0837417</td>\n",
       "<td>0.7222222</td>\n",
       "<td>0.6558936</td>\n",
       "<td>0.8222222</td>\n",
       "<td>0.8097693</td>\n",
       "<td>0.0360610</td>\n",
       "<td>0.2052705</td>\n",
       "<td>258.7070427</td>\n",
       "<td>308.3741717</td>\n",
       "<td>0.1940816</td></tr>\n",
       "<tr><td>6</td>\n",
       "<td>0.1002513</td>\n",
       "<td>0.4302463</td>\n",
       "<td>2.4417360</td>\n",
       "<td>3.2650257</td>\n",
       "<td>0.4916201</td>\n",
       "<td>0.5199749</td>\n",
       "<td>0.6573816</td>\n",
       "<td>0.6652757</td>\n",
       "<td>0.1220527</td>\n",
       "<td>0.3273232</td>\n",
       "<td>144.1735950</td>\n",
       "<td>226.5025750</td>\n",
       "<td>0.2843162</td></tr>\n",
       "<tr><td>7</td>\n",
       "<td>0.1502374</td>\n",
       "<td>0.3459178</td>\n",
       "<td>2.0532780</td>\n",
       "<td>2.8618606</td>\n",
       "<td>0.4134078</td>\n",
       "<td>0.3838564</td>\n",
       "<td>0.5762082</td>\n",
       "<td>0.5716436</td>\n",
       "<td>0.1026352</td>\n",
       "<td>0.4299584</td>\n",
       "<td>105.3277958</td>\n",
       "<td>186.1860592</td>\n",
       "<td>0.3502381</td></tr>\n",
       "<tr><td>8</td>\n",
       "<td>0.2002234</td>\n",
       "<td>0.2844205</td>\n",
       "<td>1.7203140</td>\n",
       "<td>2.5768720</td>\n",
       "<td>0.3463687</td>\n",
       "<td>0.3119030</td>\n",
       "<td>0.5188285</td>\n",
       "<td>0.5067990</td>\n",
       "<td>0.0859917</td>\n",
       "<td>0.5159501</td>\n",
       "<td>72.0313965</td>\n",
       "<td>157.6871964</td>\n",
       "<td>0.3953207</td></tr>\n",
       "<tr><td>9</td>\n",
       "<td>0.3001955</td>\n",
       "<td>0.2098441</td>\n",
       "<td>1.3318560</td>\n",
       "<td>2.1622527</td>\n",
       "<td>0.2681564</td>\n",
       "<td>0.2431147</td>\n",
       "<td>0.4353488</td>\n",
       "<td>0.4189860</td>\n",
       "<td>0.1331484</td>\n",
       "<td>0.6490985</td>\n",
       "<td>33.1855973</td>\n",
       "<td>116.2252685</td>\n",
       "<td>0.4368607</td></tr>\n",
       "<tr><td>10</td>\n",
       "<td>0.4001676</td>\n",
       "<td>0.1695731</td>\n",
       "<td>0.8879040</td>\n",
       "<td>1.8438878</td>\n",
       "<td>0.1787709</td>\n",
       "<td>0.1882520</td>\n",
       "<td>0.3712491</td>\n",
       "<td>0.3613428</td>\n",
       "<td>0.0887656</td>\n",
       "<td>0.7378641</td>\n",
       "<td>-11.2096018</td>\n",
       "<td>84.3887831</td>\n",
       "<td>0.4228291</td></tr>\n",
       "<tr><td>11</td>\n",
       "<td>0.5001396</td>\n",
       "<td>0.1380318</td>\n",
       "<td>0.6798015</td>\n",
       "<td>1.6112006</td>\n",
       "<td>0.1368715</td>\n",
       "<td>0.1525314</td>\n",
       "<td>0.3243998</td>\n",
       "<td>0.3196038</td>\n",
       "<td>0.0679612</td>\n",
       "<td>0.8058252</td>\n",
       "<td>-32.0198514</td>\n",
       "<td>61.1200555</td>\n",
       "<td>0.3827483</td></tr>\n",
       "<tr><td>12</td>\n",
       "<td>0.6001117</td>\n",
       "<td>0.1146235</td>\n",
       "<td>0.5410665</td>\n",
       "<td>1.4329279</td>\n",
       "<td>0.1089385</td>\n",
       "<td>0.1258838</td>\n",
       "<td>0.2885063</td>\n",
       "<td>0.2873322</td>\n",
       "<td>0.0540915</td>\n",
       "<td>0.8599168</td>\n",
       "<td>-45.8933511</td>\n",
       "<td>43.2927872</td>\n",
       "<td>0.3253014</td></tr>\n",
       "<tr><td>13</td>\n",
       "<td>0.7000838</td>\n",
       "<td>0.0945753</td>\n",
       "<td>0.5271930</td>\n",
       "<td>1.3035888</td>\n",
       "<td>0.1061453</td>\n",
       "<td>0.1047786</td>\n",
       "<td>0.2624651</td>\n",
       "<td>0.2612635</td>\n",
       "<td>0.0527046</td>\n",
       "<td>0.9126214</td>\n",
       "<td>-47.2807011</td>\n",
       "<td>30.3588786</td>\n",
       "<td>0.2661179</td></tr>\n",
       "<tr><td>14</td>\n",
       "<td>0.8000559</td>\n",
       "<td>0.0736148</td>\n",
       "<td>0.3884580</td>\n",
       "<td>1.1892374</td>\n",
       "<td>0.0782123</td>\n",
       "<td>0.0841132</td>\n",
       "<td>0.2394415</td>\n",
       "<td>0.2391274</td>\n",
       "<td>0.0388350</td>\n",
       "<td>0.9514563</td>\n",
       "<td>-61.1542008</td>\n",
       "<td>18.9237364</td>\n",
       "<td>0.1895682</td></tr>\n",
       "<tr><td>15</td>\n",
       "<td>0.9000279</td>\n",
       "<td>0.0521793</td>\n",
       "<td>0.2358495</td>\n",
       "<td>1.0833382</td>\n",
       "<td>0.0474860</td>\n",
       "<td>0.0627694</td>\n",
       "<td>0.2181198</td>\n",
       "<td>0.2195382</td>\n",
       "<td>0.0235784</td>\n",
       "<td>0.9750347</td>\n",
       "<td>-76.4150505</td>\n",
       "<td>8.3338246</td>\n",
       "<td>0.0939158</td></tr>\n",
       "<tr><td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0011167</td>\n",
       "<td>0.2497230</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0502793</td>\n",
       "<td>0.0375156</td>\n",
       "<td>0.2013404</td>\n",
       "<td>0.2013410</td>\n",
       "<td>0.0249653</td>\n",
       "<td>1.0</td>\n",
       "<td>-75.0277005</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialGLM: glm\n",
       "** Reported on validation data. **\n",
       "\n",
       "MSE: 0.1178330502946334\n",
       "RMSE: 0.3432681900418875\n",
       "LogLoss: 0.3827854417227382\n",
       "AUC: 0.8173891686110769\n",
       "AUCPR: 0.5982392824442148\n",
       "Gini: 0.6347783372221538\n",
       "Null degrees of freedom: 1195\n",
       "Residual degrees of freedom: 1178\n",
       "Null deviance: 1193.6689319811862\n",
       "Residual deviance: 915.6227766007898\n",
       "AIC: 951.6227766007898</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-6.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-6 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-6 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-6 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-6 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-6 .h2o-table th,\n",
       "#h2o-table-6 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-6 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-6\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3129932533175332</caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>0</th>\n",
       "<th>1</th>\n",
       "<th>Error</th>\n",
       "<th>Rate</th></tr></thead>\n",
       "    <tbody><tr><td>0</td>\n",
       "<td>863.0</td>\n",
       "<td>95.0</td>\n",
       "<td>0.0992</td>\n",
       "<td> (95.0/958.0)</td></tr>\n",
       "<tr><td>1</td>\n",
       "<td>107.0</td>\n",
       "<td>131.0</td>\n",
       "<td>0.4496</td>\n",
       "<td> (107.0/238.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>970.0</td>\n",
       "<td>226.0</td>\n",
       "<td>0.1689</td>\n",
       "<td> (202.0/1196.0)</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-7.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-7 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-7 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-7 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-7 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-7 .h2o-table th,\n",
       "#h2o-table-7 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-7 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-7\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Maximum Metrics: Maximum metrics at their respective thresholds</caption>\n",
       "    <thead><tr><th>metric</th>\n",
       "<th>threshold</th>\n",
       "<th>value</th>\n",
       "<th>idx</th></tr></thead>\n",
       "    <tbody><tr><td>max f1</td>\n",
       "<td>0.3129933</td>\n",
       "<td>0.5646552</td>\n",
       "<td>155.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1877463</td>\n",
       "<td>0.6551476</td>\n",
       "<td>231.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.4296125</td>\n",
       "<td>0.6079404</td>\n",
       "<td>107.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.4438485</td>\n",
       "<td>0.8461538</td>\n",
       "<td>105.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.9939857</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0096581</td>\n",
       "<td>1.0</td>\n",
       "<td>398.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.9939857</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_mcc</td>\n",
       "<td>0.3129933</td>\n",
       "<td>0.4602072</td>\n",
       "<td>155.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.1954052</td>\n",
       "<td>0.7478992</td>\n",
       "<td>225.0</td></tr>\n",
       "<tr><td>max mean_per_class_accuracy</td>\n",
       "<td>0.1914540</td>\n",
       "<td>0.7508026</td>\n",
       "<td>228.0</td></tr>\n",
       "<tr><td>max tns</td>\n",
       "<td>0.9939857</td>\n",
       "<td>958.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fns</td>\n",
       "<td>0.9939857</td>\n",
       "<td>237.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fps</td>\n",
       "<td>0.0074980</td>\n",
       "<td>958.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tps</td>\n",
       "<td>0.0096581</td>\n",
       "<td>238.0</td>\n",
       "<td>398.0</td></tr>\n",
       "<tr><td>max tnr</td>\n",
       "<td>0.9939857</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fnr</td>\n",
       "<td>0.9939857</td>\n",
       "<td>0.9957983</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fpr</td>\n",
       "<td>0.0074980</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tpr</td>\n",
       "<td>0.0096581</td>\n",
       "<td>1.0</td>\n",
       "<td>398.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-8.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-8 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-8 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-8 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-8 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-8 .h2o-table th,\n",
       "#h2o-table-8 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-8 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-8\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Gains/Lift Table: Avg response rate: 19.90 %, avg score: 21.55 %</caption>\n",
       "    <thead><tr><th>group</th>\n",
       "<th>cumulative_data_fraction</th>\n",
       "<th>lower_threshold</th>\n",
       "<th>lift</th>\n",
       "<th>cumulative_lift</th>\n",
       "<th>response_rate</th>\n",
       "<th>score</th>\n",
       "<th>cumulative_response_rate</th>\n",
       "<th>cumulative_score</th>\n",
       "<th>capture_rate</th>\n",
       "<th>cumulative_capture_rate</th>\n",
       "<th>gain</th>\n",
       "<th>cumulative_gain</th>\n",
       "<th>kolmogorov_smirnov</th></tr></thead>\n",
       "    <tbody><tr><td>1</td>\n",
       "<td>0.0100334</td>\n",
       "<td>0.9422827</td>\n",
       "<td>5.0252101</td>\n",
       "<td>5.0252101</td>\n",
       "<td>1.0</td>\n",
       "<td>0.9698159</td>\n",
       "<td>1.0</td>\n",
       "<td>0.9698159</td>\n",
       "<td>0.0504202</td>\n",
       "<td>0.0504202</td>\n",
       "<td>402.5210084</td>\n",
       "<td>402.5210084</td>\n",
       "<td>0.0504202</td></tr>\n",
       "<tr><td>2</td>\n",
       "<td>0.0200669</td>\n",
       "<td>0.8859422</td>\n",
       "<td>4.6064426</td>\n",
       "<td>4.8158263</td>\n",
       "<td>0.9166667</td>\n",
       "<td>0.9133265</td>\n",
       "<td>0.9583333</td>\n",
       "<td>0.9415712</td>\n",
       "<td>0.0462185</td>\n",
       "<td>0.0966387</td>\n",
       "<td>360.6442577</td>\n",
       "<td>381.5826331</td>\n",
       "<td>0.0955948</td></tr>\n",
       "<tr><td>3</td>\n",
       "<td>0.0301003</td>\n",
       "<td>0.8333880</td>\n",
       "<td>3.7689076</td>\n",
       "<td>4.4668534</td>\n",
       "<td>0.75</td>\n",
       "<td>0.8662506</td>\n",
       "<td>0.8888889</td>\n",
       "<td>0.9164643</td>\n",
       "<td>0.0378151</td>\n",
       "<td>0.1344538</td>\n",
       "<td>276.8907563</td>\n",
       "<td>346.6853408</td>\n",
       "<td>0.1302784</td></tr>\n",
       "<tr><td>4</td>\n",
       "<td>0.0401338</td>\n",
       "<td>0.7848582</td>\n",
       "<td>3.7689076</td>\n",
       "<td>4.2923669</td>\n",
       "<td>0.75</td>\n",
       "<td>0.8154020</td>\n",
       "<td>0.8541667</td>\n",
       "<td>0.8911987</td>\n",
       "<td>0.0378151</td>\n",
       "<td>0.1722689</td>\n",
       "<td>276.8907563</td>\n",
       "<td>329.2366947</td>\n",
       "<td>0.1649620</td></tr>\n",
       "<tr><td>5</td>\n",
       "<td>0.0501672</td>\n",
       "<td>0.7211959</td>\n",
       "<td>2.5126050</td>\n",
       "<td>3.9364146</td>\n",
       "<td>0.5</td>\n",
       "<td>0.7505935</td>\n",
       "<td>0.7833333</td>\n",
       "<td>0.8630777</td>\n",
       "<td>0.0252101</td>\n",
       "<td>0.1974790</td>\n",
       "<td>151.2605042</td>\n",
       "<td>293.6414566</td>\n",
       "<td>0.1839091</td></tr>\n",
       "<tr><td>6</td>\n",
       "<td>0.1003344</td>\n",
       "<td>0.4947058</td>\n",
       "<td>3.2663866</td>\n",
       "<td>3.6014006</td>\n",
       "<td>0.65</td>\n",
       "<td>0.5984199</td>\n",
       "<td>0.7166667</td>\n",
       "<td>0.7307488</td>\n",
       "<td>0.1638655</td>\n",
       "<td>0.3613445</td>\n",
       "<td>226.6386555</td>\n",
       "<td>260.1400560</td>\n",
       "<td>0.3258539</td></tr>\n",
       "<tr><td>7</td>\n",
       "<td>0.1505017</td>\n",
       "<td>0.3581829</td>\n",
       "<td>2.0100840</td>\n",
       "<td>3.0709617</td>\n",
       "<td>0.4</td>\n",
       "<td>0.4130805</td>\n",
       "<td>0.6111111</td>\n",
       "<td>0.6248593</td>\n",
       "<td>0.1008403</td>\n",
       "<td>0.4621849</td>\n",
       "<td>101.0084034</td>\n",
       "<td>207.0961718</td>\n",
       "<td>0.3891160</td></tr>\n",
       "<tr><td>8</td>\n",
       "<td>0.2006689</td>\n",
       "<td>0.3002017</td>\n",
       "<td>2.0100840</td>\n",
       "<td>2.8057423</td>\n",
       "<td>0.4</td>\n",
       "<td>0.3274207</td>\n",
       "<td>0.5583333</td>\n",
       "<td>0.5504997</td>\n",
       "<td>0.1008403</td>\n",
       "<td>0.5630252</td>\n",
       "<td>101.0084034</td>\n",
       "<td>180.5742297</td>\n",
       "<td>0.4523780</td></tr>\n",
       "<tr><td>9</td>\n",
       "<td>0.3001672</td>\n",
       "<td>0.2236892</td>\n",
       "<td>1.1824024</td>\n",
       "<td>2.2676435</td>\n",
       "<td>0.2352941</td>\n",
       "<td>0.2577997</td>\n",
       "<td>0.4512535</td>\n",
       "<td>0.4534766</td>\n",
       "<td>0.1176471</td>\n",
       "<td>0.6806723</td>\n",
       "<td>18.2402373</td>\n",
       "<td>126.7643548</td>\n",
       "<td>0.4750355</td></tr>\n",
       "<tr><td>10</td>\n",
       "<td>0.4005017</td>\n",
       "<td>0.1761387</td>\n",
       "<td>1.0469188</td>\n",
       "<td>1.9618252</td>\n",
       "<td>0.2083333</td>\n",
       "<td>0.1963835</td>\n",
       "<td>0.3903967</td>\n",
       "<td>0.3890691</td>\n",
       "<td>0.1050420</td>\n",
       "<td>0.7857143</td>\n",
       "<td>4.6918768</td>\n",
       "<td>96.1825231</td>\n",
       "<td>0.4809126</td></tr>\n",
       "<tr><td>11</td>\n",
       "<td>0.5</td>\n",
       "<td>0.1438673</td>\n",
       "<td>0.5489725</td>\n",
       "<td>1.6806723</td>\n",
       "<td>0.1092437</td>\n",
       "<td>0.1582719</td>\n",
       "<td>0.3344482</td>\n",
       "<td>0.3431412</td>\n",
       "<td>0.0546218</td>\n",
       "<td>0.8403361</td>\n",
       "<td>-45.1027470</td>\n",
       "<td>68.0672269</td>\n",
       "<td>0.4248873</td></tr>\n",
       "<tr><td>12</td>\n",
       "<td>0.6003344</td>\n",
       "<td>0.1214381</td>\n",
       "<td>0.5025210</td>\n",
       "<td>1.4837668</td>\n",
       "<td>0.1</td>\n",
       "<td>0.1325918</td>\n",
       "<td>0.2952646</td>\n",
       "<td>0.3079519</td>\n",
       "<td>0.0504202</td>\n",
       "<td>0.8907563</td>\n",
       "<td>-49.7478992</td>\n",
       "<td>48.3766766</td>\n",
       "<td>0.3625726</td></tr>\n",
       "<tr><td>13</td>\n",
       "<td>0.6998328</td>\n",
       "<td>0.0997817</td>\n",
       "<td>0.5067439</td>\n",
       "<td>1.3448591</td>\n",
       "<td>0.1008403</td>\n",
       "<td>0.1100559</td>\n",
       "<td>0.2676225</td>\n",
       "<td>0.2798162</td>\n",
       "<td>0.0504202</td>\n",
       "<td>0.9411765</td>\n",
       "<td>-49.3256126</td>\n",
       "<td>34.4859091</td>\n",
       "<td>0.3013017</td></tr>\n",
       "<tr><td>14</td>\n",
       "<td>0.8001672</td>\n",
       "<td>0.0798720</td>\n",
       "<td>0.2512605</td>\n",
       "<td>1.2077307</td>\n",
       "<td>0.05</td>\n",
       "<td>0.0891178</td>\n",
       "<td>0.2403344</td>\n",
       "<td>0.2559041</td>\n",
       "<td>0.0252101</td>\n",
       "<td>0.9663866</td>\n",
       "<td>-74.8739496</td>\n",
       "<td>20.7730741</td>\n",
       "<td>0.2075139</td></tr>\n",
       "<tr><td>15</td>\n",
       "<td>0.8996656</td>\n",
       "<td>0.0556184</td>\n",
       "<td>0.2111433</td>\n",
       "<td>1.0975134</td>\n",
       "<td>0.0420168</td>\n",
       "<td>0.0677374</td>\n",
       "<td>0.2184015</td>\n",
       "<td>0.2350939</td>\n",
       "<td>0.0210084</td>\n",
       "<td>0.9873950</td>\n",
       "<td>-78.8856719</td>\n",
       "<td>9.7513355</td>\n",
       "<td>0.1095244</td></tr>\n",
       "<tr><td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0074980</td>\n",
       "<td>0.1256303</td>\n",
       "<td>1.0</td>\n",
       "<td>0.025</td>\n",
       "<td>0.0397575</td>\n",
       "<td>0.1989967</td>\n",
       "<td>0.2154949</td>\n",
       "<td>0.0126050</td>\n",
       "<td>1.0</td>\n",
       "<td>-87.4369748</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-9.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-9 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-9 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-9 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-9 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-9 .h2o-table th,\n",
       "#h2o-table-9 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-9 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-9\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Scoring History: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>timestamp</th>\n",
       "<th>duration</th>\n",
       "<th>iteration</th>\n",
       "<th>lambda</th>\n",
       "<th>predictors</th>\n",
       "<th>deviance_train</th>\n",
       "<th>deviance_test</th>\n",
       "<th>alpha</th>\n",
       "<th>iterations</th>\n",
       "<th>training_rmse</th>\n",
       "<th>training_logloss</th>\n",
       "<th>training_r2</th>\n",
       "<th>training_auc</th>\n",
       "<th>training_pr_auc</th>\n",
       "<th>training_lift</th>\n",
       "<th>training_classification_error</th>\n",
       "<th>validation_rmse</th>\n",
       "<th>validation_logloss</th>\n",
       "<th>validation_r2</th>\n",
       "<th>validation_auc</th>\n",
       "<th>validation_pr_auc</th>\n",
       "<th>validation_lift</th>\n",
       "<th>validation_classification_error</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.000 sec</td>\n",
       "<td>1</td>\n",
       "<td>.25E0</td>\n",
       "<td>1</td>\n",
       "<td>1.0045100</td>\n",
       "<td>0.9980509</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.061 sec</td>\n",
       "<td>3</td>\n",
       "<td>.22E0</td>\n",
       "<td>2</td>\n",
       "<td>0.9946355</td>\n",
       "<td>0.9864127</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.101 sec</td>\n",
       "<td>5</td>\n",
       "<td>.2E0</td>\n",
       "<td>2</td>\n",
       "<td>0.9859035</td>\n",
       "<td>0.9760418</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.130 sec</td>\n",
       "<td>7</td>\n",
       "<td>.19E0</td>\n",
       "<td>2</td>\n",
       "<td>0.9781697</td>\n",
       "<td>0.9667699</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.164 sec</td>\n",
       "<td>9</td>\n",
       "<td>.17E0</td>\n",
       "<td>3</td>\n",
       "<td>0.9686784</td>\n",
       "<td>0.9544753</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.201 sec</td>\n",
       "<td>11</td>\n",
       "<td>.15E0</td>\n",
       "<td>3</td>\n",
       "<td>0.9587676</td>\n",
       "<td>0.9411532</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.220 sec</td>\n",
       "<td>13</td>\n",
       "<td>.14E0</td>\n",
       "<td>3</td>\n",
       "<td>0.9500953</td>\n",
       "<td>0.9293709</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.268 sec</td>\n",
       "<td>15</td>\n",
       "<td>.13E0</td>\n",
       "<td>3</td>\n",
       "<td>0.9424603</td>\n",
       "<td>0.9188844</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.295 sec</td>\n",
       "<td>17</td>\n",
       "<td>.12E0</td>\n",
       "<td>5</td>\n",
       "<td>0.9341431</td>\n",
       "<td>0.9080921</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:37</td>\n",
       "<td> 0.351 sec</td>\n",
       "<td>19</td>\n",
       "<td>.11E0</td>\n",
       "<td>5</td>\n",
       "<td>0.9236449</td>\n",
       "<td>0.8955093</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.462 sec</td>\n",
       "<td>89</td>\n",
       "<td>.13E-2</td>\n",
       "<td>18</td>\n",
       "<td>0.8045920</td>\n",
       "<td>0.7656455</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.467 sec</td>\n",
       "<td>90</td>\n",
       "<td>.12E-2</td>\n",
       "<td>18</td>\n",
       "<td>0.8044582</td>\n",
       "<td>0.7656103</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.472 sec</td>\n",
       "<td>91</td>\n",
       "<td>.11E-2</td>\n",
       "<td>18</td>\n",
       "<td>0.8043436</td>\n",
       "<td>0.7655884</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.485 sec</td>\n",
       "<td>92</td>\n",
       "<td>.1E-2</td>\n",
       "<td>18</td>\n",
       "<td>0.8042462</td>\n",
       "<td>0.7655764</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.492 sec</td>\n",
       "<td>93</td>\n",
       "<td>.92E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8041639</td>\n",
       "<td>0.7655709</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.498 sec</td>\n",
       "<td>94</td>\n",
       "<td>.84E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8040939</td>\n",
       "<td>0.7655721</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.502 sec</td>\n",
       "<td>95</td>\n",
       "<td>.77E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8040347</td>\n",
       "<td>0.7655776</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.506 sec</td>\n",
       "<td>96</td>\n",
       "<td>.7E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8039850</td>\n",
       "<td>0.7655875</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.518 sec</td>\n",
       "<td>97</td>\n",
       "<td>.64E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8039428</td>\n",
       "<td>0.7655990</td>\n",
       "<td>0.5</td>\n",
       "<td>None</td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td>\n",
       "<td></td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-10-05 18:17:38</td>\n",
       "<td> 1.528 sec</td>\n",
       "<td>98</td>\n",
       "<td>.58E-3</td>\n",
       "<td>18</td>\n",
       "<td>0.8039072</td>\n",
       "<td>0.7656134</td>\n",
       "<td>0.5</td>\n",
       "<td>98</td>\n",
       "<td>0.3522849</td>\n",
       "<td>0.4020820</td>\n",
       "<td>0.2282168</td>\n",
       "<td>0.7849415</td>\n",
       "<td>0.5691249</td>\n",
       "<td>4.9667129</td>\n",
       "<td>0.2032952</td>\n",
       "<td>0.3432682</td>\n",
       "<td>0.3827854</td>\n",
       "<td>0.2607574</td>\n",
       "<td>0.8173892</td>\n",
       "<td>0.5982393</td>\n",
       "<td>5.0252101</td>\n",
       "<td>0.1688963</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "<pre style='font-size: smaller; margin-bottom: 1em;'>[66 rows x 24 columns]</pre></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-10.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-10 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-10 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-10 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-10 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-10 .h2o-table th,\n",
       "#h2o-table-10 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-10 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-10\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Variable Importances: </caption>\n",
       "    <thead><tr><th>variable</th>\n",
       "<th>relative_importance</th>\n",
       "<th>scaled_importance</th>\n",
       "<th>percentage</th></tr></thead>\n",
       "    <tbody><tr><td>DELINQ</td>\n",
       "<td>0.7970053</td>\n",
       "<td>1.0</td>\n",
       "<td>0.1473004</td></tr>\n",
       "<tr><td>JOB.Sales</td>\n",
       "<td>0.7478484</td>\n",
       "<td>0.9383230</td>\n",
       "<td>0.1382154</td></tr>\n",
       "<tr><td>JOB.Office</td>\n",
       "<td>0.5109144</td>\n",
       "<td>0.6410426</td>\n",
       "<td>0.0944259</td></tr>\n",
       "<tr><td>JOB.Self</td>\n",
       "<td>0.4571329</td>\n",
       "<td>0.5735632</td>\n",
       "<td>0.0844861</td></tr>\n",
       "<tr><td>CLAGE</td>\n",
       "<td>0.4484834</td>\n",
       "<td>0.5627107</td>\n",
       "<td>0.0828875</td></tr>\n",
       "<tr><td>DEBTINC</td>\n",
       "<td>0.4467476</td>\n",
       "<td>0.5605328</td>\n",
       "<td>0.0825667</td></tr>\n",
       "<tr><td>DEROG</td>\n",
       "<td>0.4263194</td>\n",
       "<td>0.5349016</td>\n",
       "<td>0.0787912</td></tr>\n",
       "<tr><td>NINQ</td>\n",
       "<td>0.3037309</td>\n",
       "<td>0.3810902</td>\n",
       "<td>0.0561347</td></tr>\n",
       "<tr><td>VALUE</td>\n",
       "<td>0.2412600</td>\n",
       "<td>0.3027081</td>\n",
       "<td>0.0445890</td></tr>\n",
       "<tr><td>MORTDUE</td>\n",
       "<td>0.2384487</td>\n",
       "<td>0.2991808</td>\n",
       "<td>0.0440695</td></tr>\n",
       "<tr><td>LOAN</td>\n",
       "<td>0.2180768</td>\n",
       "<td>0.2736203</td>\n",
       "<td>0.0403044</td></tr>\n",
       "<tr><td>REASON.HomeImp</td>\n",
       "<td>0.1797686</td>\n",
       "<td>0.2255551</td>\n",
       "<td>0.0332244</td></tr>\n",
       "<tr><td>CLNO</td>\n",
       "<td>0.1485205</td>\n",
       "<td>0.1863481</td>\n",
       "<td>0.0274492</td></tr>\n",
       "<tr><td>REASON.DebtCon</td>\n",
       "<td>0.1373842</td>\n",
       "<td>0.1723755</td>\n",
       "<td>0.0253910</td></tr>\n",
       "<tr><td>YOJ</td>\n",
       "<td>0.0438345</td>\n",
       "<td>0.0549990</td>\n",
       "<td>0.0081014</td></tr>\n",
       "<tr><td>JOB.Other</td>\n",
       "<td>0.0353093</td>\n",
       "<td>0.0443025</td>\n",
       "<td>0.0065258</td></tr>\n",
       "<tr><td>JOB.Mgr</td>\n",
       "<td>0.0299615</td>\n",
       "<td>0.0375926</td>\n",
       "<td>0.0055374</td></tr>\n",
       "<tr><td>JOB.ProfExe</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div><pre style=\"font-size: smaller; margin: 1em 0 0 0;\">\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.</pre>"
      ],
      "text/plain": [
       "Model Details\n",
       "=============\n",
       "H2OGeneralizedLinearEstimator : Generalized Linear Modeling\n",
       "Model Key: glmfit\n",
       "\n",
       "\n",
       "GLM Model: summary\n",
       "    family    link    regularization                                 lambda_search                                                                 number_of_predictors_total    number_of_active_predictors    number_of_iterations    training_frame\n",
       "--  --------  ------  ---------------------------------------------  ----------------------------------------------------------------------------  ----------------------------  -----------------------------  ----------------------  ----------------\n",
       "    binomial  logit   Elastic Net (alpha = 0.5, lambda = 9.244E-4 )  nlambda = 100, lambda.max = 0.2455, lambda.min = 9.244E-4, lambda.1se = -1.0  18                            17                             93                      py_3_sid_a269\n",
       "\n",
       "ModelMetricsBinomialGLM: glm\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.12410463474559494\n",
       "RMSE: 0.3522848772592927\n",
       "LogLoss: 0.40208197287665776\n",
       "AUC: 0.7849415147958837\n",
       "AUCPR: 0.5691248967099318\n",
       "Gini: 0.5698830295917674\n",
       "Null degrees of freedom: 3580\n",
       "Residual degrees of freedom: 3563\n",
       "Null deviance: 3597.150438148379\n",
       "Residual deviance: 2879.711089742623\n",
       "AIC: 2915.711089742623\n",
       "\n",
       "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.2574171649920362\n",
       "       0     1    Error    Rate\n",
       "-----  ----  ---  -------  --------------\n",
       "0      2446  414  0.1448   (414.0/2860.0)\n",
       "1      314   407  0.4355   (314.0/721.0)\n",
       "Total  2760  821  0.2033   (728.0/3581.0)\n",
       "\n",
       "Maximum Metrics: Maximum metrics at their respective thresholds\n",
       "metric                       threshold    value     idx\n",
       "---------------------------  -----------  --------  -----\n",
       "max f1                       0.257417     0.527886  207\n",
       "max f2                       0.152297     0.626112  279\n",
       "max f0point5                 0.388606     0.557126  144\n",
       "max accuracy                 0.567259     0.835242  90\n",
       "max precision                0.998809     1         0\n",
       "max recall                   0.00123775   1         399\n",
       "max specificity              0.998809     1         0\n",
       "max absolute_mcc             0.266185     0.402098  203\n",
       "max min_per_class_accuracy   0.17828      0.706643  258\n",
       "max mean_per_class_accuracy  0.214315     0.719503  234\n",
       "max tns                      0.998809     2860      0\n",
       "max fns                      0.998809     717       0\n",
       "max fps                      0.00626272   2860      398\n",
       "max tps                      0.00123775   721       399\n",
       "max tnr                      0.998809     1         0\n",
       "max fnr                      0.998809     0.994452  0\n",
       "max fpr                      0.00626272   1         398\n",
       "max tpr                      0.00123775   1         399\n",
       "\n",
       "Gains/Lift Table: Avg response rate: 20.13 %, avg score: 20.13 %\n",
       "group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    score      cumulative_response_rate    cumulative_score    capture_rate    cumulative_capture_rate    gain      cumulative_gain    kolmogorov_smirnov\n",
       "-------  --------------------------  -----------------  --------  -----------------  ---------------  ---------  --------------------------  ------------------  --------------  -------------------------  --------  -----------------  --------------------\n",
       "1        0.0100531                   0.946682           4.96671   4.96671            1                0.980507   1                           0.980507            0.0499307       0.0499307                  396.671   396.671            0.0499307\n",
       "2        0.0201061                   0.85234            4.82875   4.89773            0.972222         0.898794   0.986111                    0.93965             0.0485437       0.0984743                  382.875   389.773            0.0981247\n",
       "3        0.0301592                   0.747956           3.863     4.55282            0.777778         0.798407   0.916667                    0.892569            0.038835        0.137309                   286.3     355.282            0.134162\n",
       "4        0.0402122                   0.691234           3.17318   4.20791            0.638889         0.715245   0.847222                    0.848238            0.0319001       0.169209                   217.318   320.791            0.161517\n",
       "5        0.0502653                   0.62673            3.58707   4.08374            0.722222         0.655894   0.822222                    0.809769            0.036061        0.20527                    258.707   308.374            0.194082\n",
       "6        0.100251                    0.430246           2.44174   3.26503            0.49162          0.519975   0.657382                    0.665276            0.122053        0.327323                   144.174   226.503            0.284316\n",
       "7        0.150237                    0.345918           2.05328   2.86186            0.413408         0.383856   0.576208                    0.571644            0.102635        0.429958                   105.328   186.186            0.350238\n",
       "8        0.200223                    0.284421           1.72031   2.57687            0.346369         0.311903   0.518828                    0.506799            0.0859917       0.51595                    72.0314   157.687            0.395321\n",
       "9        0.300195                    0.209844           1.33186   2.16225            0.268156         0.243115   0.435349                    0.418986            0.133148        0.649098                   33.1856   116.225            0.436861\n",
       "10       0.400168                    0.169573           0.887904  1.84389            0.178771         0.188252   0.371249                    0.361343            0.0887656       0.737864                   -11.2096  84.3888            0.422829\n",
       "11       0.50014                     0.138032           0.679801  1.6112             0.136872         0.152531   0.3244                      0.319604            0.0679612       0.805825                   -32.0199  61.1201            0.382748\n",
       "12       0.600112                    0.114623           0.541066  1.43293            0.108939         0.125884   0.288506                    0.287332            0.0540915       0.859917                   -45.8934  43.2928            0.325301\n",
       "13       0.700084                    0.0945753          0.527193  1.30359            0.106145         0.104779   0.262465                    0.261263            0.0527046       0.912621                   -47.2807  30.3589            0.266118\n",
       "14       0.800056                    0.0736148          0.388458  1.18924            0.0782123        0.0841132  0.239442                    0.239127            0.038835        0.951456                   -61.1542  18.9237            0.189568\n",
       "15       0.900028                    0.0521793          0.235849  1.08334            0.047486         0.0627694  0.21812                     0.219538            0.0235784       0.975035                   -76.4151  8.33382            0.0939158\n",
       "16       1                           0.00111668         0.249723  1                  0.0502793        0.0375156  0.20134                     0.201341            0.0249653       1                          -75.0277  0                  0\n",
       "\n",
       "ModelMetricsBinomialGLM: glm\n",
       "** Reported on validation data. **\n",
       "\n",
       "MSE: 0.1178330502946334\n",
       "RMSE: 0.3432681900418875\n",
       "LogLoss: 0.3827854417227382\n",
       "AUC: 0.8173891686110769\n",
       "AUCPR: 0.5982392824442148\n",
       "Gini: 0.6347783372221538\n",
       "Null degrees of freedom: 1195\n",
       "Residual degrees of freedom: 1178\n",
       "Null deviance: 1193.6689319811862\n",
       "Residual deviance: 915.6227766007898\n",
       "AIC: 951.6227766007898\n",
       "\n",
       "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3129932533175332\n",
       "       0    1    Error    Rate\n",
       "-----  ---  ---  -------  --------------\n",
       "0      863  95   0.0992   (95.0/958.0)\n",
       "1      107  131  0.4496   (107.0/238.0)\n",
       "Total  970  226  0.1689   (202.0/1196.0)\n",
       "\n",
       "Maximum Metrics: Maximum metrics at their respective thresholds\n",
       "metric                       threshold    value     idx\n",
       "---------------------------  -----------  --------  -----\n",
       "max f1                       0.312993     0.564655  155\n",
       "max f2                       0.187746     0.655148  231\n",
       "max f0point5                 0.429613     0.60794   107\n",
       "max accuracy                 0.443849     0.846154  105\n",
       "max precision                0.993986     1         0\n",
       "max recall                   0.00965813   1         398\n",
       "max specificity              0.993986     1         0\n",
       "max absolute_mcc             0.312993     0.460207  155\n",
       "max min_per_class_accuracy   0.195405     0.747899  225\n",
       "max mean_per_class_accuracy  0.191454     0.750803  228\n",
       "max tns                      0.993986     958       0\n",
       "max fns                      0.993986     237       0\n",
       "max fps                      0.00749797   958       399\n",
       "max tps                      0.00965813   238       398\n",
       "max tnr                      0.993986     1         0\n",
       "max fnr                      0.993986     0.995798  0\n",
       "max fpr                      0.00749797   1         399\n",
       "max tpr                      0.00965813   1         398\n",
       "\n",
       "Gains/Lift Table: Avg response rate: 19.90 %, avg score: 21.55 %\n",
       "group    cumulative_data_fraction    lower_threshold    lift      cumulative_lift    response_rate    score      cumulative_response_rate    cumulative_score    capture_rate    cumulative_capture_rate    gain      cumulative_gain    kolmogorov_smirnov\n",
       "-------  --------------------------  -----------------  --------  -----------------  ---------------  ---------  --------------------------  ------------------  --------------  -------------------------  --------  -----------------  --------------------\n",
       "1        0.0100334                   0.942283           5.02521   5.02521            1                0.969816   1                           0.969816            0.0504202       0.0504202                  402.521   402.521            0.0504202\n",
       "2        0.0200669                   0.885942           4.60644   4.81583            0.916667         0.913326   0.958333                    0.941571            0.0462185       0.0966387                  360.644   381.583            0.0955948\n",
       "3        0.0301003                   0.833388           3.76891   4.46685            0.75             0.866251   0.888889                    0.916464            0.0378151       0.134454                   276.891   346.685            0.130278\n",
       "4        0.0401338                   0.784858           3.76891   4.29237            0.75             0.815402   0.854167                    0.891199            0.0378151       0.172269                   276.891   329.237            0.164962\n",
       "5        0.0501672                   0.721196           2.51261   3.93641            0.5              0.750593   0.783333                    0.863078            0.0252101       0.197479                   151.261   293.641            0.183909\n",
       "6        0.100334                    0.494706           3.26639   3.6014             0.65             0.59842    0.716667                    0.730749            0.163866        0.361345                   226.639   260.14             0.325854\n",
       "7        0.150502                    0.358183           2.01008   3.07096            0.4              0.41308    0.611111                    0.624859            0.10084         0.462185                   101.008   207.096            0.389116\n",
       "8        0.200669                    0.300202           2.01008   2.80574            0.4              0.327421   0.558333                    0.5505              0.10084         0.563025                   101.008   180.574            0.452378\n",
       "9        0.300167                    0.223689           1.1824    2.26764            0.235294         0.2578     0.451253                    0.453477            0.117647        0.680672                   18.2402   126.764            0.475036\n",
       "10       0.400502                    0.176139           1.04692   1.96183            0.208333         0.196383   0.390397                    0.389069            0.105042        0.785714                   4.69188   96.1825            0.480913\n",
       "11       0.5                         0.143867           0.548973  1.68067            0.109244         0.158272   0.334448                    0.343141            0.0546218       0.840336                   -45.1027  68.0672            0.424887\n",
       "12       0.600334                    0.121438           0.502521  1.48377            0.1              0.132592   0.295265                    0.307952            0.0504202       0.890756                   -49.7479  48.3767            0.362573\n",
       "13       0.699833                    0.0997817          0.506744  1.34486            0.10084          0.110056   0.267622                    0.279816            0.0504202       0.941176                   -49.3256  34.4859            0.301302\n",
       "14       0.800167                    0.079872           0.251261  1.20773            0.05             0.0891178  0.240334                    0.255904            0.0252101       0.966387                   -74.8739  20.7731            0.207514\n",
       "15       0.899666                    0.0556184          0.211143  1.09751            0.0420168        0.0677374  0.218401                    0.235094            0.0210084       0.987395                   -78.8857  9.75134            0.109524\n",
       "16       1                           0.00749797         0.12563   1                  0.025            0.0397575  0.198997                    0.215495            0.012605        1                          -87.437   0                  0\n",
       "\n",
       "Scoring History: \n",
       "     timestamp            duration    iteration    lambda    predictors    deviance_train      deviance_test       alpha    iterations    training_rmse       training_logloss     training_r2          training_auc        training_pr_auc     training_lift      training_classification_error    validation_rmse     validation_logloss    validation_r2        validation_auc      validation_pr_auc    validation_lift    validation_classification_error\n",
       "---  -------------------  ----------  -----------  --------  ------------  ------------------  ------------------  -------  ------------  ------------------  -------------------  -------------------  ------------------  ------------------  -----------------  -------------------------------  ------------------  --------------------  -------------------  ------------------  -------------------  -----------------  ---------------------------------\n",
       "     2023-10-05 18:17:37  0.000 sec   1            .25E0     1             1.0045100357856405  0.9980509464725638  0.5\n",
       "     2023-10-05 18:17:37  0.061 sec   3            .22E0     2             0.9946354860817245  0.9864126793254803  0.5\n",
       "     2023-10-05 18:17:37  0.101 sec   5            .2E0      2             0.9859035403430406  0.9760418435888577  0.5\n",
       "     2023-10-05 18:17:37  0.130 sec   7            .19E0     2             0.9781697250290142  0.9667699237562071  0.5\n",
       "     2023-10-05 18:17:37  0.164 sec   9            .17E0     3             0.9686784218176835  0.9544752889432686  0.5\n",
       "     2023-10-05 18:17:37  0.201 sec   11           .15E0     3             0.9587676036112146  0.9411531794664825  0.5\n",
       "     2023-10-05 18:17:37  0.220 sec   13           .14E0     3             0.9500952786201636  0.9293708568093554  0.5\n",
       "     2023-10-05 18:17:37  0.268 sec   15           .13E0     3             0.9424603199349196  0.918884378267228   0.5\n",
       "     2023-10-05 18:17:37  0.295 sec   17           .12E0     5             0.934143096516142   0.9080921324097252  0.5\n",
       "     2023-10-05 18:17:37  0.351 sec   19           .11E0     5             0.9236448771247918  0.8955093341627816  0.5\n",
       "---  ---                  ---         ---          ---       ---           ---                 ---                 ---      ---           ---                 ---                  ---                  ---                 ---                 ---                ---                              ---                 ---                   ---                  ---                 ---                  ---                ---\n",
       "     2023-10-05 18:17:38  1.462 sec   89           .13E-2    18            0.8045920176677348  0.7656454925772931  0.5\n",
       "     2023-10-05 18:17:38  1.467 sec   90           .12E-2    18            0.8044581526528738  0.7656102932355412  0.5\n",
       "     2023-10-05 18:17:38  1.472 sec   91           .11E-2    18            0.804343644390629   0.7655884396559107  0.5\n",
       "     2023-10-05 18:17:38  1.485 sec   92           .1E-2     18            0.804246194799186   0.7655763506822736  0.5\n",
       "     2023-10-05 18:17:38  1.492 sec   93           .92E-3    18            0.8041639457533155  0.7655708834454764  0.5\n",
       "     2023-10-05 18:17:38  1.498 sec   94           .84E-3    18            0.8040939470898276  0.7655720881120318  0.5\n",
       "     2023-10-05 18:17:38  1.502 sec   95           .77E-3    18            0.8040347348308917  0.7655776424675973  0.5\n",
       "     2023-10-05 18:17:38  1.506 sec   96           .7E-3     18            0.8039850013216244  0.7655874603666687  0.5\n",
       "     2023-10-05 18:17:38  1.518 sec   97           .64E-3    18            0.8039427775722173  0.7655990044572286  0.5\n",
       "     2023-10-05 18:17:38  1.528 sec   98           .58E-3    18            0.8039071637538451  0.7656134388669195  0.5      98            0.3522848772592927  0.40208197287665776  0.22821675700859534  0.7849415147958837  0.5691248967099318  4.966712898751734  0.20329516894722144              0.3432681900418875  0.3827854417227382    0.26075735307166414  0.8173891686110769  0.5982392824442148   5.025210084033613  0.1688963210702341\n",
       "[66 rows x 24 columns]\n",
       "\n",
       "\n",
       "Variable Importances: \n",
       "variable        relative_importance    scaled_importance    percentage\n",
       "--------------  ---------------------  -------------------  ------------\n",
       "DELINQ          0.797005               1                    0.1473\n",
       "JOB.Sales       0.747848               0.938323             0.138215\n",
       "JOB.Office      0.510914               0.641043             0.0944259\n",
       "JOB.Self        0.457133               0.573563             0.0844861\n",
       "CLAGE           0.448483               0.562711             0.0828875\n",
       "DEBTINC         0.446748               0.560533             0.0825667\n",
       "DEROG           0.426319               0.534902             0.0787912\n",
       "NINQ            0.303731               0.38109              0.0561347\n",
       "VALUE           0.24126                0.302708             0.044589\n",
       "MORTDUE         0.238449               0.299181             0.0440695\n",
       "LOAN            0.218077               0.27362              0.0403044\n",
       "REASON.HomeImp  0.179769               0.225555             0.0332244\n",
       "CLNO            0.14852                0.186348             0.0274492\n",
       "REASON.DebtCon  0.137384               0.172376             0.025391\n",
       "YOJ             0.0438345              0.054999             0.00810138\n",
       "JOB.Other       0.0353093              0.0443025            0.00652578\n",
       "JOB.Mgr         0.0299615              0.0375926            0.0055374\n",
       "JOB.ProfExe     0                      0                    0\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glm = H2OGeneralizedLinearEstimator(family=\"binomial\", model_id=\"glmfit\", lambda_search=True)\n",
    "glm.train(x=x, y=y, training_frame=train, validation_frame=validation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.5513512979207219, 0.8486897717666948]]\n"
     ]
    }
   ],
   "source": [
    "# Check the model performance and print its accuracy\n",
    "glm_performance = glm.model_performance(test)\n",
    "print(glm_performance.accuracy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glm.pre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Register Models in SAS Model Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "model_prefix = \"glmfit\"\n",
    "binary_folder = Path.cwd() / \"data/hmeqModels/H2OBinaryGLM/\" # Changes needed by user\n",
    "mojo_folder = Path.cwd() / \"data/hmeqModels/H2OMOJOGLM/\" # Changes needed by user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "# Save the model as a H2O binary model file\n",
    "pzmm.PickleModel.pickle_trained_model(\n",
    "    model_prefix=model_prefix + \"_binary\",\n",
    "    trained_model=glm, \n",
    "    pickle_path=binary_folder, \n",
    "    is_h2o_model=True, \n",
    "    is_binary_model=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "# Save the model as a H2O MOJO model file\n",
    "pzmm.PickleModel.pickle_trained_model(\n",
    "    model_prefix=model_prefix + \"_mojo\",\n",
    "    trained_model=glm,\n",
    "    pickle_path=mojo_folder, \n",
    "    is_h2o_model=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputVar.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OBinaryGLM/inputVar.json\n",
      "inputVar.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OMOJOGLM/inputVar.json\n",
      "outputVar.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OBinaryGLM/outputVar.json\n",
      "outputVar.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OMOJOGLM/outputVar.json\n",
      "ModelProperties.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OBinaryGLM/ModelProperties.json\n",
      "ModelProperties.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OMOJOGLM/ModelProperties.json\n",
      "fileMetadata.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OBinaryGLM/fileMetadata.json\n",
      "fileMetadata.json was successfully written and saved to /Users/dalmoo/Documents/GitHub/python-sasctl/examples/data/hmeqModels/H2OMOJOGLM/fileMetadata.json\n"
     ]
    }
   ],
   "source": [
    "train_df = train.as_data_frame()\n",
    "# Write input variable mapping to a json file\n",
    "pzmm.JSONFiles.write_var_json(train_df[x], is_input=True, json_path=binary_folder)\n",
    "pzmm.JSONFiles.write_var_json(train_df[x], is_input=True, json_path=mojo_folder)\n",
    "\n",
    "# Set output variables and assign an event threshold, then write output variable mapping\n",
    "output_var = pd.DataFrame(\n",
    "    columns=[\"EM_CLASSIFICATION\", \"EM_EVENTPROBABILITY\"],\n",
    "    data=[[\"A\", 0.5]]\n",
    ")\n",
    "pzmm.JSONFiles.write_var_json(output_var, is_input=False, json_path=binary_folder)\n",
    "pzmm.JSONFiles.write_var_json(output_var, is_input=False, json_path=mojo_folder)\n",
    "\n",
    "# Write model properties to a json file\n",
    "pzmm.JSONFiles.write_model_properties_json(\n",
    "    model_name=model_prefix + \"_binary\",\n",
    "    model_desc=\"Binary H2O model.\",\n",
    "    target_variable=y,\n",
    "    target_values=[\"0\", \"1\"],\n",
    "    json_path=binary_folder,\n",
    "    modeler=\"sasdemo\"\n",
    ")\n",
    "pzmm.JSONFiles.write_model_properties_json(\n",
    "    model_name=model_prefix + \"_mojo\",\n",
    "    model_desc=\"MOJO H2O model.\",\n",
    "    target_variable=y,\n",
    "    target_values=[\"0\", \"1\"],\n",
    "    json_path=mojo_folder,\n",
    "    modeler=\"sasdemo\"\n",
    ")\n",
    "\n",
    "# Write model metadata to a json file\n",
    "pzmm.JSONFiles.write_file_metadata_json(model_prefix=model_prefix + \"_binary\", json_path=binary_folder)\n",
    "pzmm.JSONFiles.write_file_metadata_json(model_prefix=model_prefix + \"_mojo\", json_path=mojo_folder, is_h2o_model=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "import getpass\n",
    "username = getpass.getpass()\n",
    "password = getpass.getpass()\n",
    "host = \"sas.demo.com\"\n",
    "sess = Session(host, username, password, protocol=\"http\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All model files were zipped to ~\\examples\\data\\hmeqModels\\H2OBinaryGLM.\n",
      "Model was successfully imported into SAS Model Manager as glmfit_binary with the following UUID: 5929a748-f9b2-4285-b73a-45c40659b4b0.\n",
      "Model score code was written successfully to ~\\examples\\data\\hmeqModels\\H2OBinaryGLM\\score_glmfit_binary.py and uploaded to SAS Model Manager.\n",
      "All model files were zipped to ~\\examples\\data\\hmeqModels\\H2OMOJOGLM.\n",
      "Model was successfully imported into SAS Model Manager as glmfit_mojo with the following UUID: 4c5dd027-d442-4860-9e96-9c26060dc727.\n",
      "Model score code was written successfully to ~\\examples\\data\\hmeqModels\\H2OMOJOGLM\\score_glmfit_mojo.py and uploaded to SAS Model Manager.\n"
     ]
    }
   ],
   "source": [
    "binary_model = pzmm.ImportModel.import_model(\n",
    "    model_files=binary_folder, \n",
    "    model_prefix=model_prefix + \"_binary\",\n",
    "    project=\"H2OModels\", \n",
    "    input_data=train_df[x], \n",
    "    predict_method=[glm.predict, [list]], \n",
    "    binary_h2o_model=True, \n",
    "    score_metrics=[\"EM_CLASSIFICATION\", \"EM_EVENTPROBABILITY\"],\n",
    "    missing_values=True,\n",
    "    overwrite_model=True,\n",
    "    model_file_name=\"glmfit_binary.pickle\"\n",
    ")\n",
    "pzmm.ScoreCode.score_code = \"\"\n",
    "\n",
    "mojo_model = pzmm.ImportModel.import_model(\n",
    "    model_files=mojo_folder, \n",
    "    model_prefix=model_prefix + \"_mojo\",\n",
    "    project=\"H2OModels\", \n",
    "    input_data=train_df[x], \n",
    "    predict_method=[glm.predict, [list]], \n",
    "    mojo_model=True, \n",
    "    score_metrics=[\"EM_CLASSIFICATION\", \"EM_EVENTPROBABILITY\"],\n",
    "    missing_values=True,\n",
    "    overwrite_model=True,\n",
    "    model_file_name=\"glmfit_mojo.mojo\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Run a Score Test in SAS Model Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-14T17:18:16.283114900Z",
     "start_time": "2023-06-14T17:18:08.890088300Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Publish the model to the SAS Microanalytic Score destination in SAS Model Manager\n",
    "module = publish_model(mojo_model[0], \"maslocal\", name=\"HMEQMOJO_publish\", replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-14T17:18:34.236925200Z",
     "start_time": "2023-06-14T17:18:34.209029800Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<StreamHandler stderr (INFO)>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instantiate a API call logger to visualize score calls in realtime\n",
    "sess.add_stderr_logger(level=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-14T17:18:47.064393600Z",
     "start_time": "2023-06-14T17:18:37.850162200Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 POST http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n",
      "HTTP/1.1 201 http://demo.sas.com/microanalyticScore/modules/hmeqmojo_publish/steps/score\n"
     ]
    }
   ],
   "source": [
    "# Convert h2o dataframe to pandas dataframe\n",
    "X = train[0:10,:].as_data_frame()\n",
    "result = []\n",
    "# Step through the rows of data and collect the score from SAS Microanalytic Score publish destination\n",
    "for index, row in X.iterrows():\n",
    "    result.append(module.score(row))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-14T17:19:21.811834300Z",
     "start_time": "2023-06-14T17:19:21.772752700Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0.0, '0', 0.9470777188580571),\n",
      " (0.0, '0', 0.8697221847625589),\n",
      " (0.0, '0', 0.932352798571864),\n",
      " (0.0, '0', 0.9579361586939033),\n",
      " (0.0, '0', 0.8422587334478235),\n",
      " (0.0, '0', 0.9112367410859918),\n",
      " (0.0, '0', 0.7222737872462567),\n",
      " (0.0, '0', 0.8016194612045261),\n",
      " (0.0, '0', 0.9285316160726795),\n",
      " (0.0, '0', 0.9619797616826327)]\n"
     ]
    }
   ],
   "source": [
    "# Scoring results\n",
    "pp.pprint(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "yeehaw",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}