examples/pzmm_custom_kpi_model_parameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrating Key Performance Indicators and Hyperparameters Into SAS Model Manager\n",
    "SAS Model Manager generates certain Key Performance Indicators (KPIs) automatically based on user created performance definitions. However, we may want to include other KPIs to measure certain things not tracked by SAS Model Manager.\n",
    "\n",
    "This can be done by performing local tests on models we\"ve passed to SAS Model Manager, then passing up the resulting values as custom KPI values.\n",
    "\n",
    "For certain python models, sasctl will also generate a json file containing the hyperparameters of the model, making them easily accessible for future use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python Package Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard Library\n",
    "from pathlib import Path\n",
    "import warnings\n",
    "import getpass\n",
    "\n",
    "# Third Party\n",
    "import pandas as pd\n",
    "import pickle\n",
    "\n",
    "# Application Specific\n",
    "from sasctl import Session\n",
    "from sasctl.pzmm.model_parameters import ModelParameters as mp\n",
    "\n",
    "# Global Package Options\n",
    "pd.options.mode.chained_assignment = None  # default=\"warn\"\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "This example is built to be run after the [PZMM Binary Classification Model Import](./pzmm_binary_classification_model_import.ipynb) example notebook is run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = getpass.getpass()\n",
    "password = getpass.getpass()\n",
    "host = \"demo.sas.com\" # Changes required by user\n",
    "sess = Session(host, username, password, protocol=\"http\") # For TLS-enabled servers, change protocol value to \"https\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating Model and Project Properties\n",
    "In order to allow for performance definitions to be run in SAS Model Manager, certain properties need to be set for both the model and the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sasctl._services.model_repository import ModelRepository as mr\n",
    "\n",
    "model_name = \"DecisionTreeClassifier\"\n",
    "model_path = Path.cwd() / \"data/hmeqModels/DecisionTreeClassifier/\"\n",
    "\n",
    "model = mr.get_model(model_name)\n",
    "\n",
    "model[\"targetEvent\"] = \"1\"\n",
    "model[\"targetVariable\"] = \"BAD\"\n",
    "model[\"function\"] = \"classification\"\n",
    "model[\"targetLevel\"] = \"Binary\"\n",
    "model[\"eventProbVar\"] = \"EM_EVENTPROBABILITY\"\n",
    "\n",
    "model = mr.update_model(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = mr.get_project(\"HMEQModels\")\n",
    "\n",
    "variables = model[\"inputVariables\"] + model[\"outputVariables\"]\n",
    "\n",
    "project[\"targetVariable\"] = \"BAD\"\n",
    "project[\"variables\"] = variables\n",
    "project[\"targetLevel\"] = \"Binary\"\n",
    "project[\"targetEventValue\"] = \"1\"\n",
    "project[\"classTargetValues\"] = \".5\"\n",
    "project[\"function\"] = \"classification\"\n",
    "project[\"eventProbabilityVariable\"] = \"EM_EVENTPROBABILITY\"\n",
    "\n",
    "project = mr.update_project(project)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Generation\n",
    "If the hyperparameter json file is not generated automatically, this code block will generate it and add it to SAS Model Manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(Path(model_path) / \"DecisionTreeClassifier.pickle\", \"rb\") as pickle_model:\n",
    "    tree_model = pickle.load(pickle_model)\n",
    "\n",
    "mp.generate_hyperparameters(tree_model, model_name, model_path)\n",
    "\n",
    "with open(model_path / f\"{model_name}Hyperparameters.json\", \"r\") as f:\n",
    "    mr.add_model_content(model_name, f, f\"{model_name}Hyperparameters.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model has been uploaded to SAS Model Manager, custom hyperparameters can be added to the hyperparameter json file using the add_hyperparamters function. The custom hyperparameters are passed in as kwargs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp.add_hyperparameters(model_name, example=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Definition\n",
    "To create a performance definition, we first have to pass up data for the performance definition to run on. All data used for performance defintions should be named using the following format: \n",
    "\n",
    "{Table Prefix}\\_{Time}\\_{Time Label}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sasctl._services.cas_management import CASManagement as cas\n",
    "\n",
    "for x in range(1, 5):\n",
    "    cas.upload_file(\n",
    "        file=f\"data/HMEQPERF_{x}_Q{x}.csv\", \n",
    "        name=f\"HMEQPERF_{x}_Q{x}\")\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After pushing up the data, the performance definition can be created. When the performance definition is run, the KPIs are generated within Model Manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sasctl._services.model_management import ModelManagement as mm\n",
    "\n",
    "perf_task = mm.create_performance_definition(table_prefix=\"hmeqperf\", project=\"HMEQModels\", scoring_required=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = mr.get_project(\"HMEQModels\")\n",
    "\n",
    "perf_job = mm.execute_performance_definition(perf_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the performance definition is run, it is possible to update the hyperparameter json file to include the KPIs that have been generated. This is not a necessary step, but could be helpful when analyzing which hyperparameters lead to better KPIs. Note that you need the performance jobs to have created the MM_STD performance table before running the `update_kpis()` method below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "check_perf_job = mm.get(f\"/performanceTasks/{perf_job[0].taskId}/performanceJobs\")\n",
    "for j in check_perf_job:\n",
    "    print(f\"{j.model}: {j.state}\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp.update_kpis(\"HMEQModels\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom KPIs\n",
    "It is also possible to generate custom key performance indicators and pass them up to SAS Model Manager. Below, using the same data sets as were used in the SAS performance definition, the recall score is calculated, and then passed up to the KPI table in SAS Model Manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import jaccard_score\n",
    "\n",
    "predictor_columns = [\"LOAN\", \"MORTDUE\", \"VALUE\", \"YOJ\", \"DEROG\", \"DELINQ\", \"CLAGE\", \"NINQ\", \"CLNO\", \"DEBTINC\"]\n",
    "target_column = \"BAD\"\n",
    "\n",
    "jaccard_list = list()\n",
    "time_labels = list()\n",
    "time_sks = list()\n",
    "name = [\"jaccard\" for x in range(4)]\n",
    "\n",
    "for x in range(1, 5):\n",
    "    test_data = pd.read_csv(f\"data/HMEQPERF_{x}_Q{x}.csv\")\n",
    "    x_test = test_data[predictor_columns]\n",
    "    y_test = test_data[target_column]\n",
    "    test_data_predictions = tree_model.predict(x_test)\n",
    "    jaccard = jaccard_score(y_test, test_data_predictions)\n",
    "    jaccard_list.append(jaccard)\n",
    "    time_labels.append(f\"Q{x}\")\n",
    "    time_sks.append(x)\n",
    "\n",
    "print(jaccard_list)\n",
    "\n",
    "model = mr.get_model(model_name)\n",
    "mm.create_custom_kpi(\n",
    "    model=model.id,\n",
    "    project=\"HMEQModels\",\n",
    "    kpiName=name,\n",
    "    kpiValue=jaccard_list,\n",
    "    timeLabel=time_labels,\n",
    "    timeSK=time_sks\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the KPIs have been generated, the hyperparameter file can updated, and the new KPIs will appear in the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "mp.update_kpis(\"HMEQModels\")\n",
    "\n",
    "hyperparameters = mp.get_hyperparameters(model_name)\n",
    "\n",
    "print(json.dumps(hyperparameters, indent=4))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dev-py38",
   "language": "python",
   "name": "dev-py38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}