examples/pzmm_regression_model_import.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "Copyright © 2023, SAS Institute Inc., Cary, NC, USA.  All Rights Reserved.\n",
    "SPDX-License-Identifier: Apache-2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# USA Housing Dataset : Build and Import Trained Regression Model into SAS Model Manager\n",
    "\n",
    "This notebook provides an example of how to build and train a simple Python model and then import the model into SAS Model Manager (on either SAS Viya 3.5 or SAS Viya 4) using the USA Housing data set. Lines of code that must be modified by the user, such as directory paths or the host server are noted with the comment \"_Changes required by user._\"\n",
    "\n",
    "_**Note:** If you download only this notebook and not the rest of the repository, you must also download the hmeq.csv file from the data folder in the examples directory. These files are used when executing this notebook example._\n",
    "\n",
    "Here are the steps shown in this notebook:\n",
    "\n",
    "1. Import, review, and preprocess data for model training.\n",
    "2. Build, train, and assess a scikit-learn linear regression model.\n",
    "3. Serialize the model into a pickle file.\n",
    "4. Write the metadata JSON files needed for importing into SAS Model Manager as well as optional files for fit statistics and ROC/Lift charts.\n",
    "4. Write a score code Python file for model scoring.\n",
    "5. Zip the pickle, JSON, and score code files into an archive file.\n",
    "6. Import the ZIP archive file to SAS Model Manager via the Session object and relevant function call."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Python Package Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [],
   "source": [
    "# Standard Library\n",
    "from pathlib import Path\n",
    "import warnings\n",
    "\n",
    "# Third Party\n",
    "import matplotlib.pyplot as plt \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# Application specific\n",
    "import sasctl.pzmm as pzmm\n",
    "from sasctl import Session\n",
    "from sasctl.services import model_repository as modelRepo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Global Package Options\n",
    "pd.options.mode.chained_assignment = None  # default=\"warn\"\n",
    "plt.rc(\"font\", size=14)\n",
    "# Ignore warnings from pandas about SWAT using a feature that will be depreciated soon\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Import and Review Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5000, 7)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_data = pd.read_csv(\"data/USA_Housing.csv\",sep= \",\")\n",
    "housing_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Avg_Area_Income</th>\n",
       "      <th>Avg_Area_House_Age</th>\n",
       "      <th>Avg_Area_Number_of_Rooms</th>\n",
       "      <th>Avg_Area_Number_of_Bedrooms</th>\n",
       "      <th>Area_Population</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>79545.45857</td>\n",
       "      <td>5.682861</td>\n",
       "      <td>7.009188</td>\n",
       "      <td>4.09</td>\n",
       "      <td>23086.80050</td>\n",
       "      <td>1.059034e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>79248.64245</td>\n",
       "      <td>6.002900</td>\n",
       "      <td>6.730821</td>\n",
       "      <td>3.09</td>\n",
       "      <td>40173.07217</td>\n",
       "      <td>1.505891e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>61287.06718</td>\n",
       "      <td>5.865890</td>\n",
       "      <td>8.512727</td>\n",
       "      <td>5.13</td>\n",
       "      <td>36882.15940</td>\n",
       "      <td>1.058988e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>63345.24005</td>\n",
       "      <td>7.188236</td>\n",
       "      <td>5.586729</td>\n",
       "      <td>3.26</td>\n",
       "      <td>34310.24283</td>\n",
       "      <td>1.260617e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59982.19723</td>\n",
       "      <td>5.040555</td>\n",
       "      <td>7.839388</td>\n",
       "      <td>4.23</td>\n",
       "      <td>26354.10947</td>\n",
       "      <td>6.309435e+05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Avg_Area_Income  Avg_Area_House_Age  Avg_Area_Number_of_Rooms  \\\n",
       "0      79545.45857            5.682861                  7.009188   \n",
       "1      79248.64245            6.002900                  6.730821   \n",
       "2      61287.06718            5.865890                  8.512727   \n",
       "3      63345.24005            7.188236                  5.586729   \n",
       "4      59982.19723            5.040555                  7.839388   \n",
       "\n",
       "   Avg_Area_Number_of_Bedrooms  Area_Population         Price  \n",
       "0                         4.09      23086.80050  1.059034e+06  \n",
       "1                         3.09      40173.07217  1.505891e+06  \n",
       "2                         5.13      36882.15940  1.058988e+06  \n",
       "3                         3.26      34310.24283  1.260617e+06  \n",
       "4                         4.23      26354.10947  6.309435e+05  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_data = housing_data.drop([\"Address\"], axis=1)\n",
    "housing_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "Collapsed": "false",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Avg_Area_Income', 'Avg_Area_House_Age', 'Avg_Area_Number_of_Rooms',\n",
       "       'Avg_Area_Number_of_Bedrooms', 'Area_Population', 'Price'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Preprocess Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1500, 5)\n",
      "(3500, 5)\n"
     ]
    }
   ],
   "source": [
    "# Input \n",
    "predictor_columns = [\"Avg_Area_Income\", \"Avg_Area_House_Age\", \"Avg_Area_Number_of_Rooms\", \n",
    "                    \"Avg_Area_Number_of_Bedrooms\", \"Area_Population\"]\n",
    "\n",
    "# Target\n",
    "target_column = \"Price\"\n",
    "x = housing_data[predictor_columns]\n",
    "y = housing_data[target_column]\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=42)\n",
    "\n",
    "# For missing values, impute the data set's mean value\n",
    "x_test.fillna(x_test.mean(), inplace=True)\n",
    "x_train.fillna(x_train.mean(), inplace=True)\n",
    "print(x_test.shape)\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Create, Train, and Assess Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(normalize=True)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Linear Regression Training\n",
    "lrm = LinearRegression(normalize=True)\n",
    "lrm.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear Regression Model Accuracy = 91.47%\n"
     ]
    }
   ],
   "source": [
    "# Test Predictions\n",
    "from sklearn import metrics\n",
    "lrm_predict = lrm.predict(x_test)\n",
    "print(\"Linear Regression Model Accuracy = \" + str(np.round(metrics.r2_score(y_test, lrm_predict)*100,2)) + \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "Collapsed": "false"
   },
   "source": [
    "### Zip file for registering into SAS Model Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model LinearRegression was successfully pickled and saved to ~\\data\\USAHousingModels\\LinearRegression\\LinearRegression.pickle.\n"
     ]
    }
   ],
   "source": [
    "# Model name within SAS Model Manager\n",
    "model_prefix = \"LinearRegression\"\n",
    "# Directory location for the model files\n",
    "zip_folder = Path.cwd() / \"data/USAHousingModels/LinearRegression\" # Changes required by user\n",
    "# Output variables expected in SAS Model Manager\n",
    "score_metrics = [\"EM_PREDICTION\"]\n",
    "\n",
    "pzmm.PickleModel.pickle_trained_model(\n",
    "    model_prefix=model_prefix,\n",
    "    trained_model=lrm,\n",
    "    pickle_path=zip_folder\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "Collapsed": "false"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputVar.json was successfully written and saved to ~\\data\\USAHousingModels\\LinearRegression\\inputVar.json\n",
      "outputVar.json was successfully written and saved to ~\\data\\USAHousingModels\\LinearRegression\\outputVar.json\n",
      "ModelProperties.json was successfully written and saved to ~\\data\\USAHousingModels\\LinearRegression\\ModelProperties.json\n",
      "fileMetadata.json was successfully written and saved to ~\\data\\USAHousingModels\\LinearRegression\\fileMetadata.json\n"
     ]
    }
   ],
   "source": [
    "def write_json_files(data, predict, target, path, prefix):    \n",
    "    # Write input variable mapping to a json file\n",
    "    pzmm.JSONFiles.write_var_json(input_data=data[predict], is_input=True, json_path=path)\n",
    "    \n",
    "    # Set output variables and assign an event threshold, then write output variable mapping\n",
    "    output_var = pd.DataFrame(columns=score_metrics, data=[[0.5]]) # data argument includes example expected types for outputs\n",
    "    pzmm.JSONFiles.write_var_json(output_var, is_input=False, json_path=path)\n",
    "        \n",
    "    # Write model properties to a json file\n",
    "    pzmm.JSONFiles.write_model_properties_json(\n",
    "        model_name=prefix, \n",
    "        target_variable=target, # Target variable to make predictions about \n",
    "        json_path=path, \n",
    "        model_desc=f\"Description for the {prefix} model.\",\n",
    "        model_algorithm=\"\",\n",
    "        modeler=\"sasdemo\",\n",
    "    )\n",
    "    \n",
    "    # Write model metadata to a json file so that SAS Model Manager can properly identify all model files\n",
    "    pzmm.JSONFiles.write_file_metadata_json(model_prefix=prefix, json_path=path)\n",
    "\n",
    "write_json_files(housing_data, predictor_columns, target_column, zip_folder, model_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "username = getpass.getpass()\n",
    "password = getpass.getpass()\n",
    "host = \"demo.sas.com\" # Changes required by user\n",
    "sess = Session(host, username, password, protocol=\"http\") # For TLS-enabled servers, change protocol value to \"https\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model score code was written successfully to ~\\data\\USAHousingModels\\LinearRegression\\score_LinearRegression.py and uploaded to SAS Model Manager.\n",
      "All model files were zipped to ~\\data\\USAHousingModels\\LinearRegression.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "~\\sasctl\\pzmm\\import_model.py:45: UserWarning: No project with the name or UUID RegressionModelExample was found.\n",
      "  warn(f\"No project with the name or UUID {project} was found.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A new project named RegressionModelExample was created.\n",
      "Model was successfully imported into SAS Model Manager as LinearRegression with the following UUID: b9936acb-6668-4b41-906f-171954f74d30.\n"
     ]
    }
   ],
   "source": [
    "model_response = pzmm.ImportModel.import_model(\n",
    "    model_files=zip_folder, # Where are the model files?\n",
    "    model_prefix=model_prefix, # What is the model name?\n",
    "    project=\"RegressionModelExample\", # What is the project name?\n",
    "    input_data=x, # What does example input data look like?\n",
    "    predict_method=[lrm.predict, [float]], # What is the predict method and what does it return?\n",
    "    score_metrics=score_metrics, # What are the output variables?\n",
    "    overwrite_model=True, # Overwrite the model if it already exists?\n",
    "    model_file_name=model_prefix + \".pickle\", # How was the model file serialized?\n",
    "    missing_values=True # Does the data include missing values?\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f9708d3f38eeab835578f0695c8890716ee809285281a28db6e379a5abca1310"
  },
  "kernelspec": {
   "display_name": "dev-py38",
   "language": "python",
   "name": "dev-py38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}