advanced_functionality/search/ml_experiment_management_using_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing ML Experimentation using Amazon SageMaker Search\n",
    "_**Organize, track and evaluate model training runs**_\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "2. [Prerequisites and Preprocessing](#Prequisites-and-Preprocessing)\n",
    "  1. [Permissions and environment variables](#Permissions-and-environment-variables)\n",
    "  2. [Data ingestion](#Data-ingestion)\n",
    "  3. [Data inspection](#Data-inspection)\n",
    "  4. [Data conversion](#Data-conversion)\n",
    "3. [Training the linear model](#Training-the-linear-model)\n",
    "  1. [Conduct first model training experiment](#Conduct-first-model-training-experiment)\n",
    "  2. [Conduct second model training experiment](#Conduct-second-model-training-experiment)\n",
    "  3. [Conduct third model training experiment](#Conduct-third-model-training-experiment)\n",
    "4. [Use Amazon SageMaker Search to organize and evaluate experiments](#Use-Amazon-SageMaker-Search-to-organize-and-evaluate-experiments)\n",
    "    1. [Visualize the leaderboard](#Visualize-the-leaderboard)\n",
    "5. [Set up hosting for the model](#Set-up-hosting-for-the-model)\n",
    "6. [Validate the model for use](#Validate-the-model-for-use)\n",
    "7. [Tracing the lineage of a model starting from an endpoint](#Tracing-the-lineage-of-a-model-starting-from-an-endpoint)\n",
    "    1. [Visualize the training job details](#Visualize-the-training-job-details)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Welcome to our example introducing Amazon SageMaker Search!  Amazon SageMaker Search lets you quickly find and evaluate the most relevant model training runs from potentially hundreds and thousands of your Amazon SageMaker model training jobs.",
    "\n",
    "Developing a machine learning model requires continuous experimentation, trying new learning algorithms and tuning hyper parameters, all the while observing the impact of such changes on model performance and accuracy. This iterative exercise often leads to explosion of hundreds of model training experiments and model versions, slowing down the convergence and discovery of “winning” model. In addition, the information explosion makes it very hard down the line to trace back the lineage of a model version i.e. the unique combination of datasets, algorithms and parameters that brewed that model in the first place. \n",
    "\n",
    "We will show you how to use Amazon SageMaker Search to quickly and easily organize, track and evaluate your model training jobs on Amazon SageMaker. You can search on all the defining attributes from the learning algorithm used, hyper parameter settings, training datasets used to even the tags you have added on the model training jobs. You can also quickly compare and rank your training runs based on their performance metrics, such as training loss and validation accuracy, thus creating leaderboards for identifying “winning” models that can be deployed into production environments. \n",
    "\n",
    "Finally we will show you an example using Amazon SageMaker search for quickly tracing back the complete lineage of a model version deployed in a live environment, right up until the data sets used in training and validating the model.\n",
    "\n",
    "The model that we will train today uses the [Amazon SageMaker Linear Learner Algorithm](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner.html). We're analyzing the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset which consists of images of handwritten digits, from zero to nine.  We'll use the individual pixel values from each 28 x 28 grayscale image to predict a yes or no label of whether the digit is a 0 or some other digit (1, 2, 3, ... 9).\n",
    "\n",
    "The method that we'll use is a linear binary classifier. Linear models are supervised learning algorithms used for solving either classification or regression problems.  As input, the model is given labeled examples ( **`x`**, `y`). **`x`** is a high dimensional vector and `y` is a numeric label.  Since we are doing binary classification, the algorithm expects the label to be either 0 or 1.  The algorithm learns a linear threshold function for classification, mapping the vector **`x`** to an approximation of the label `y`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prequisites and Preprocessing\n",
    "\n",
    "### Permissions and environment variables\n",
    "_This notebook was created and tested on an ml.t2.medium notebook instance._\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.  \n",
    "- A tag that acts as a **unique label** to track all the model training runs as we experiment with different hyperparameter values. Tags give you a flexible way to assign and associate your model training experiments with a specific business project, a research lab or a data science team, thus letting you meaningfully categorize and catalog your model training experiments\n",
    "- The name of the endpoint where we will finally deploy the \"winning\" model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "bucket = '<name of your S3 bucket>'\n",
    "prefix = 'sagemaker/DEMO-linear-mnist'\n",
    "\n",
    "#replace with a tag key and value of your choice \n",
    "tagKey = 'Project'\n",
    "tagValue = 'Project_Binary_Classifier'\n",
    "\n",
    "#name for endpoint where the winning model will be depoloyed\n",
    "endpointName = '<name for your endpoint>'\n",
    "\n",
    "# Define IAM role\n",
    "import boto3\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data ingestion\n",
    "Next, we read the dataset from an online URL into memory, for preprocessing prior to training. For small datasets, such as this one, reading into memory isn't onerous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import pickle, gzip, numpy, urllib.request, json\n",
    "\n",
    "# Load the dataset\n",
    "urllib.request.urlretrieve(\"http://deeplearning.net/data/mnist/mnist.pkl.gz\", \"mnist.pkl.gz\")\n",
    "with gzip.open('mnist.pkl.gz', 'rb') as f:\n",
    "    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data inspection\n",
    "Once the dataset is imported, it's typical as part of the machine learning process to inspect the data. As an example, let's go ahead and look at one of the digits that is part of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.figsize\"] = (2,10)\n",
    "\n",
    "def show_digit(img, caption='', subplot=None):\n",
    "    if subplot==None:\n",
    "        _,(subplot)=plt.subplots(1,1)\n",
    "    imgr=img.reshape((28,28))\n",
    "    subplot.axis('off')\n",
    "    subplot.imshow(imgr, cmap='gray')\n",
    "    plt.title(caption)\n",
    "\n",
    "show_digit(train_set[0][40], 'This is a {}'.format(train_set[1][40]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data conversion\n",
    "The Amazon SageMaker implementation of Linear Learner takes recordIO-wrapped protobuf, where the data we have today is a pickle-ized numpy array on disk. So we will perform data conversion using the Amazon SageMaker Python SDK, imported as `sagemaker` below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import numpy as np\n",
    "import sagemaker.amazon.common as smac\n",
    "\n",
    "vectors = np.array([t.tolist() for t in train_set[0]]).astype('float32')\n",
    "labels = np.where(np.array([t.tolist() for t in train_set[1]]) == 0, 1, 0).astype('float32')\n",
    "\n",
    "buf = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(buf, vectors, labels)\n",
    "buf.seek(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload training data\n",
    "Now that we've created our recordIO-wrapped protobuf, we'll need to upload it to S3, so that Amazon SageMaker training can use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import os\n",
    "\n",
    "key = 'recordio-pb-data'\n",
    "boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', key)).upload_fileobj(buf)\n",
    "s3_train_data = 's3://{}/{}/train/{}'.format(bucket, prefix, key)\n",
    "print('uploaded training data location: {}'.format(s3_train_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also setup an output S3 location for the model artifact that will be output as the result of training with the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_location_1 = 's3://{}/{}/output-1'.format(bucket, prefix)\n",
    "print('training artifacts will be uploaded to: {}'.format(output_location_1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the linear model\n",
    "Once we have the data preprocessed and available in the correct format for training, the next step is to actually train the model using the data. First, let's specify our algorithm container. More details on algorithm containers can be found in [AWS documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(boto3.Session().region_name, 'linear-learner')\n",
    "print(\"model will be trained using the container: \" + str(container))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conduct first model training experiment\n",
    "We'll use the Amazon SageMaker Python SDK to kick off training, and monitor status until it is completed.  In this example that takes between 7 and 11 minutes. Make sure to pass in the necessary hyperparameters:\n",
    "- `feature_dim` is set to 784, which is the number of pixels in each 28 x 28 image.\n",
    "- `predictor_type` is set to 'binary_classifier' since we are trying to predict whether the image is or is not a 0.\n",
    "- `mini_batch_size` is set to 100. Selecting a reasonable value relative to the dataset is appropriate in most cases.\n",
    "\n",
    "In our first model training expriment we are setting the mini_batch_size as 100. We will experiment further with values of 200 and 300 as well and then use Amazon SageMaker Search to organize and evaluate the three model training runs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "smclient = boto3.client(service_name='sagemaker')\n",
    "sess = sagemaker.session.Session(sagemaker_client=smclient)\n",
    "\n",
    "linear_1 = sagemaker.estimator.Estimator(container,\n",
    "                                        role, \n",
    "                                        train_instance_count=1, \n",
    "                                        train_instance_type='ml.c4.xlarge',\n",
    "                                        output_path=output_location_1,\n",
    "                                        tags=[{\"Key\":tagKey, \"Value\":tagValue}],\n",
    "                                        sagemaker_session=sess)\n",
    "\n",
    "linear_1.set_hyperparameters(feature_dim=784,\n",
    "                           predictor_type='binary_classifier',\n",
    "                           mini_batch_size=100)\n",
    "\n",
    "linear_1.fit({'train': s3_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conduct second model training experiment\n",
    "Now we'll try the mini_batch_size of 200 and train another version of our binary classifier model. Also note that we are using the same tag to label all our experiments so that we can group them later for comparison and analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#S3 location for second model version\n",
    "output_location_2 = 's3://{}/{}/output-2'.format(bucket, prefix)\n",
    "\n",
    "linear_2 = sagemaker.estimator.Estimator(container,\n",
    "                                       role, \n",
    "                                       train_instance_count=1, \n",
    "                                       train_instance_type='ml.c4.xlarge',\n",
    "                                       output_path=output_location_2,\n",
    "                                       tags=[{\"Key\":tagKey, \"Value\":tagValue}],\n",
    "                                       sagemaker_session=sess)\n",
    "\n",
    "linear_2.set_hyperparameters(feature_dim=784,\n",
    "                           predictor_type='binary_classifier',\n",
    "                           mini_batch_size=200)\n",
    "\n",
    "linear_2.fit({'train': s3_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conduct third model training experiment\n",
    "Now we'll try the mini_batch_size of 300 and train another version of our binary classifier model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#S3 location for third model version\n",
    "output_location_3 = 's3://{}/{}/output-3'.format(bucket, prefix)\n",
    "\n",
    "linear_3 = sagemaker.estimator.Estimator(container,\n",
    "                                       role, \n",
    "                                       train_instance_count=1, \n",
    "                                       train_instance_type='ml.c4.xlarge',\n",
    "                                       output_path=output_location_3,\n",
    "                                       tags=[{\"Key\":tagKey, \"Value\":tagValue}],\n",
    "                                       sagemaker_session=sess)\n",
    "\n",
    "linear_3.set_hyperparameters(feature_dim=784,\n",
    "                           predictor_type='binary_classifier',\n",
    "                           mini_batch_size=300)\n",
    "\n",
    "linear_3.fit({'train': s3_train_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Amazon SageMaker Search to organize and evaluate experiments\n",
    "Usually you will experiment with tuning multiple hyperparameters or even try new learning algorithms and training datasets resulting in potentially hundreds of model training runs and model versions. However, for the sake of simplicity, we are only tuning mini_batch_size in this example, trying only three different values resulting in as many model versions. Now we will use [Search](https://docs.aws.amazon.com/sagemaker/latest/dg/search.html) to **group together** the three model training runs and **evaluate** the best performing model by ranking and comparing them on a metric of our choice. \n",
    "\n",
    "**For grouping** the relevant model training runs together, we will search the model training jobs by the unique label or tag that we have been using as a tracking label to track our experiments.  \n",
    "\n",
    "**For model evaluation**, Amazon SageMaker Linear Learner Algorithm emits variety of [metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner-tuning.html) that are automatically published to Amazon CloudWatch for monitoring and visualization. You can easily graph timeseries curves such as training curves and validation curves. If you are training your own algorithm, you can [specify the metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/training-metrics.html) that you want Amazon SageMaker to monitor and publish to Amazon CloudWatch. \n",
    "\n",
    "For the sake of this example, we will compare the three binary classification model versions models we created before on the final value of objective_loss metric evaluated on the MNIST training dataset. We will rank model training runs by their objective_loss in ascending order, thus putting the best performing model on the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import pandas\n",
    "import numpy as np\n",
    "\n",
    "search_params={\n",
    "   \"MaxResults\": 10,\n",
    "   \"Resource\": \"TrainingJob\",\n",
    "   \"SearchExpression\": { \n",
    "      \"Filters\": [ \n",
    "         { \n",
    "            \"Name\": \"Tags.\" + str(tagKey),\n",
    "            \"Operator\": \"Equals\",\n",
    "            \"Value\": tagValue\n",
    "         },\n",
    "         {\n",
    "            \"Name\": \"TrainingJobStatus\",\n",
    "            \"Operator\": \"Equals\",\n",
    "            \"Value\": \"Completed\"\n",
    "         }\n",
    "      ]\n",
    "   },\n",
    "  \"SortBy\": \"Metrics.train:objective_loss:final\",\n",
    "  \"SortOrder\": \"Ascending\"\n",
    "}\n",
    "\n",
    "smclient = boto3.client(service_name='sagemaker')\n",
    "results = smclient.search(**search_params)\n",
    "print(\"The search query returned \" + str(len(results['Results'])) + \" training jobs.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the leaderboard\n",
    "For ease of comparison, we will print a **leaderboard** of model training runs that displays the best performing model on the top. We will also print a scatter plot as an example of alternative way for visualizing the impact of changing the mini_batch_size on model performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Now we will print the leaderboard and scatter plot\n",
    "rows=[]\n",
    "coord=[]\n",
    "for result in results['Results']: \n",
    "    trainingJob = result['TrainingJob']\n",
    "    metrics = trainingJob['FinalMetricDataList']\n",
    "    accuracy = metrics[[x['MetricName'] for x in metrics].index('train:objective_loss:final')]['Value']\n",
    "    rows.append([trainingJob['TrainingJobName'],\n",
    "                 trainingJob['TrainingJobStatus'],\n",
    "                 trainingJob['HyperParameters']['mini_batch_size'],\n",
    "                 accuracy])\n",
    "    coord.append([float(trainingJob['HyperParameters']['mini_batch_size']), accuracy])\n",
    "\n",
    "headers=[\"Training Job Name\", \"Training Job Status\", \"Mini Batch Size\", \"Objective Loss\"]\n",
    "df = pandas.DataFrame(data=rows,columns=headers)\n",
    "from IPython.display import HTML\n",
    "display(HTML(df.to_html()))\n",
    "\n",
    "axes=['mini_batch_size', 'train:objective_loss:final'] \n",
    "df = pandas.DataFrame(data=coord, columns=axes)\n",
    "ax1 = df.plot.scatter(x='mini_batch_size',\n",
    "                      y='train:objective_loss:final',\n",
    "                      c='DarkBlue')\n",
    "ax1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up hosting for the model\n",
    "Now that we've found our best performing model (in this example the one with mini_batch_size=100), we can deploy it behind an Amazon SageMaker real-time hosted endpoint.  This will allow out to make predictions (or inference) from the model dyanamically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_predictor = linear_1.deploy(initial_instance_count=1,\n",
    "                                 instance_type='ml.m4.xlarge', endpoint_name=endpointName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate the model for use\n",
    "Finally, we can now validate the model for use using the Amazon SageMaker Python SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer, json_deserializer\n",
    "\n",
    "linear_predictor.content_type = 'text/csv'\n",
    "linear_predictor.serializer = csv_serializer\n",
    "linear_predictor.deserializer = json_deserializer\n",
    "\n",
    "result = linear_predictor.predict(train_set[0][30:31])\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracing the lineage of a model starting from an endpoint\n",
    "Now we will present an example of how you can use the Amazon SageMaker Search to trace the antecedents of a model deployed at an endpoint i.e. unique combination of algorithms, datasets, and parameters that brewed the model in first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#first get the endpoint config for the relevant endpoint\n",
    "endpoint_config = smclient.describe_endpoint_config(EndpointConfigName=endpointName)\n",
    "\n",
    "#now get the model name for the model deployed at the endpoint. \n",
    "#in this case we know that there is only one model version (aka Production Variant) deployed at the endpoint\n",
    "model_name = endpoint_config['ProductionVariants'][0]['ModelName']\n",
    "\n",
    "#now look up the S3 URI of the model artifacts\n",
    "model = smclient.describe_model(ModelName=model_name)\n",
    "modelURI = model['PrimaryContainer']['ModelDataUrl']\n",
    "print(str(modelURI))\n",
    "\n",
    "#search for the training job that created the model artifacts at above S3 URI location\n",
    "search_params={\n",
    "   \"MaxResults\": 1,\n",
    "   \"Resource\": \"TrainingJob\",\n",
    "   \"SearchExpression\": { \n",
    "      \"Filters\": [ \n",
    "         { \n",
    "            \"Name\": \"ModelArtifacts.S3ModelArtifacts\",\n",
    "            \"Operator\": \"Equals\",\n",
    "            \"Value\": modelURI\n",
    "         }\n",
    "      ]\n",
    "   },\n",
    "}\n",
    "\n",
    "results = smclient.search(**search_params)\n",
    "print(\"The search query returned \" + str(len(results['Results'])) + \" training jobs.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the training job details\n",
    "For ease of visualization, we will now print all the relevant details of the model training job such as the learning algorithm used, chosen hyperparameter values and training dataset used in creating the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now print all the relevant model training details such as algorithm used, training dataset URI, hyper parameter settings \n",
    "#and performance metrics recorded during the training run \n",
    "trainingJob = results['Results'][0]['TrainingJob']\n",
    "metrics = trainingJob['FinalMetricDataList']\n",
    "metricnames = [x['MetricName'] for x in metrics]\n",
    "metricvalues = [x['Value'] for x in metrics]\n",
    "hyperparams = trainingJob['HyperParameters']\n",
    "\n",
    "headers=[\"Training Job Name\", \"Training Datasource URI\", \"Training Algorithm\"] + list(hyperparams.keys()) + metricnames\n",
    "rows=[]\n",
    "rows.append([trainingJob['TrainingJobName'],\n",
    "             trainingJob['InputDataConfig'][0]['DataSource']['S3DataSource']['S3Uri'],\n",
    "             trainingJob['AlgorithmSpecification']['TrainingImage']] + list(hyperparams.values()) + metricvalues\n",
    "            )\n",
    "          \n",
    "df = pandas.DataFrame(data=rows,columns=headers)\n",
    "\n",
    "# Set CSS properties for th elements in dataframe\n",
    "th_props = [\n",
    "  ('font-size', '11px'),\n",
    "  ('text-align', 'center'),\n",
    "  ('font-weight', 'bold'),\n",
    "  ('color', '#6d6d6d'),\n",
    "  ('background-color', '#f7f7f9')\n",
    "  ]\n",
    "\n",
    "# Set CSS properties for td elements in dataframe\n",
    "td_props = [\n",
    "  ('font-size', '11px'),\n",
    "  ('text-align', 'center')\n",
    "  ]\n",
    "\n",
    "# Set table styles\n",
    "styles = [\n",
    "  dict(selector=\"th\", props=th_props),\n",
    "  dict(selector=\"td\", props=td_props)\n",
    "  ]\n",
    "html = (df.style.set_table_styles(styles))\n",
    "from IPython.display import display, HTML\n",
    "html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional) Delete the Endpoint\n",
    "\n",
    "If you're ready to be done with this notebook, please run the delete_endpoint line in the cell below.  This will remove the hosted endpoint you created and avoid any charges from a stray instance being left on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sagemaker.Session().delete_endpoint(linear_predictor.endpoint)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}