seed_code/00-ml-build-train/notebooks/01-Prepare-Data-ML-Framework-Container.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Prepare data for ML - Framework Container\n",
    "\n",
    "**SageMaker Studio Kernel**: Data Science\n",
    "\n",
    "In this exercise you will do:\n",
    " - Run a Preprocessing Job using Amazon SageMaker Processing Job for preparing data for training ML models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Part 1/2 - Setup\n",
    "Here we'll import some libraries and define some variables. You can also take a look on the scripts that were previously created for preparing the data and training our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import logging\n",
    "import sagemaker\n",
    "from sagemaker.processing import FrameworkProcessor, ProcessingInput, ProcessingOutput\n",
    "from sagemaker.sklearn.estimator import SKLearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "logging.basicConfig(level=logging.INFO)\n",
    "LOGGER = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "s3_client = boto3.client(\"s3\")\n",
    "\n",
    "region = boto3.session.Session().region_name\n",
    "role_name = \"mlops-sagemaker-execution-role\"\n",
    "role = \"arn:aws:iam::{}:role/{}\".format(boto3.client('sts').get_caller_identity().get('Account'), role_name)\n",
    "\n",
    "kms_account_id = boto3.client('sts').get_caller_identity().get('Account')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Global configurations\n",
    "\n",
    "Configuration variables used for Processing, Training, and registration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "kms_alias = \"ml-kms\"\n",
    "\n",
    "bucket_name = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "boto_session = boto3.Session(region_name=region)\n",
    "\n",
    "sagemaker_client = boto_session.client(\"sagemaker\")\n",
    "runtime_client = boto_session.client(\"sagemaker-runtime\")\n",
    "\n",
    "sagemaker_session = sagemaker.session.Session(\n",
    "    boto_session=boto_session,\n",
    "    sagemaker_client=sagemaker_client,\n",
    "    sagemaker_runtime_client=runtime_client,\n",
    "    default_bucket=bucket_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "kms_key = \"arn:aws:kms:{}:{}:alias/{}\".format(region, kms_account_id, kms_alias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Part 2/2: Run the processing job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Step 1/3: Create the Processing Job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Define input variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processing_artifact_path = \"artifact/processing\"\n",
    "processing_artifact_name = \"sourcedir.tar.gz\"\n",
    "processing_framework_version = \"0.23-1\"\n",
    "processing_instance_count = 1\n",
    "processing_instance_type = \"ml.t3.large\"\n",
    "processing_input_files_path = \"data/input\"\n",
    "processing_output_files_path = \"data/output\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Get the dataset and upload it to an S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Download the \n",
    "# clean the buckets first\n",
    "s3_client.delete_object(Bucket=bucket_name, Key=processing_input_files_path)\n",
    "\n",
    "input_data = sagemaker_session.upload_data('./../data/data.csv', key_prefix=processing_input_files_path)\n",
    "\n",
    "LOGGER.info(input_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Create Processor\n",
    "\n",
    "#### Compress source code for installing additional python modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! pygmentize ./../algorithms/processing/src/processing.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! ./../algorithms/buildspec.sh processing $bucket_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processor = FrameworkProcessor(\n",
    "    estimator_cls=SKLearn,\n",
    "    framework_version=processing_framework_version,\n",
    "    role=role,\n",
    "    instance_count=processing_instance_count,\n",
    "    instance_type=processing_instance_type,\n",
    "    output_kms_key=kms_key,\n",
    "    sagemaker_session=sagemaker_session\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "run_args = processor.get_run_args(\n",
    "        \"processing.py\",\n",
    "        source_dir=\"s3://{}/{}/{}\".format(bucket_name,\n",
    "                                      processing_artifact_path,\n",
    "                                      processing_artifact_name\n",
    "                                      ),\n",
    "        inputs=[\n",
    "            ProcessingInput(\n",
    "                input_name=\"input\",\n",
    "                source=\"s3://{}/{}\".format(bucket_name, processing_input_files_path),\n",
    "                destination=\"/opt/ml/processing/input\"\n",
    "            )\n",
    "        ],\n",
    "        outputs=[\n",
    "            ProcessingOutput(\n",
    "                output_name=\"output\",\n",
    "                source=\"/opt/ml/processing/output\",\n",
    "                destination=\"s3://{}/{}\".format(bucket_name, processing_output_files_path))\n",
    "        ]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processor.run(\n",
    "    code=run_args.code,\n",
    "    inputs=run_args.inputs,\n",
    "    outputs=run_args.outputs,\n",
    "    wait=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We have just seen how to prepare data using Amazon SageMaker Processing Job. If you want to provide a Custom Python script for training a ML model using a SageMaker Framework Container, you can execute the following lab.\n",
    "\n",
    " > [Train-Build-Model-Framework-Container](./05-Train-Build-Model-Framework-Container.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If you want to create a Custom Framework Container and provide a Custom Python script for training a ML model, you can execute the following lab (Optional).\n",
    "\n",
    " > [Train-Custom-Script-Container](./06-Train-Build-Model-Custom-Script-Container.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to create a Custom Container for training a ML model, you can execute the following lab (Optional).\n",
    "\n",
    " > [Train-Custom-Container](./07-Train-Build-Model-Custom-Container.ipynb)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "If you want to explore Amazon SageMaker Feature Store, you can execute the following lab (Optional).\n",
    "\n",
    " > [Store-Features](./04-Store-Features.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to create a Custom Framework Container and provide a Custom Python script as input for the processing Job, you can execute the following lab (Optional).\n",
    "\n",
    " > [Prepare-Data-ML-Custom-Script-Container](./02-Prepare-Data-ML-Custom-Script-Container.ipynb)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you want to create a Custom Container for the processing Job, you can execute the following lab (Optional).\n",
    "\n",
    " > [Prepare-Data-ML-Custom-Container](./03-Prepare-Data-ML-Custom-Container.ipynb)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "interpreter": {
   "hash": "7830b2e0dcc405ab83456d8c26dd7c2db32ddf1a7b2e64ef505b215ebac66515"
  },
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3 (ipykernel)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}