Lab1_Feature_Eng_and_Store.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature transformation with Amazon SageMaker processing job and Feature Store\n",
    "\n",
    "### Introduction\n",
    "\n",
    "In this lab you will start with the raw [Women's Clothing Reviews](https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews) dataset and prepare it to train a BERT-based natural language processing (NLP) model. The model will be used to classify customer reviews into positive (1), neutral (0) and negative (-1) sentiment.\n",
    "\n",
    "You will convert the original review text into machine-readable features used by BERT. To perform the required feature transformation you will configure an Amazon SageMaker processing job, which will be running a custom Python script.\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "- [1. Configure the SageMaker Feature Store](#c2w1-1.)\n",
    "  - [1.1. Configure dataset](#c2w1-1.1.)\n",
    "  - [1.2. Configure the SageMaker feature store](#c2w1-1.2.)\n",
    "    - [Exercise 1](#c2w1-ex-1)\n",
    "- [2. Transform the dataset](#c2w1-2.)\n",
    "    - [Exercise 2](#c2w1-ex-2)\n",
    "    - [Exercise 3](#c2w1-ex-3)\n",
    "- [3. Query the Feature Store](#c2w1-3.)\n",
    "  - [3.1. Export training, validation, and test datasets from the Feature Store](#c2w1-3.1.)\n",
    "    - [Exercise 4](#c2w1-ex-4)\n",
    "  - [3.2. Export TSV from Feature Store](#c2w1-3.2.)\n",
    "  - [3.3. Check that the dataset in the Feature Store is balanced by sentiment](#c2w1-3.3.)\n",
    "    - [Exercise 5](#c2w1-ex-5)\n",
    "    - [Exercise 6](#c2w1-ex-6)\n",
    "    - [Exercise 7](#c2w1-ex-7)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "sparkmagic 0.20.4 requires nest-asyncio==1.5.5, but you have nest-asyncio 1.5.7 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mCollecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.\n",
      "Collecting package metadata (repodata.json): ...working... done\n",
      "Solving environment: ...working... \n",
      "The environment is inconsistent, please check the package plan carefully\n",
      "The following packages are causing the inconsistency:\n",
      "\n",
      "  - defaults/linux-64::anaconda-client==1.7.2=py37_0\n",
      "  - defaults/noarch::anaconda-project==0.8.4=py_0\n",
      "  - defaults/linux-64::bokeh==1.4.0=py37_0\n",
      "  - defaults/noarch::dask==2.11.0=py_0\n",
      "  - defaults/linux-64::distributed==2.11.0=py37_0\n",
      "  - defaults/linux-64::spyder==4.0.1=py37_0\n",
      "  - defaults/linux-64::watchdog==0.10.2=py37_0\n",
      "done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /opt/conda\n",
      "\n",
      "  added / updated specs:\n",
      "    - pytorch==1.6.0\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    ca-certificates-2023.08.22 |       h06a4308_0         123 KB\n",
      "    cudatoolkit-10.2.89        |       hfd86e86_1       365.1 MB\n",
      "    ninja-1.10.2               |       h06a4308_5           8 KB\n",
      "    ninja-base-1.10.2          |       hd09550d_5         109 KB\n",
      "    openssl-1.1.1w             |       h7f8727e_0         3.7 MB\n",
      "    pytorch-1.6.0              |py3.7_cuda10.2.89_cudnn7.6.5_0       537.7 MB  pytorch\n",
      "    pyyaml-5.3.1               |   py37h7b6447c_0         181 KB\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:       906.9 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  cudatoolkit        pkgs/main/linux-64::cudatoolkit-10.2.89-hfd86e86_1 None\n",
      "  ninja              pkgs/main/linux-64::ninja-1.10.2-h06a4308_5 None\n",
      "  ninja-base         pkgs/main/linux-64::ninja-base-1.10.2-hd09550d_5 None\n",
      "  pytorch            pytorch/linux-64::pytorch-1.6.0-py3.7_cuda10.2.89_cudnn7.6.5_0 None\n",
      "  pyyaml             pkgs/main/linux-64::pyyaml-5.3.1-py37h7b6447c_0 None\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "  ca-certificates    conda-forge::ca-certificates-2023.7.2~ --> pkgs/main::ca-certificates-2023.08.22-h06a4308_0 None\n",
      "  openssl                                 1.1.1v-h7f8727e_0 --> 1.1.1w-h7f8727e_0 None\n",
      "\n",
      "\n",
      "Preparing transaction: ...working... done\n",
      "Verifying transaction: ...working... done\n",
      "Executing transaction: ...working... done\n",
      "Retrieving notices: ...working... done\n",
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# please ignore warning messages during the installation\n",
    "!pip install --disable-pip-version-check -q sagemaker==2.35.0\n",
    "!conda install -q -y pytorch==1.6.0 -c pytorch\n",
    "!pip install --disable-pip-version-check -q transformers==3.5.1\n",
    "!pip install -q protobuf==3.20.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/boto3/compat.py:82: PythonDeprecationWarning: Boto3 will no longer support Python 3.7 starting December 13, 2023. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.8 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/\n",
      "  warnings.warn(warning, PythonDeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import botocore\n",
    "\n",
    "config = botocore.config.Config(user_agent_extra='dlai-pds/c2/w1')\n",
    "\n",
    "# low-level service client of the boto3 session\n",
    "sm = boto3.client(service_name='sagemaker', \n",
    "                  config=config)\n",
    "\n",
    "featurestore_runtime = boto3.client(service_name='sagemaker-featurestore-runtime', \n",
    "                                    config=config)\n",
    "\n",
    "sess = sagemaker.Session(sagemaker_client=sm,\n",
    "                         sagemaker_featurestore_runtime_client=featurestore_runtime)\n",
    "\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = sess.boto_region_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-1.'></a>\n",
    "# 1. Configure the SageMaker Feature Store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-1.1.'></a>\n",
    "### 1.1. Configure dataset\n",
    "The raw dataset is in the public S3 bucket. Let's start by specifying the S3 location of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://dlai-practical-data-science/data/raw/\n"
     ]
    }
   ],
   "source": [
    "raw_input_data_s3_uri = 's3://dlai-practical-data-science/data/raw/'\n",
    "print(raw_input_data_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List the files in the S3 bucket (in this case it will be just one file):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-30 02:21:06    8457214 womens_clothing_ecommerce_reviews.csv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $raw_input_data_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-1.2.'></a>\n",
    "### 1.2. Configure the SageMaker feature store\n",
    "\n",
    "As the result of the transformation, in addition to generating files in S3 bucket, you will also save the transformed data in the **Amazon SageMaker Feature Store** to be used by others in your organization, for example. \n",
    "\n",
    "To configure a Feature Store you need to setup a **Feature Group**. This is the main resource containing all of the metadata related to the data stored in the Feature Store. A Feature Group should contain a list of **Feature Definitions**. A Feature Definition consists of a name and the data type. The Feature Group also contains an online store configuration and an offline store configuration controlling where the data is stored. Enabling the online store allows quick access to the latest value for a record via the [GetRecord API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_feature_store_GetRecord.html). The offline store allows storage of the data in your S3 bucket. You will be using the offline store in this lab.\n",
    "\n",
    "Let's setup the Feature Group name and the Feature Store offline prefix in S3 bucket (you will use those later in the lab):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature group name: reviews-feature-group-1694591880\n",
      "Feature store offline prefix in S3: reviews-feature-store-1694591880\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "timestamp = int(time.time())\n",
    "\n",
    "feature_group_name = 'reviews-feature-group-' + str(timestamp)\n",
    "feature_store_offline_prefix = 'reviews-feature-store-' + str(timestamp)\n",
    "\n",
    "print('Feature group name: {}'.format(feature_group_name))\n",
    "print('Feature store offline prefix in S3: {}'.format(feature_store_offline_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking two features from the original raw dataset (`Review Text` and `Rating`), you will transform it preparing to be used for the model training and then to be saved in the Feature Store. Here you will define the related features to be stored as a list of `FeatureDefinition`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.feature_store.feature_definition import (\n",
    "    FeatureDefinition,\n",
    "    FeatureTypeEnum,\n",
    ")\n",
    "\n",
    "feature_definitions= [\n",
    "    # unique ID of the review\n",
    "    FeatureDefinition(feature_name='review_id', feature_type=FeatureTypeEnum.STRING), \n",
    "    # ingestion timestamp\n",
    "    FeatureDefinition(feature_name='date', feature_type=FeatureTypeEnum.STRING),\n",
    "    # sentiment: -1 (negative), 0 (neutral) or 1 (positive). It will be found the Rating values (1, 2, 3, 4, 5)\n",
    "    FeatureDefinition(feature_name='sentiment', feature_type=FeatureTypeEnum.STRING), \n",
    "    # label ID of the target class (sentiment)\n",
    "    FeatureDefinition(feature_name='label_id', feature_type=FeatureTypeEnum.STRING),\n",
    "    # reviews encoded with the BERT tokenizer\n",
    "    FeatureDefinition(feature_name='input_ids', feature_type=FeatureTypeEnum.STRING),\n",
    "    # original Review Text\n",
    "    FeatureDefinition(feature_name='review_body', feature_type=FeatureTypeEnum.STRING),\n",
    "    # train/validation/test label\n",
    "    FeatureDefinition(feature_name='split_type', feature_type=FeatureTypeEnum.STRING)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-1'></a>\n",
    "### Exercise 1\n",
    "\n",
    "Create the feature group using the feature definitions defined above.\n",
    "\n",
    "**Instructions:** Use the `FeatureGroup` function passing the defined above feature group name and the feature definitions.\n",
    "\n",
    "```python\n",
    "feature_group = FeatureGroup(\n",
    "    name=..., # Feature Group name\n",
    "    feature_definitions=..., # a list of Feature Definitions\n",
    "    sagemaker_session=sess # SageMaker session\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FeatureGroup(name='reviews-feature-group-1694591880', sagemaker_session=<sagemaker.session.Session object at 0x7f1a780fb0d0>, feature_definitions=[FeatureDefinition(feature_name='review_id', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='date', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='sentiment', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='label_id', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='input_ids', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='review_body', feature_type=<FeatureTypeEnum.STRING: 'String'>), FeatureDefinition(feature_name='split_type', feature_type=<FeatureTypeEnum.STRING: 'String'>)])\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.feature_store.feature_group import FeatureGroup\n",
    "\n",
    "feature_group = FeatureGroup(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    name=feature_group_name, # Replace None\n",
    "    feature_definitions=feature_definitions, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    sagemaker_session=sess\n",
    ")\n",
    "\n",
    "print(feature_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will use the defined Feature Group later in this lab, the actual creation of the Feature Group will take place in the processing job. Now let's move into the setup of the processing job to transform the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-2.'></a>\n",
    "# 2. Transform the dataset\n",
    "\n",
    "You will configure a SageMaker processing job to run a custom Python script to balance and transform the raw data into a format used by BERT model.\n",
    "\n",
    "Set the transformation parameters including the instance type, instance count, and train/validation/test split percentages. For the purposes of this lab, you will use a relatively small instance type. Please refer to [this](https://aws.amazon.com/sagemaker/pricing/) link for additional instance types that may work for your use case outside of this lab.\n",
    "\n",
    "You can also choose whether you want to balance the dataset or not. In this case, you will balance the dataset to avoid class imbalance in the target variable, `sentiment`. \n",
    "\n",
    "Another important parameter of the model is the `max_seq_length`, which specifies the maximum length of the classified reviews for the RoBERTa model. If the sentence is shorter than the maximum length parameter, it will be padded. In another case, when the sentence is longer, it will be truncated from the right side.\n",
    "\n",
    "Since a smaller `max_seq_length` leads to faster training and lower resource utilization, you want to find the smallest power-of-2 that captures `100%` of our reviews.  For this dataset, the `100th` percentile is `115`.  However, it's best to stick with powers-of-2 when using BERT. So let's choose `128` as this is the smallest power-of-2 greater than `115`. You will see below how the shorter sentences will be padded to a maximum length.\n",
    "\n",
    "\n",
    "```\n",
    "mean        52.512374\n",
    "std         31.387048\n",
    "min          1.000000\n",
    "10%         10.000000\n",
    "20%         22.000000\n",
    "30%         32.000000\n",
    "40%         41.000000\n",
    "50%         51.000000\n",
    "60%         61.000000\n",
    "70%         73.000000\n",
    "80%         88.000000\n",
    "90%         97.000000\n",
    "100%       115.000000\n",
    "max        115.000000\n",
    "```\n",
    "\n",
    "![](images/distribution_num_words_per_review.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "processing_instance_type='ml.c5.xlarge'\n",
    "processing_instance_count=1\n",
    "train_split_percentage=0.90\n",
    "validation_split_percentage=0.05\n",
    "test_split_percentage=0.05\n",
    "balance_dataset=True\n",
    "max_seq_length=128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To balance and transform our data, you will use a scikit-learn-based processing job. This is essentially a generic Python processing job with scikit-learn pre-installed. You can specify the version of scikit-learn you wish to use. Also pass the SageMaker execution role, processing instance type and instance count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "processor = SKLearnProcessor(\n",
    "    framework_version='0.23-1',\n",
    "    role=role,\n",
    "    instance_type=processing_instance_type,\n",
    "    instance_count=processing_instance_count,\n",
    "    env={'AWS_DEFAULT_REGION': region},                             \n",
    "    max_runtime_in_seconds=7200\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The processing job will be running the Python code from the file `src/prepare_data.py`. In the following exercise you will review the contents of the file and familiarize yourself with main parts of it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-2'></a>\n",
    "### Exercise 2\n",
    "\n",
    "1. Open the file [src/prepare_data.py](src/prepare_data.py). Go through the comments to understand its content.\n",
    "2. Find and review the `convert_to_bert_input_ids()` function, which contains the RoBERTa `tokenizer` configuration.\n",
    "3. Complete method `encode_plus` of the RoBERTa `tokenizer`. Pass the `max_seq_length` as a value for the argument `max_length`. It defines a pad to a maximum length specified.\n",
    "4. Save the file [src/prepare_data.py](src/prepare_data.py) (with the menu command File -> Save Python File)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _This cell will take approximately 1-2 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "662b3c66eb9542f884b2fbcf21aac0f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=898823.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37e51574d214455f92cea53f56744b5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=456318.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "##################\n",
      "Updated correctly!\n",
      "##################\n"
     ]
    }
   ],
   "source": [
    "import sys, importlib\n",
    "sys.path.append('src/')\n",
    "\n",
    "# import the `prepare_data.py` module\n",
    "import prepare_data\n",
    "\n",
    "# reload the module if it has been previously loaded \n",
    "if 'prepare_data' in sys.modules:\n",
    "    importlib.reload(prepare_data)\n",
    "\n",
    "input_ids = prepare_data.convert_to_bert_input_ids(\"this product is great!\", max_seq_length)\n",
    "    \n",
    "updated_correctly = False\n",
    "\n",
    "if len(input_ids) != max_seq_length:\n",
    "    print('#######################################################################################################')\n",
    "    print('Please check that the function \\'convert_to_bert_input_ids\\' in the file src/prepare_data.py is complete.')\n",
    "    print('#######################################################################################################')\n",
    "    raise Exception('Please check that the function \\'convert_to_bert_input_ids\\' in the file src/prepare_data.py is complete.')\n",
    "else:\n",
    "    print('##################')\n",
    "    print('Updated correctly!')\n",
    "    print('##################')\n",
    "\n",
    "    updated_correctly = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the results of tokenization for the given example (*\\\"this product is great!\\\"*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 9226, 1152, 16, 372, 328, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
      "Length of the sequence: 128\n"
     ]
    }
   ],
   "source": [
    "input_ids = prepare_data.convert_to_bert_input_ids(\"this product is great!\", max_seq_length)\n",
    "\n",
    "print(input_ids)\n",
    "print('Length of the sequence: {}'.format(len(input_ids)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch the processing job with the custom script passing defined above parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Job Name:  sagemaker-scikit-learn-2023-09-13-07-59-28-704\n",
      "Inputs:  [{'InputName': 'input-1', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://dlai-practical-data-science/data/raw/', 'LocalPath': '/opt/ml/processing/input/data/', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'ShardedByS3Key', 'S3CompressionType': 'None'}}, {'InputName': 'code', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/input/code/prepare_data.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n",
      "Outputs:  [{'OutputName': 'sentiment-train', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-train', 'LocalPath': '/opt/ml/processing/output/sentiment/train', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'sentiment-validation', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-validation', 'LocalPath': '/opt/ml/processing/output/sentiment/validation', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'sentiment-test', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-test', 'LocalPath': '/opt/ml/processing/output/sentiment/test', 'S3UploadMode': 'EndOfJob'}}]\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "if (updated_correctly):\n",
    "\n",
    "    processor.run(code='src/prepare_data.py',\n",
    "              inputs=[\n",
    "                    ProcessingInput(source=raw_input_data_s3_uri,\n",
    "                                    destination='/opt/ml/processing/input/data/',\n",
    "                                    s3_data_distribution_type='ShardedByS3Key')\n",
    "              ],\n",
    "              outputs=[\n",
    "                    ProcessingOutput(output_name='sentiment-train',\n",
    "                                     source='/opt/ml/processing/output/sentiment/train',\n",
    "                                     s3_upload_mode='EndOfJob'),\n",
    "                    ProcessingOutput(output_name='sentiment-validation',\n",
    "                                     source='/opt/ml/processing/output/sentiment/validation',\n",
    "                                     s3_upload_mode='EndOfJob'),\n",
    "                    ProcessingOutput(output_name='sentiment-test',\n",
    "                                     source='/opt/ml/processing/output/sentiment/test',\n",
    "                                     s3_upload_mode='EndOfJob')\n",
    "              ],\n",
    "              arguments=['--train-split-percentage', str(train_split_percentage),\n",
    "                         '--validation-split-percentage', str(validation_split_percentage),\n",
    "                         '--test-split-percentage', str(test_split_percentage),\n",
    "                         '--balance-dataset', str(balance_dataset),\n",
    "                         '--max-seq-length', str(max_seq_length),                         \n",
    "                         '--feature-store-offline-prefix', str(feature_store_offline_prefix),\n",
    "                         '--feature-group-name', str(feature_group_name)                         \n",
    "              ],\n",
    "              logs=True,\n",
    "              wait=False)\n",
    "\n",
    "else:\n",
    "    print('#######################################')\n",
    "    print('Please update the code correctly above.')\n",
    "    print('#######################################')    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the information about the processing jobs using the `describe` function. The result is in dictionary format. Let's pull the processing job name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing job name: sagemaker-scikit-learn-2023-09-13-07-59-28-704\n"
     ]
    }
   ],
   "source": [
    "scikit_processing_job_name = processor.jobs[-1].describe()['ProcessingJobName']\n",
    "\n",
    "print('Processing job name: {}'.format(scikit_processing_job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-3'></a>\n",
    "### Exercise 3\n",
    "\n",
    "Pull the processing job status from the processing job description.\n",
    "\n",
    "**Instructions**: Print the keys of the processing job description dictionary, choose the one related to the status of the processing job and print the value of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['ProcessingInputs', 'ProcessingOutputConfig', 'ProcessingJobName', 'ProcessingResources', 'StoppingCondition', 'AppSpecification', 'Environment', 'RoleArn', 'ProcessingJobArn', 'ProcessingJobStatus', 'LastModifiedTime', 'CreationTime', 'ResponseMetadata'])\n"
     ]
    }
   ],
   "source": [
    "print(processor.jobs[-1].describe().keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing job status: InProgress\n"
     ]
    }
   ],
   "source": [
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "scikit_processing_job_status = processor.jobs[-1].describe()['ProcessingJobStatus'] # Replace None\n",
    "### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "print('Processing job status: {}'.format(scikit_processing_job_status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the created processing job in the AWS console.\n",
    "\n",
    "**Instructions**: \n",
    "- open the link\n",
    "- notice that you are in the section `Amazon SageMaker` -> `Processing jobs`\n",
    "- check the name of the processing job, its status and other available information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/processing-jobs/sagemaker-scikit-learn-2023-09-13-07-59-28-704\">processing job</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/processing-jobs/{}\">processing job</a></b>'.format(region, scikit_processing_job_name)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for about 5 minutes to review the CloudWatch Logs. You may open the file [src/prepare_data.py](src/prepare_data.py) again and examine the outputs of the code in the CloudWatch logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=/aws/sagemaker/ProcessingJobs;prefix=sagemaker-scikit-learn-2023-09-13-07-59-28-704;streamFilter=typeLogStreamPrefix\">CloudWatch logs</a> after about 5 minutes</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/cloudwatch/home?region={}#logStream:group=/aws/sagemaker/ProcessingJobs;prefix={};streamFilter=typeLogStreamPrefix\">CloudWatch logs</a> after about 5 minutes</b>'.format(region, scikit_processing_job_name)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the completion of the processing job you can also review the output in the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/?region=us-east-1&tab=overview\">S3 output data</a> after the processing job has completed</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/{}/?region={}&tab=overview\">S3 output data</a> after the processing job has completed</b>'.format(bucket, scikit_processing_job_name, region)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for the processing job to complete.\n",
    "\n",
    "### _This cell will take approximately 15 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".................................................................................................................................................!CPU times: user 727 ms, sys: 53.9 ms, total: 781 ms\n",
      "Wall time: 12min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "running_processor = sagemaker.processing.ProcessingJob.from_processing_name(\n",
    "    processing_job_name=scikit_processing_job_name,\n",
    "    sagemaker_session=sess\n",
    ")\n",
    "\n",
    "running_processor.wait(logs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Please wait until ^^ Processing Job ^^ completes above_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the transformed and balanced data in the S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-train\n",
      "s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-validation\n",
      "s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-test\n"
     ]
    }
   ],
   "source": [
    "processing_job_description = running_processor.describe()\n",
    "\n",
    "output_config = processing_job_description['ProcessingOutputConfig']\n",
    "for output in output_config['Outputs']:\n",
    "    if output['OutputName'] == 'sentiment-train':\n",
    "        processed_train_data_s3_uri = output['S3Output']['S3Uri']\n",
    "    if output['OutputName'] == 'sentiment-validation':\n",
    "        processed_validation_data_s3_uri = output['S3Output']['S3Uri']\n",
    "    if output['OutputName'] == 'sentiment-test':\n",
    "        processed_test_data_s3_uri = output['S3Output']['S3Uri']\n",
    "        \n",
    "print(processed_train_data_s3_uri)\n",
    "print(processed_validation_data_s3_uri)\n",
    "print(processed_test_data_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-13 08:12:25    4891635 part-algo-1-womens_clothing_ecommerce_reviews.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $processed_train_data_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-13 08:12:26     274902 part-algo-1-womens_clothing_ecommerce_reviews.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $processed_validation_data_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-13 08:12:26     268629 part-algo-1-womens_clothing_ecommerce_reviews.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $processed_test_data_s3_uri/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the data into the folder `balanced`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-train/part-algo-1-womens_clothing_ecommerce_reviews.tsv to balanced/sentiment-train/part-algo-1-womens_clothing_ecommerce_reviews.tsv\n",
      "download: s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-validation/part-algo-1-womens_clothing_ecommerce_reviews.tsv to balanced/sentiment-validation/part-algo-1-womens_clothing_ecommerce_reviews.tsv\n",
      "download: s3://sagemaker-us-east-1-662464974816/sagemaker-scikit-learn-2023-09-13-07-59-28-704/output/sentiment-test/part-algo-1-womens_clothing_ecommerce_reviews.tsv to balanced/sentiment-test/part-algo-1-womens_clothing_ecommerce_reviews.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp $processed_train_data_s3_uri/part-algo-1-womens_clothing_ecommerce_reviews.tsv ./balanced/sentiment-train/\n",
    "!aws s3 cp $processed_validation_data_s3_uri/part-algo-1-womens_clothing_ecommerce_reviews.tsv ./balanced/sentiment-validation/\n",
    "!aws s3 cp $processed_test_data_s3_uri/part-algo-1-womens_clothing_ecommerce_reviews.tsv ./balanced/sentiment-test/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the training, validation and test data outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "review_id\tsentiment\tlabel_id\tinput_ids\treview_body\tdate\n",
      "5075\t-1\t0\t[0, 14693, 7, 5, 6929, 1437, 42, 3588, 64, 28, 865, 15158, 50, 3841, 17317, 4, 939, 865, 15158, 8, 516, 16380, 228, 5, 9223, 15, 5, 6929, 1437, 8, 70, 5, 39336, 33, 2198, 28704, 8, 4366, 4, 24, 1326, 101, 89, 18, 10, 2451, 20951, 878, 149, 358, 27595, 4, 98, 5779, 11, 5, 1318, 8, 5, 27963, 4, 11522, 328, 939, 851, 24, 65, 1823, 999, 142, 24, 18, 11962, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tAccording to the label  this dress can be hand washed or dry cleaned. i hand washed and line dried per the instructions on the label  and all the seams have completely shrunk and gathered. it looks like there's a drawstring running through every seam. so disappointed in the quality and the labeling. awful! i gave it one extra star because it's cute.\t2023-09-13T08:06:39Z\n",
      "7517\t-1\t0\t[0, 100, 657, 11956, 12, 14307, 1810, 2985, 9304, 19, 103, 12189, 8, 2451, 6755, 1437, 98, 8366, 939, 21, 2422, 2283, 7, 192, 209, 4, 939, 300, 106, 11, 12339, 8, 51, 1299, 8, 1415, 1969, 235, 66, 9, 5, 3298, 1437, 1468, 21, 3793, 8, 3041, 219, 53, 45, 350, 7174, 4, 9574, 5, 39328, 16, 2198, 160, 3867, 939, 300, 10, 31559, 1763, 1437, 939, 300, 127, 4505, 1836, 784, 8, 51, 1299, 101, 579, 50, 2735, 1437, 939, 6254, 115, 120, 106, 15, 8, 939, 115, 45, 23595, 62, 4, 939, 1017, 240, 23, 513, 80, 10070, 2514, 13, 106, 7, 2564, 23, 70, 4, 182, 5779, 1437, 40, 784, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tI love relaxed-fit wide leg pants with some pockets and draw string  so naturally i was super excited to see these. i got them in gray and they felt and looked perfect right out of the bag  material was soft and flowy but not too thin. unfortunately the sizing is completely off unless i got a defective pair  i got my usual size l and they felt like s or smaller  i barely could get them on and i could not zip up. i'd need at least two sizes larger for them to fit at all. very disappointed  will l\t2023-09-13T08:06:39Z\n",
      "19266\t-1\t0\t[0, 100, 770, 7, 657, 42, 13855, 6439, 4, 24, 1326, 15652, 11, 5, 1152, 1345, 4, 939, 524, 195, 108, 245, 113, 59, 5962, 29882, 8, 4634, 21, 8642, 15, 162, 4, 939, 1381, 960, 24965, 6539, 62, 5, 3318, 23, 5, 13977, 7, 1045, 10, 723, 516, 1437, 2498, 10, 3143, 9, 19728, 16889, 44259, 118, 1705, 75, 2045, 7, 146, 24, 173, 4, 89, 21, 95, 98, 203, 10199, 15, 162, 328, 24, 1299, 101, 939, 21, 2498, 181, 1176, 16711, 36, 463, 45, 11, 10, 11962, 128, 10724, 110, 181, 1176, 16711, 66, 108, 169, 322, 939, 657, 5, 1114, 8, 5, 1345, 2092, 7, 33, 41, 15652, 516, 8, 847, 4, 15, 162, 1437, 42, 13855, 6439, 21, 95, 2, 1, 1, 1]\t\"I wanted to love this jumpsuit. it looks adorable in the product photo. i am 5'5\"\" about 120lbs and thus was overwhelming on me. i tried everything--- pulling up the tie at the waist to create a higher line  wearing a variety of heel heights----i couldn't seem to make it work. there was just so much fabric on me! it felt like i was wearing pajamas (and not in a cute 'wear your pajamas out' way). i love the idea and the photo appears to have an adorable line and cut. on me  this jumpsuit was just\"\t2023-09-13T08:06:39Z\n",
      "413\t-1\t0\t[0, 100, 437, 12483, 5, 1421, 2018, 5, 23204, 16, 23, 513, 195, 108, 466, 113, 1437, 24, 16, 2829, 30197, 15, 69, 1437, 1437, 53, 24, 1819, 630, 75, 356, 2422, 30197, 4, 939, 437, 195, 108, 306, 113, 8, 63, 2422, 30197, 15, 162, 4, 190, 19, 239, 13977, 10844, 4, 1437, 67, 1437, 24, 16, 5784, 24, 17414, 4, 939, 190, 1381, 24, 19, 10, 740, 5602, 223, 53, 5, 24, 17414, 24787, 376, 235, 149, 5, 740, 5602, 4, 1437, 350, 1099, 1437, 63, 10, 11962, 1521, 1437, 95, 782, 103, 35837, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\t\"I'm assuming the model showing the sweater is at least 5'9\"\"  it is slightly cropped on her   but it certainly doesn't look super cropped. i'm 5'4\"\" and its super cropped on me. even with high waist jeans.  also  it is incredibly itchy. i even tried it with a cami under but the itchy wool came right through the cami.  too bad  its a cute design  just needs some tweaking.\"\t2023-09-13T08:06:39Z\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 ./balanced/sentiment-train/part-algo-1-womens_clothing_ecommerce_reviews.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "review_id\tsentiment\tlabel_id\tinput_ids\treview_body\tdate\n",
      "6395\t1\t2\t[0, 713, 16, 10, 12405, 1437, 30228, 6399, 4, 939, 437, 195, 108, 406, 1437, 18817, 23246, 8, 41, 3023, 29, 2564, 6683, 4, 15331, 101, 209, 1437, 14, 218, 75, 2564, 41783, 352, 15, 127, 124, 1719, 127, 2129, 24210, 36, 4138, 15557, 43, 8, 146, 162, 356, 101, 939, 524, 2934, 62, 1359, 4, 2579, 328, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tThis is a subtle  classy shirt. i'm 5'7  122 lbs and an xs fit perfectly. shirts like these  that don't fit snugly on my back cover my poor posture (genetic) and make me look like i am standing up straight. nice!\t2023-09-13T08:06:39Z\n",
      "766\t1\t2\t[0, 100, 3584, 5, 23491, 1732, 1437, 8, 5, 3195, 16, 4613, 8, 11577, 4, 24, 2564, 127, 1354, 16467, 1437, 8, 79, 16, 195, 108, 698, 845, 939, 206, 79, 40, 28, 2498, 42, 9540, 747, 1437, 79, 6138, 24, 16506, 79, 16, 883, 107, 793, 8, 34, 10, 2038, 633, 1437, 98, 79, 40, 356, 5, 233, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\t\"I purchased the coral version  and the color is wonderful and vibrant. it fit my daughter beautifully  and she is 5'10\"\". i think she will be wearing this coat often  she loves it!!! she is 23 years old and has a professional job  so she will look the part.\"\t2023-09-13T08:06:39Z\n",
      "9620\t1\t2\t[0, 43361, 7, 3568, 19, 909, 50, 10521, 4, 939, 40, 3568, 25, 10, 173, 8443, 1437, 45, 4784, 95, 41, 66, 10724, 2125, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tEasy to wear with black or grey. i will wear as a work jacket  not necessarily just an outwear piece\t2023-09-13T08:06:39Z\n",
      "3232\t1\t2\t[0, 713, 3588, 16, 98, 1256, 11, 621, 4, 5, 8089, 32, 372, 111, 5, 2440, 8, 2272, 8, 95, 2721, 11, 621, 4, 5, 2564, 16, 326, 1872, 4, 939, 1369, 7, 33, 10, 2440, 7494, 8, 115, 2496, 24, 95, 25, 5, 1421, 34, 1437, 53, 269, 24, 630, 75, 240, 24, 4, 24, 16, 2422, 34203, 190, 396, 10, 7494, 4, 939, 524, 95, 10, 2549, 9152, 9, 195, 108, 245, 113, 1437, 8, 939, 1395, 3568, 20250, 50, 24, 74, 8386, 5, 1929, 4, 53, 1271, 10, 132, 112, 73, 176, 113, 27288, 16, 1969, 7, 489, 24, 160, 5, 1929, 36, 2362, 23385, 7059, 956, 322, 939, 218, 75, 2333, 582, 455, 425, 1437, 53, 939, 524, 7785, 939, 222, 11, 42, 2]\t\"This dress is so pretty in person. the colors are great - the blue and green and just beautiful in person. the fit is tts. i happen to have a blue belt and could style it just as the model has  but really it doesn't need it. it is super flattering even without a belt. i am just a hair shy of 5'5\"\"  and i cannot wear flats or it would drag the floor. but adding a 2 1/2\"\" wedge is perfect to keep it off the floor (no hemming needed). i don't usually pay full price  but i am glad i did in this case.\"\t2023-09-13T08:06:39Z\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 ./balanced/sentiment-validation/part-algo-1-womens_clothing_ecommerce_reviews.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "review_id\tsentiment\tlabel_id\tinput_ids\treview_body\tdate\n",
      "7555\t0\t1\t[0, 713, 13454, 16, 11962, 1437, 489, 11, 1508, 5, 1104, 16, 4940, 11708, 4, 53, 24, 18, 1307, 4, 939, 2740, 41, 37863, 29, 8, 24, 18, 182, 7082, 8, 385, 31766, 219, 1437, 61, 16, 533, 5, 2496, 53, 77, 222, 39328, 120, 7, 25477, 116, 67, 1437, 10, 4940, 11708, 1437, 5072, 10, 2345, 9, 33639, 906, 13178, 13454, 13, 68, 2466, 116, 117, 169, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tThis tee is cute  keep in mind the white is totally sheer. but it's huge. i ordered an xxs and it's very loose and drapey  which is likely the style but when did sizing get to unrealistic? also  a totally sheer  basically a sort of fancier cotton tee for $60? no way.\t2023-09-13T08:06:39Z\n",
      "16022\t-1\t0\t[0, 100, 2854, 19, 97, 34910, 4, 24, 16, 2721, 53, 350, 3229, 198, 5, 7050, 4, 939, 524, 2333, 41, 3023, 23115, 8, 5, 3023, 23115, 21, 3229, 11, 5, 7050, 4, 939, 2162, 5, 650, 8, 24, 21, 202, 9800, 4, 16748, 939, 56, 7, 671, 24, 4, 939, 524, 195, 108, 306, 113, 1437, 23006, 23246, 8, 10, 2631, 438, 1437, 30, 117, 839, 939, 524, 1687, 380, 7050, 196, 98, 42, 16, 2200, 5425, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\t\"I agree with other reviewers. it is beautiful but too tight around the chest. i am usually an xsmall and the xsmall was tight in the chest. i bought the small and it was still uncomfortable. sadly i had to return it. i am 5'4\"\"  134 lbs and a 34c  by no means i am considered big chested so this is highly unusual.\"\t2023-09-13T08:06:39Z\n",
      "6891\t0\t1\t[0, 10365, 44184, 131, 1109, 8, 3473, 1468, 53, 5, 3318, 12, 20020, 782, 10, 6399, 19, 10, 828, 55, 3184, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tToo floppy; light and comfortable material but the tie-neck needs a shirt with a bit more structure.\t2023-09-13T08:06:39Z\n",
      "5182\t0\t1\t[0, 713, 16576, 1237, 182, 739, 5579, 12963, 11, 5, 4716, 1459, 516, 4, 89, 21, 67, 117, 3989, 7, 5, 16576, 36, 415, 513, 25, 24, 1415, 15, 127, 809, 322, 5, 3195, 21, 2579, 36, 179, 5, 6907, 12, 571, 31766, 1173, 30863, 43, 1437, 53, 3284, 5, 16576, 95, 1415, 101, 10, 10178, 15, 127, 809, 36, 612, 642, 322, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tThis skirt runs very large--even in the petite line. there was also no shape to the skirt (at least as it looked on my body). the color was nice (in the pink-grapeish hue)  but ultimately the skirt just looked like a tent on my body (00p).\t2023-09-13T08:06:39Z\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 ./balanced/sentiment-test/part-algo-1-womens_clothing_ecommerce_reviews.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-3.'></a>\n",
    "# 3. Query the Feature Store\n",
    "In addition to transforming the data and saving in S3 bucket, the processing job populates the feature store with the transformed and balanced data.  Let's query this data using Amazon Athena."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-3.1.'></a>\n",
    "### 3.1. Export training, validation, and test datasets from the Feature Store\n",
    "\n",
    "Here you will do the export only for the training dataset, as an example. \n",
    "\n",
    "Use `athena_query()` function to create an Athena query for the defined above Feature Group. Then you can pull the table name of the Amazon Glue Data Catalog table which is auto-generated by Feature Store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Glue Catalog table name: reviews_feature_group_1694591880_1694592368\n",
      "Running query: \n",
      "    SELECT date,\n",
      "        review_id,\n",
      "        sentiment, \n",
      "        label_id,\n",
      "        input_ids,\n",
      "        review_body\n",
      "    FROM \"reviews_feature_group_1694591880_1694592368\" \n",
      "    WHERE split_type='train' \n",
      "    LIMIT 5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "feature_store_query = feature_group.athena_query()\n",
    "\n",
    "feature_store_table = feature_store_query.table_name\n",
    "\n",
    "query_string = \"\"\"\n",
    "    SELECT date,\n",
    "        review_id,\n",
    "        sentiment, \n",
    "        label_id,\n",
    "        input_ids,\n",
    "        review_body\n",
    "    FROM \"{}\" \n",
    "    WHERE split_type='train' \n",
    "    LIMIT 5\n",
    "\"\"\".format(feature_store_table)\n",
    "\n",
    "print('Glue Catalog table name: {}'.format(feature_store_table))\n",
    "print('Running query: {}'.format(query_string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the S3 location for the query results.  This allows us to re-use the query results for future queries if the data has not changed.  We can even share this S3 location between team members to improve query performance for common queries on data that does not change often."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-662464974816/query_results/reviews-feature-store-1694591880/\n"
     ]
    }
   ],
   "source": [
    "output_s3_uri = 's3://{}/query_results/{}/'.format(bucket, feature_store_offline_prefix)\n",
    "print(output_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-4'></a>\n",
    "### Exercise 4\n",
    "\n",
    "Query the feature store.\n",
    "\n",
    "**Instructions**: Use `feature_store_query.run` function passing the constructed above query string and the location of the output S3 bucket.\n",
    "\n",
    "```python\n",
    "feature_store_query.run(\n",
    "    query_string=..., # query string\n",
    "    output_location=... # location of the output S3 bucket\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "feature_store_query.run(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    query_string=query_string, # Replace None\n",
    "    output_location=output_s3_uri # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    ")\n",
    "\n",
    "feature_store_query.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>review_id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>label_id</th>\n",
       "      <th>input_ids</th>\n",
       "      <th>review_body</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2023-09-13T08:06:39Z</td>\n",
       "      <td>10636</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[0, 100, 657, 5, 6184, 8, 8089, 9, 42, 3588, 4, 5, 1318, 9, 5, 10199, 16, 372, 4, 939, 3225, 274...</td>\n",
       "      <td>I love the pattern and colors of this dress. the quality of the fabric is great. i initially ord...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2023-09-13T08:06:39Z</td>\n",
       "      <td>9495</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>[0, 100, 657, 5951, 10844, 4, 939, 308, 484, 1763, 1437, 8, 939, 206, 51, 10397, 157, 1437, 8, 3...</td>\n",
       "      <td>I love ag jeans. i own several pair  and i think they wash well  and wear well. that being said ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2023-09-13T08:06:39Z</td>\n",
       "      <td>12806</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[0, 19065, 410, 3294, 3588, 328, 13, 162, 24, 2551, 7, 422, 1528, 7, 1836, 4, 939, 437, 195, 108...</td>\n",
       "      <td>Great little shift dress! for me it seemed to run true to size. i'm 5' 4.5\" and curvy. the regul...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2023-09-13T08:06:39Z</td>\n",
       "      <td>15798</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>[0, 100, 657, 5, 11956, 34203, 2564, 9, 209, 9304, 53, 5, 13504, 1597, 1237, 7586, 102, 319, 4, ...</td>\n",
       "      <td>I love the relaxed flattering fit of these pants but the navy die runs..a lot. the first time i ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2023-09-13T08:06:39Z</td>\n",
       "      <td>17494</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[0, 100, 33, 155, 15029, 9, 209, 1855, 10968, 939, 657, 106, 98, 203, 4, 51, 283, 3793, 101, 10,...</td>\n",
       "      <td>I have 3 pairs of these chinos i love them so much. they come soft like a well-worn favorite but...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   date  review_id  sentiment  label_id  \\\n",
       "0  2023-09-13T08:06:39Z      10636          1         2   \n",
       "1  2023-09-13T08:06:39Z       9495          0         1   \n",
       "2  2023-09-13T08:06:39Z      12806          1         2   \n",
       "3  2023-09-13T08:06:39Z      15798          0         1   \n",
       "4  2023-09-13T08:06:39Z      17494          1         2   \n",
       "\n",
       "                                                                                             input_ids  \\\n",
       "0  [0, 100, 657, 5, 6184, 8, 8089, 9, 42, 3588, 4, 5, 1318, 9, 5, 10199, 16, 372, 4, 939, 3225, 274...   \n",
       "1  [0, 100, 657, 5951, 10844, 4, 939, 308, 484, 1763, 1437, 8, 939, 206, 51, 10397, 157, 1437, 8, 3...   \n",
       "2  [0, 19065, 410, 3294, 3588, 328, 13, 162, 24, 2551, 7, 422, 1528, 7, 1836, 4, 939, 437, 195, 108...   \n",
       "3  [0, 100, 657, 5, 11956, 34203, 2564, 9, 209, 9304, 53, 5, 13504, 1597, 1237, 7586, 102, 319, 4, ...   \n",
       "4  [0, 100, 33, 155, 15029, 9, 209, 1855, 10968, 939, 657, 106, 98, 203, 4, 51, 283, 3793, 101, 10,...   \n",
       "\n",
       "                                                                                           review_body  \n",
       "0  I love the pattern and colors of this dress. the quality of the fabric is great. i initially ord...  \n",
       "1  I love ag jeans. i own several pair  and i think they wash well  and wear well. that being said ...  \n",
       "2  Great little shift dress! for me it seemed to run true to size. i'm 5' 4.5\" and curvy. the regul...  \n",
       "3  I love the relaxed flattering fit of these pants but the navy die runs..a lot. the first time i ...  \n",
       "4  I have 3 pairs of these chinos i love them so much. they come soft like a well-worn favorite but...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.set_option(\"max_colwidth\", 100)\n",
    "\n",
    "df_feature_store = feature_store_query.as_dataframe()\n",
    "df_feature_store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the Feature Store in SageMaker Studio\n",
    "\n",
    "![](images/sm_studio_extensions_featurestore.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-3.2.'></a>\n",
    "### 3.2. Export TSV from Feature Store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the output as a TSV file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_feature_store.to_csv('./feature_store_export.tsv',\n",
    "                        sep='\\t',\n",
    "                        index=False,\n",
    "                        header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "date\treview_id\tsentiment\tlabel_id\tinput_ids\treview_body\n",
      "2023-09-13T08:06:39Z\t10636\t1\t2\t[0, 100, 657, 5, 6184, 8, 8089, 9, 42, 3588, 4, 5, 1318, 9, 5, 10199, 16, 372, 4, 939, 3225, 2740, 127, 4505, 1836, 3023, 29, 1437, 53, 303, 24, 10, 828, 929, 219, 98, 939, 2740, 258, 41, 3023, 4182, 8, 41, 37863, 29, 4, 5, 37863, 29, 21, 55, 15898, 8, 5, 3023, 4182, 21, 350, 765, 11, 127, 2979, 4, 939, 524, 202, 15, 5, 8146, 59, 5, 3023, 29, 8, 5, 37863, 29, 4, 5, 37863, 29, 34, 10, 15976, 2564, 1437, 53, 24, 16, 67, 3229, 11, 5, 11044, 8, 939, 5170, 141, 24, 40, 28, 114, 939, 33, 7, 1338, 13, 402, 111, 40, 24, 28, 350, 3229, 116, 190, 600, 5, 3023, 29, 1237, 2829, 380, 1437, 939, 64, 2]\tI love the pattern and colors of this dress. the quality of the fabric is great. i initially ordered my usual size xs  but found it a bit roomy so i ordered both an xsp and an xxs. the xxs was more fitted and the xsp was too short in my opinion. i am still on the fence about the xs and the xxs. the xxs has a tighter fit  but it is also tight in the bust and i wonder how it will be if i have to reach for something - will it be too tight? even though the xs runs slightly big  i can still work with\n",
      "2023-09-13T08:06:39Z\t9495\t0\t1\t[0, 100, 657, 5951, 10844, 4, 939, 308, 484, 1763, 1437, 8, 939, 206, 51, 10397, 157, 1437, 8, 3568, 157, 4, 14, 145, 26, 1437, 939, 1017, 393, 3584, 10, 1763, 9, 5, 22, 620, 3623, 324, 113, 847, 137, 4, 51, 33, 5, 276, 1318, 65, 74, 1057, 31, 5951, 1437, 8, 939, 101, 14, 5, 35713, 2025, 75, 1528, 22877, 10844, 1437, 53, 33, 10, 410, 22523, 4, 24, 18, 55, 15, 12, 90, 10082, 13, 42, 191, 4, 13, 162, 1437, 5, 696, 21, 55, 19, 5, 3195, 4, 939, 645, 6907, 4, 77, 51, 2035, 1437, 51, 58, 2721, 11, 5, 410, 4136, 3298, 4, 127, 1144, 22904, 10, 1451, 4, 1368, 4623, 857, 328, 11962, 92, 10844, 328, 4682, 1437, 885, 2]\t\"I love ag jeans. i own several pair  and i think they wash well  and wear well. that being said  i'd never purchased a pair of the \"\"stevie\"\" cut before. they have the same quality one would expect from ag  and i like that the ankles aren't true skinny jeans  but have a little width. it's more on-trend for this season. for me  the issue was more with the color. i order pink. when they arrived  they were beautiful in the little plastic bag. my heart skipped a beat. hooray! cute new jeans! except  w\"\n",
      "2023-09-13T08:06:39Z\t12806\t1\t2\t[0, 19065, 410, 3294, 3588, 328, 13, 162, 24, 2551, 7, 422, 1528, 7, 1836, 4, 939, 437, 195, 108, 204, 4, 245, 113, 8, 5350, 11454, 4, 5, 1675, 1836, 158, 2564, 1969, 1437, 59, 155, 113, 1065, 5, 4117, 4, 10, 269, 1531, 1521, 19, 5, 21997, 4, 64, 75, 2067, 7, 3568, 683, 5, 1958, 16, 2198, 1613, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\t\"Great little shift dress! for me it seemed to run true to size. i'm 5' 4.5\"\" and curvy. the regular size 10 fit perfect  about 3\"\" above the knee. a really fun design with the fringe. can't wait to wear once the snow is completely gone.\"\n",
      "2023-09-13T08:06:39Z\t15798\t0\t1\t[0, 100, 657, 5, 11956, 34203, 2564, 9, 209, 9304, 53, 5, 13504, 1597, 1237, 7586, 102, 319, 4, 5, 78, 86, 939, 15158, 106, 326, 222, 98, 228, 9223, 15, 5, 6929, 4, 939, 222, 127, 220, 7511, 9, 16937, 8, 960, 1224, 2440, 73, 10521, 328, 939, 1381, 14784, 106, 456, 19, 25271, 7, 278, 5, 3195, 8, 5952, 14, 40, 244, 4, 3739, 9, 3195, 878, 202, 600, 1437, 98, 7, 28, 1522, 10397, 8, 172, 422, 21, 1843, 19, 1085, 11, 24, 7, 2382, 24, 137, 608, 55, 16937, 4, 939, 437, 2396, 106, 142, 939, 101, 5, 9304, 1437, 53, 770, 7, 905, 47, 11269, 139, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\tI love the relaxed flattering fit of these pants but the navy die runs..a lot. the first time i washed them t did so per instructions on the label. i did my next load of laundry and everything turned blue/ grey! i tried washing them again with vinegar to set the color and hopefully that will help. lots of color running still though  so to be safe wash and then run washer with nothing in it to clean it before doing more laundry. i'm keeping them because i like the pants  but wanted to let you kno\n"
     ]
    }
   ],
   "source": [
    "!head -n 5 ./feature_store_export.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload TSV to the S3 bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upload: ./feature_store_export.tsv to s3://sagemaker-us-east-1-662464974816/feature_store/feature_store_export.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp ./feature_store_export.tsv s3://$bucket/feature_store/feature_store_export.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the file in the S3 bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-13 08:13:55       5014 feature_store/feature_store_export.tsv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls --recursive s3://$bucket/feature_store/feature_store_export.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-3.3.'></a>\n",
    "### 3.3. Check that the dataset in the Feature Store is balanced by sentiment\n",
    "\n",
    "Now you can setup an Athena query to check that the stored dataset is balanced by the target class `sentiment`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-5'></a>\n",
    "### Exercise 5\n",
    "\n",
    "Write an SQL query to count the total number of the reviews per `sentiment` stored in the Feature Group.\n",
    "\n",
    "**Instructions**: Pass the SQL statement of the form \n",
    "\n",
    "```sql\n",
    "SELECT category_column, COUNT(*) AS new_column_name\n",
    "FROM table_name\n",
    "GROUP BY category_column\n",
    "```\n",
    "\n",
    "into the variable `query_string_count_by_sentiment`. Here you would need to use the column `sentiment` and give a name `count_reviews` to the new column with the counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "feature_store_query_2 = feature_group.athena_query()\n",
    "\n",
    "# Replace all None\n",
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "query_string_count_by_sentiment = \"\"\"\n",
    "SELECT sentiment, COUNT(*) AS sentiment_count\n",
    "FROM \"{}\"\n",
    "GROUP BY sentiment\n",
    "\"\"\".format(feature_store_table)\n",
    "### END SOLUTION - DO NOT delete this comment for grading purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-6'></a>\n",
    "### Exercise 6\n",
    "\n",
    "Query the feature store.\n",
    "\n",
    "**Instructions**: Use `run` function of the Feature Store query, passing the new query string `query_string_count_by_sentiment`. The output S3 bucket will remain unchanged. You can follow the example above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_store_query_2.run(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    query_string=query_string_count_by_sentiment, # Replace None\n",
    "    output_location=None # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    ")\n",
    "\n",
    "feature_store_query_2.wait()\n",
    "\n",
    "df_count_by_sentiment = feature_store_query_2.as_dataframe()\n",
    "df_count_by_sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c2w1-ex-7'></a>\n",
    "### Exercise 7\n",
    "\n",
    "Visualize the result of the query in the bar plot, showing the count of the reviews by sentiment value.\n",
    "\n",
    "**Instructions**: Pass the resulting data frame `df_count_by_sentiment` into the `barplot` function of the `seaborn` library.\n",
    "\n",
    "```python\n",
    "sns.barplot(\n",
    "    data=..., \n",
    "    x='...', \n",
    "    y='...',\n",
    "    color=\"blue\"\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "\n",
    "sns.barplot(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    data=None, # Replace None\n",
    "    x=None, # Replace None\n",
    "    y=None, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    color=\"blue\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the notebook and `prepare_data.py` file into S3 bucket for grading purposes.\n",
    "\n",
    "**Note**: you may need to save the file before the upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp ./C2_W1_Assignment.ipynb s3://$bucket/C2_W1_Assignment_Learner.ipynb\n",
    "!aws s3 cp ./src/prepare_data.py s3://$bucket/src/C2_W1_prepare_data_Learner.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please go to the main lab window and click on `Submit` button (see the `Finish the lab` section of the instructions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}