notebooks/GenAI/AWS_GenAI_Huggingface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "92021b22-3fbf-4489-9e88-aea4d73f3529",
   "metadata": {},
   "source": [
    "# Training and Deploying Huggingface Models on Sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b722b9e",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfbc2ea5-1aca-4322-a15c-7bcbb3925ef6",
   "metadata": {},
   "source": [
    "This tutorial will focus on utilizing Hugging Face Hub which is a repository for users to share and download machine learning models, datasets, and demos. AWS has partnerd with Hugging Face to allow users to utilize these resources without the need to manually create a account or token with Hugging Face. All resources are avaiable using the sagemaker.huggingface API.\n",
    "\n",
    "For this tutorial we will load in a model and dataset from Hugging Face and train and test our model before deploying it on Sagemaker. The model we will be deploying is Flan T5 and the datasets is [ccdv/pubmed-summarization](https://huggingface.co/datasets/ccdv/pubmed-summarization)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84dacee7",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "For this tutorial it is recommened to use 1 GPU to speed up processes, this notebooks was run the machinetype ml.p3.2xlarge.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4c9338e",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "+ Learn how to access and import Hugging Face models directly to SageMaker\n",
    "+ Learn how to finetune an LLM based on new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "448d8b6a",
   "metadata": {},
   "source": [
    "## Get Started\n",
    "\n",
    "### Install packages and set up environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35361056-a599-48d8-b687-a3e1570c1ffc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enable GPU to be in persistant mode\n",
    "! nvidia-smi -pm 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "161dea96-601e-4194-a285-51b0c4403e9d",
   "metadata": {},
   "source": [
    "Hugging Face **transformers** are an open-source framework that allow you to utilize APIs and tools to download pretrained models, set hyperparameters, tokenize datasets, and further tune them to suit your needs. Here we are updating SageMaker as well as installing the transformers package and **datasets** so that we can have access to Hugging Face datasets. As a bonus we are adding the S3 feature to help download datasets that may already be in a S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6e5884b-ac90-42d4-aafd-d34d5495d24d",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install \"sagemaker\" \"transformers\" \"datasets[s3]\" --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36efac5f-d4e8-45ce-b720-0508e4b4801c",
   "metadata": {},
   "source": [
    "Next we install **sentencepiece** which is an unsupervised text tokenizer and **accelerate**, another Hugging Face tool that allows PyTorch models to run on multiple GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ec2af2b-dfdb-4862-87a4-b5c7a8227f86",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install \"sentencepiece\" \"accelerate\" --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2442bd22-9653-4f3b-8fbd-cfa6a9f87366",
   "metadata": {},
   "source": [
    "### Set up your Sagemaker Session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e077b8ea-4f9e-4c1c-95ba-a103a6e0fcbb",
   "metadata": {},
   "source": [
    "The following commands have Sagemaker create a session that will automatically create a bucket which will store our training and testing datasets, extract our role id, and region both will be used later for hypertuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d7555e7-0e8a-4141-9338-8c666569c5f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "sagemaker_session_bucket = None\n",
    "if sagemaker_session_bucket is None and sess is not None:\n",
    "    sagemaker_session_bucket = sess.default_bucket()\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n",
    "print(f\"sagemaker session region: {sess.boto_region_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b287a7-3035-48c1-8520-a7d3de4f925b",
   "metadata": {},
   "source": [
    "### Download your dataset from Huggingface"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd58ed15-7e8f-4f24-b68f-4a5b5fcd5cf2",
   "metadata": {},
   "source": [
    "We will be downloading huggingface dataset 'ccdv/pubmed-summarization' which contains article titles and their abstracts which will help train our model to summarize the scientific articles. Once the dataset is loaded we'll split the data into test and train datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ef5bc62-c95d-4c5c-b220-d678264c6155",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFaceModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f59e17e-c006-45ee-be0b-766774f9d420",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "# load dataset\n",
    "train_dataset, test_dataset = load_dataset(\"ccdv/pubmed-summarization\", split=[\"train\", \"test\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3399abb1-af8f-46ee-92ea-c8344eeddd09",
   "metadata": {},
   "source": [
    "## Finetuning our Model Locally"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed6ddff1-2636-4e3b-88ee-e3c86c584245",
   "metadata": {},
   "source": [
    "Now that we have our datasets we can upload our model which will be the small version of Flan T5.\n",
    "\n",
    "**Flan T5** is a text-to-text generation model and an advancement to the original T5 model and can be run on both CPUs and GPUs. **Text-to-text** is a method of creating text by using a neural network to generate new text from a given input. These T5 models can be fine-tuned for various zero shot NLP tasks that we have seen and heard of before: text classification, summarization, translation, and question-answering. Text-to-text is not to be confused by text2text generation which is a earlier version of T5 that is designed specifically for sequence-to-sequence tasks, such as machine translation and text generation and is limited to these task where as T5 models are more flexible due to the wider range of NLP tasks they can execute.\n",
    "\n",
    "Because it is a seq2seq class model we will be using the transformer **AutoModelForSeq2Seq** to help find and load our pretrained model architecture. Then we will assign an **AutoTokenizer** to preprocess the text of our inputs (the test and train datasets) into an array of numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1988cbcb-4bec-4aa2-a356-a211584ceacb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
    "model_name=\"google/flan-t5-small\"\n",
    "\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ca0419-0075-4f62-becf-b859312cea22",
   "metadata": {},
   "source": [
    "Now that we have loaded the architecture of our model and configured it to tokenize our inputs we can now implement a tokenization functions to start processing our datasets.\n",
    "\n",
    "The function below will tokenize each row of our dataset based on the 'article' column. now that we have our function the next step is to implement the **map** function to interate our **tokenize function** to our loaded datasets.Then the last step will be to set our data format to be suitable for Pytorch. As you can see there are three columns represented in the dataset:\n",
    "- **input_ids:** ID for each token, as each text is broken up into sequences (which can be words or subwords) and converted to tokens within our dataset they are assign an ID.\n",
    "- **attention_masks:** Tokens that should be ignored by the model usually represented by a 0. Masking can be done when some sequences are not the same length so they can not belong in the same tensor and need to be padded.\n",
    "- **abstracts:** The new name of the abstract column, which is the column we are implementing the new PyTorch format "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce73b475-2809-4505-901b-53daf3577693",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create tokenization function\n",
    "def tokenize(batch):\n",
    "    return tokenizer(batch[\"article\"], padding=\"max_length\", truncation=True)\n",
    "\n",
    "\n",
    "# tokenize train and test datasets\n",
    "train_dataset = train_dataset.map(tokenize, batched=True)\n",
    "test_dataset = test_dataset.map(tokenize, batched=True)\n",
    "\n",
    "# set dataset format for PyTorch\n",
    "train_dataset =  train_dataset.rename_column(\"abstract\", \"abstracts\")\n",
    "train_dataset.set_format(\"torch\", columns=[\"input_ids\", \"attention_mask\", \"abstracts\"])\n",
    "test_dataset = test_dataset.rename_column(\"abstract\", \"abstracts\")\n",
    "test_dataset.set_format(\"torch\", columns=[\"input_ids\", \"attention_mask\", \"abstracts\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ffd612-abde-4666-8c85-cc7069de2129",
   "metadata": {},
   "source": [
    "The first step to training our model other than setting up our datasets is to set our **hyperparameters**. Hyperparameters depend on your training script and for this one we need to identify our model, the location of our train and test files, etc. iN this case we are using a one created by Hugging Face."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c06bef19-cc3c-476f-943c-78368e9f49e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "training_args = TrainingArguments(output_dir=\"test_trainer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff31d69-9f54-4235-a377-7c5e758fbca8",
   "metadata": {},
   "source": [
    "Next create setting to evaluate the models accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24bbe62e-9140-4bef-88ae-3e5029ddb25c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import evaluate\n",
    "\n",
    "metric = evaluate.load(\"accuracy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b82caeba-2daa-4526-b67d-04f45d4a9934",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "    logits, labels = eval_pred\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "    return metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5b50ec0-87b8-4578-96aa-e26bda9d99b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer\n",
    "\n",
    "training_args = TrainingArguments(output_dir=\"test_trainer\", evaluation_strategy=\"epoch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df2225ac-8e92-4a14-a368-eebff9ead6bf",
   "metadata": {},
   "source": [
    "Finally we can train our model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e59332ae-c9e3-4a9b-9a7c-7020c87227da",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=test_dataset,\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35520bb-b6ca-4996-b87e-2fbfdcfc0dff",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ac841f6-c65e-4ebf-8c42-3030e2f92cb0",
   "metadata": {},
   "source": [
    "### Setting up our Datasets for Training "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ce7e0e-9012-497b-acc1-68cf488bee50",
   "metadata": {},
   "source": [
    "We are using the training script run_summarization.py which will help train our Flan T5 model to summarize our pubmed datasets. To pass inputs to this script we first need to convert our datasets into csv format and then push them into our S3 bucket with the help of the boto3 package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a082dfc-6aa3-4e18-a413-90192ac5446e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "import boto3\n",
    "\n",
    "#convert train dataset to csv and push to S3 bucket\n",
    "csv_buffer = BytesIO()\n",
    "train_dataset.to_csv(csv_buffer)\n",
    "s3_resource = boto3.resource('s3')\n",
    "s3_resource.Object(f'{sess.default_bucket()}', 'train.csv').put(Body=csv_buffer.getvalue())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "234385ca-99f9-414e-be47-0e6f618c3eb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "#convert test dataset to csv and push to S3 bucket\n",
    "csv_buffer = BytesIO()\n",
    "test_dataset.to_csv(csv_buffer)\n",
    "s3_resource = boto3.resource('s3')\n",
    "s3_resource.Object(f'{sess.default_bucket()}', 'test.csv').put(Body=csv_buffer.getvalue())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f18630a7-109c-4f53-9233-1842f5c27029",
   "metadata": {},
   "source": [
    "Here we will be saving the location of our datasets and group with a label called **data** which will be used when we execute the training of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebc1bc39-a554-473b-949a-d9588f6e7fb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save train_dataset to s3\n",
    "training_input_path = f's3://{sess.default_bucket()}/train.csv'\n",
    "\n",
    "# save test_dataset to s3\n",
    "test_input_path = f's3://{sess.default_bucket()}/test.csv'\n",
    "\n",
    "# Group the training and testing data since this will be the input to our hugging face estimator\n",
    "data = {\n",
    "    'train': training_input_path,\n",
    "    'test': test_input_path\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9204b6dc-8f6e-407e-8c68-a036a6a5b7c9",
   "metadata": {},
   "source": [
    "### Training our ModelFinetuning our Model via Vertex AI Training API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a66d0c47-f6df-4b79-87a6-a637b04ebc87",
   "metadata": {},
   "source": [
    "The first step to training our model other than setting up our datasets is to set our **hyperparameters**. Hyperparameters depend on your training script and for this one we need to identify our model, the location of our train and test files, if we want to train and test our model, etc other hyperparameters are defined on the huggingface transformers Github [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization).\n",
    "\n",
    "our train andn test file locations are 'opt/ml/input/data/train' and 'opt/ml/input/data/test' because Sagemaker will train our model on a docker container and pull our files from our bucket and store then in a test and train directory. It will then output a model file back into our S3 bucket by first storing it into the directory '/opt/ml/model'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5266d52a-3e4d-4320-8bdd-d8610da46b7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#location of our run_summarization.py training file, which will be downloaded automatically\n",
    "git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.26.0'} # v4.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b7b346b-66b3-4557-b959-8ce2b6d677a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters, which are passed into the training job\n",
    "hyperparameters={'per_device_train_batch_size': 2,\n",
    "                 'per_device_eval_batch_size': 4,\n",
    "                 'model_name_or_path': 'google/flan-t5-small',\n",
    "                 'train_file': '/opt/ml/input/data/train/train.csv',\n",
    "                 'test_file':'/opt/ml/input/data/test/test.csv',\n",
    "                 'text_column':'article',\n",
    "                 'summary_column':'abstracts',\n",
    "                 #'source_prefix': \"summarize: \",\n",
    "                 'do_train': True,\n",
    "                 'do_eval': False,\n",
    "                 'do_predict': True,\n",
    "                 'predict_with_generate': True,\n",
    "                 'output_dir': '/opt/ml/model',\n",
    "                 'num_train_epochs': 3,\n",
    "                 'learning_rate': 5e-5,\n",
    "                 'seed': 7,\n",
    "                 'fp16': True,\n",
    "                 }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d893eb99-d117-4dc5-b3c7-28e66b038256",
   "metadata": {},
   "source": [
    "To help things move faster we will enable dataparallel which will break up the training tasks and run them in parallel. This does require we run our training on more then one instance and limits the machinetypes we can use:\n",
    "ml.p3.16xlarge, ml.p3dn.24xlarge, ml.p4d.24xlarge, ml.p4de.24xlarge\n",
    "\n",
    "**Note:** If you choose not to use the distributor, commment out that line of code and the line of code that says \"dictribute=distribute\" in the huggingface estimator set up.\n",
    "\n",
    "Now we can set up our huggingface estimator which is a Sagemaker managed execution environment, run via a Docker container. The container will run our training script on our model utilizing a machine type of our choosing and pass our hyperparameters to the container as well. In the end, the estimator will create a Hugging Face directory in the default bucket and output a model.tar.gz file which we can deploy to a endpoint for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "def3b0e0-59ad-4e6d-b297-1b0da318b32b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# configuration for running training on smdistributed Data Parallel\n",
    "distribution = {'smdistributed':{'dataparallel':{ 'enabled': True }}}\n",
    "\n",
    "from sagemaker.huggingface import HuggingFace\n",
    "\n",
    "# create the Estimator\n",
    "huggingface_estimator = HuggingFace(\n",
    "      entry_point='run_summarization.py', # script\n",
    "      source_dir='./examples/pytorch/summarization', # relative path to example\n",
    "      git_config=git_config,\n",
    "      instance_type='ml.p4d.24xlarge',\n",
    "      checkpoint_s3_uri=f's3://{sess.default_bucket()}/checkpoints1',\n",
    "      #checkpoint_local_path='/opt/ml/checkpoints',\n",
    "      instance_count=2,\n",
    "      transformers_version='4.26.0',\n",
    "      pytorch_version='1.13.1',\n",
    "      py_version='py39',\n",
    "      role=role,\n",
    "      hyperparameters = hyperparameters,\n",
    "      distribution = distribution\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca946a34-f3d0-44c1-9273-12e442e89dab",
   "metadata": {},
   "source": [
    "Start the training. We can monitor and view logs on the console by going to `Sagemaker > Training > Training Jobs.`\n",
    "\n",
    "**Warning:** If you recieve a **ResourceLimitExceeded** error it's because there are not enough resources on AWS to use this instance at the moment. To solve this error either try another instance type or try running the training again to see if resources have become available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "264dccbf-37bb-4ef9-810f-c2f16eac15d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# starting the train job\n",
    "huggingface_estimator.fit(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ffad6d-261d-42e1-8913-4be129c04381",
   "metadata": {},
   "source": [
    "### Deploy the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dee7811-559f-4bdc-b56e-b932b0831c0f",
   "metadata": {},
   "source": [
    "Here we are creating an endpoint and deploying our model to the endpoint. Next we will feed the model some inputs and check that it produces an accurate and consise summary.\n",
    "\n",
    "We are deploying our enpoint using 1 GPU which can take 20min to run, feel free to try out other machine types that utilize more GPUs, or a new generation of GPU (just know it will be a bit more expensive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbdbe201-408f-4d4b-8821-14163b8b282a",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = huggingface_estimator.deploy(initial_instance_count=1, instance_type=\"ml.g4dn.xlarge\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d97f45-4861-41cf-ad38-a5cc169c1996",
   "metadata": {},
   "source": [
    "**Optional** \n",
    "\n",
    "If your model takes a long time to train you can come back to your notebook later without worrying about it stopping. If you choose to do this the following code is another way to obtain our model.tar.gz file from your bucket and deploy it. Remember you can monitor your training on the console by going to `Sagemaker > Training > Training Jobs.`\n",
    "\n",
    "Sometimes you need to search in your default bucket to look for your model.tar.gz file it will be in one of the directories that says 'huggingface-pytorch-training'. We are deploying our enpoint using 1 GPU which can take 20min to run, feel free to try out other machine types that utilize more GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f2823a4-71fc-4adf-9911-a1df6ebfed29",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "#huggingface directory that holds your model.tar.gz file\n",
    "huggingface_directory= \"<enter in the directory name>\"\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "   model_data=f's3://{sess.default_bucket()}/{huggingface_directory}/output/model.tar.gz',  # path to your trained SageMaker model\n",
    "   role=role,                                            # IAM role with permissions to create an endpoint\n",
    "   transformers_version=\"4.26\",                           # Transformers version used\n",
    "   pytorch_version=\"1.13\",                                # PyTorch version used\n",
    "   py_version='py39',                                    # Python version used\n",
    ")\n",
    "\n",
    "# deploy model to SageMaker Inference\n",
    "predictor = huggingface_model.deploy(\n",
    "   initial_instance_count=1,\n",
    "   instance_type=\"ml.g4dn.xlarge\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb0d4855-f01c-4e06-a84f-099439fdb344",
   "metadata": {},
   "source": [
    "### Submit Inputs and Parameters to the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fc59aab-9152-4a69-a636-539965b92bea",
   "metadata": {},
   "source": [
    "Now we can pass in text for our model to summarize. Below you will see that we have provided a paragraph about SARS-CoV-2 as our prompt, we also have some parameters that we specify to further tune our model to get a concise summary of what our prompt is about.\n",
    "\n",
    "- **Max_Length:** Max number of words to generate.\n",
    "- **Num_Return_Sequences:** Number of different outputs to generate. For our example we want one sentence or sequence.\n",
    "- **Temperature:** Controls randomness, higher values increase diversity meaning a more unique response make the model to think harder. Must be a number from 0 to 1.\n",
    "- **Top_p (nucleus):** The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Must be a number from 0 to 1.\n",
    "- **Top_k**: Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens.This means the model chooses the most probable words. Lower values eliminate fewer coherent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b4a419-ee5b-4459-bb7a-ac811cb079d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt =  \"\"\"Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a\n",
    "highly transmissible and pathogenic coronavirus that emerged in late 2019 and has \n",
    "caused a pandemic of acute respiratory disease, named ‘coronavirus disease 2019’ (COVID-19), \n",
    "which threatens human health and public safety. In this Review, we describe the basic virology of \n",
    "SARS-CoV-2, including genomic characteristics and receptor use, highlighting its key difference \n",
    "from previously known coronaviruses. We summarize current knowledge of clinical, epidemiological and \n",
    "pathological features of COVID-19, as well as recent progress in animal models and antiviral treatment \n",
    "approaches for SARS-CoV-2 infection. We also discuss the potential wildlife hosts and zoonotic origin \n",
    "of this emerging virus in detail.\"\"\"\n",
    "\n",
    "payload = ({\n",
    "    \"inputs\": prompt,\n",
    "    \"parameters\": {\"max_length\": 2000, \n",
    "                   \"num_return_sequences\": 1, \n",
    "                   \"temperature\":0.6,\n",
    "                   \"top_k\": 50, \n",
    "                   \"top_p\": 0.95,\n",
    "                   \"do_sample\": True,\n",
    "                  }\n",
    "})\n",
    "predictor.predict(payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39bd0579",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "And that is it! You have imported a model from Hugging Face, retrained it using custom data, and then passed in a prompt to generate a model response. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d153411d",
   "metadata": {},
   "source": [
    "## Clean Up\n",
    "Remember to delete your endpoint, model, buckets, and shutdown or delete your Sagemaker notebook to avoid additional charges!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2f65b0-b748-4567-8e7e-7268162a88e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}