E10_News_Summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "executed-wrestling",
   "metadata": {},
   "source": [
    "# LMS Exploration | 10. News_Summary_NLP\n",
    "\n",
    "---\n",
    "\n",
    "**[Introduce]**  \n",
    "\n",
    "뉴스 내용을 읽고 헤드라인 기사를 생성주는 모델을 만들어보겠습니다.  \n",
    "Text Summarization(텍스트 요약) 기법 중 Abstractive Summarization(추상적 요약) 접근법을 먼저 사용해보고,    \n",
    "이후 Extractive Summarization(추출적 요약) 접근법을 사용해서 비교해보겠습니다.   \n",
    "* 추상적 요약은 원문으로부터 내용이 요약된 새로운 문장을 생성해내는 것으로 자연어 생성(Natural Language Generation, NLG)의 영역입니다.\n",
    "* 추출적 요약은 단어 그대로 원문에서 문장들을 추출해서 요약하는 방식으로 문장 분류(Text Classification) 문제로 볼 수 있습니다. \n",
    "\n",
    "**[Data]**  \n",
    "\n",
    "* 98401개의 뉴스기사데이터를 사용합니다.  [News Summary Dataset](https://github.com/sunnysai12345/News_Summary)\n",
    "* news_summary_more Dataset에는 총 98401개의 사례가 있는데, text와 headlines 의 정보를 포함합니다.\n",
    "* text 열의 내용을 요약한 것이 headlines 입니다. \n",
    "* 추상적 요약을 하는 경우에는 text를 본문으로, headlines를 이미 요약된 데이터로 삼아서 모델을 학습합니다.\n",
    "* 반면 추출적 요약을 하는 경우에는 오직 text열만을 사용합니다. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vanilla-gabriel",
   "metadata": {},
   "source": [
    "## 라이브러리와 패키지 불러오기 \n",
    "* 데이터 전처리 과정에서 NLTK의 불용어(stopwords)를 사용하여 불용어 제거 작업을 진행하기위해 nltk를 불러옵니다. \n",
    "* [NLTK](https://www.nltk.org/)는 Natural Language Toolkit의 축약어로 영어 기호, 통계, 자연어 처리를 위한 라이브러리입니다.\n",
    "* NLTK에는 I, my, me, over, 조사, 접미사와 같이 문장에는 자주 등장하지만, 의미를 분석하고 요약하는 데는 거의 의미가 없는 100여개의 불용어가 미리 정리되어 있습니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "forward-entrance",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.corpus import stopwords\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "import urllib.request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "contemporary-maker",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /aiffel/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "apparent-headline",
   "metadata": {},
   "source": [
    "## 데이터 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "applicable-buying",
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/sunnysai12345/News_Summary/master/news_summary_more.csv\", filename=\"news_summary_more.csv\")\n",
    "data = pd.read_csv('news_summary_more.csv', encoding='iso-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "meaningful-australia",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98401\n"
     ]
    }
   ],
   "source": [
    "#- 랜덤한 샘플 10개 출력\n",
    "data.sample(10)\n",
    "print(len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unknown-evanescence",
   "metadata": {},
   "source": [
    "# 1. 추상적 요약\n",
    "\n",
    "* text 시퀀스를 입력받으면, headlines 시퀀스를 예측하도록 인공신경망을 훈련시키고자 합니다. \n",
    "\n",
    "# 1-1. 데이터 전처리\n",
    "\n",
    "* null 데이터 제거\n",
    "* 중복 데이터 제거\n",
    "* 텍스트 정규화\n",
    "* 불용어 제거\n",
    "* 훈련데이터와 테스트데이터 나누기\n",
    "* 단어 집합(vocabulary) 만들기 및 정수 인코딩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "special-nirvana",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text 열에서 중복을 배제한 유일한 샘플의 수 : 98360\n",
      "headlines 열에서 중복을 배제한 유일한 샘플의 수 : 98280\n"
     ]
    }
   ],
   "source": [
    "print('text 열에서 중복을 배제한 유일한 샘플의 수 :', data['text'].nunique())\n",
    "print('headlines 열에서 중복을 배제한 유일한 샘플의 수 :', data['headlines'].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "appropriate-usage",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20754</th>\n",
       "      <td>Aegon Life iTerm offers long-term care with en...</td>\n",
       "      <td>Safeguard yourself against life's unpleasant s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44594</th>\n",
       "      <td>Berger introduces T20 of painting with Express...</td>\n",
       "      <td>Berger Paints has launched Berger Express Pain...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51012</th>\n",
       "      <td>Longest Test innings was longer than 10 footba...</td>\n",
       "      <td>Former Pakistani cricketer Hanif Mohammad batt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52021</th>\n",
       "      <td>Aus bowler took hat-trick in his 3rd ODI, neve...</td>\n",
       "      <td>Australia's Anthony Stuart took a hat-trick in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53578</th>\n",
       "      <td>Lara named daughter after Sydney where he hit ...</td>\n",
       "      <td>Former Windies' captain Brian Lara named his f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               headlines  \\\n",
       "20754  Aegon Life iTerm offers long-term care with en...   \n",
       "44594  Berger introduces T20 of painting with Express...   \n",
       "51012  Longest Test innings was longer than 10 footba...   \n",
       "52021  Aus bowler took hat-trick in his 3rd ODI, neve...   \n",
       "53578  Lara named daughter after Sydney where he hit ...   \n",
       "\n",
       "                                                    text  \n",
       "20754  Safeguard yourself against life's unpleasant s...  \n",
       "44594  Berger Paints has launched Berger Express Pain...  \n",
       "51012  Former Pakistani cricketer Hanif Mohammad batt...  \n",
       "52021  Australia's Anthony Stuart took a hat-trick in...  \n",
       "53578  Former Windies' captain Brian Lara named his f...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data['text'].duplicated()].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "committed-virginia",
   "metadata": {},
   "source": [
    "* headlines 는 간단한 요약들이 많아서  text 가 달라도 headlines 는 동일할 수 있습니다.\n",
    "* 그러나 text 자체가 중복된 경우엔 중복 샘플이므로 제거해야 합니다.         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "basic-tumor",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수: 98360\n"
     ]
    }
   ],
   "source": [
    "data.drop_duplicates(subset=['text'], inplace=True) #- inplace=True 를 설정하면 DataFrame 타입 값을 return 하지 않고 data 내부를 직접적으로 바꿉니다\n",
    "print('전체 샘플수:', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affecting-muslim",
   "metadata": {},
   "source": [
    "중복이 제거되면서 샘플의 수가 98360 개로 줄었지만, 여전히 Null 값을 가지는 데이터는 남아있을 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "amateur-kernel",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "headlines    0\n",
      "text         0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(data.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "settled-violence",
   "metadata": {},
   "source": [
    "확인해보니 null 값은 없습니다. \n",
    "98401 개의 전체 데이터에서 약 41개의 데이터가 제거되었습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "statewide-roommate",
   "metadata": {},
   "source": [
    "## 텍스트 정규화 (text normalization)\n",
    "98360개의 단어중에는 같은 의미인데도 다른 표현으로 쓰여 다른 단어로 간주되는 경우가 있습니다.   \n",
    "이러한 경우, 기계학습 전에 미리 같은 표현으로 통일시켜주어 기계의 연산량을 줄여줄 수 있습니다.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "shaped-championship",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "정규화 사전의 수:  120\n"
     ]
    }
   ],
   "source": [
    "contractions = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
    "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
    "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
    "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
    "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
    "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
    "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
    "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
    "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
    "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
    "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
    "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
    "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
    "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
    "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
    "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
    "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
    "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
    "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
    "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
    "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
    "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
    "                           \"you're\": \"you are\", \"you've\": \"you have\"}\n",
    "\n",
    "print(\"정규화 사전의 수: \", len(contractions))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "south-statement",
   "metadata": {},
   "source": [
    "## 불용어(stopwords)\n",
    "일반적으로 텍스트에는 자주 등장하지만 자연어 처리를 할 때 실질적으로 도움이 되지 않는 단어들이 존재합니다.  \n",
    "NLRK에서 제공하는 불용어 리스트를 참조해 샘플에서 불용어를 제거하겠습니다.  \n",
    "  \n",
    "NLTK에서 미리 정의하여 제공하고 있는 불용어는 총 179개 입니다. 이 외에도\n",
    "* 모든 영어문자 소문자로 만들기\n",
    "* 섞여 있는 hrml 태그 제거\n",
    "* 정규표현식으로 각종 특수문자 제거하기 \n",
    "등의 과정으로 필요한 내용만 학습시킬 수 있도록 데이터를 정제하겠습니다. \n",
    "\n",
    "**이때, text의 불용어는 제거하되, 기사의 짧은 헤드라인이 자연스러운 문장이 되기 위해서는 headlines에는 불용어를 남기는 것이 적절하다고 판단했습니다.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "behavioral-register",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "불용어 개수 : 179\n",
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
     ]
    }
   ],
   "source": [
    "print('불용어 개수 :', len(stopwords.words('english') ))\n",
    "print(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "liked-quebec",
   "metadata": {},
   "outputs": [],
   "source": [
    "#- 데이터 전처리 함수\n",
    "\n",
    "def preprocess_sentence(sentence, remove_stopwords=True):\n",
    "    sentence = sentence.lower() # 텍스트 소문자화\n",
    "    sentence = BeautifulSoup(sentence, \"lxml\").text # <br />, <a href = ...> 등의 html 태그 제거\n",
    "    sentence = re.sub(r'\\([^)]*\\)', '', sentence) # 괄호로 닫힌 문자열 (...) 제거 Ex) my husband (and myself!) for => my husband for\n",
    "    sentence = re.sub('\"','', sentence) # 쌍따옴표 \" 제거\n",
    "    sentence = ' '.join([contractions[t] if t in contractions else t for t in sentence.split(\" \")]) # 약어 정규화\n",
    "    sentence = re.sub(r\"'s\\b\",\"\", sentence) # 소유격 제거. Ex) roland's -> roland\n",
    "    sentence = re.sub(\"[^a-zA-Z]\", \" \", sentence) # 영어 외 문자(숫자, 특수문자 등) 공백으로 변환\n",
    "    sentence = re.sub('[m]{2,}', 'mm', sentence) # m이 3개 이상이면 2개로 변경. Ex) ummmmmmm yeah -> umm yeah\n",
    "    \n",
    "    #- 불용어 제거 (Text)\n",
    "    if remove_stopwords:\n",
    "        tokens = ' '.join(word for word in sentence.split() if not word in stopwords.words('english') if len(word) >1)\n",
    "    else:\n",
    "        tokens = ' '.join(word for word in sentence.split() if len(word) > 1)\n",
    "    return tokens   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "naughty-aircraft",
   "metadata": {},
   "source": [
    "## 훈련 데이터 전체에 대한 전처리\n",
    "* text 는 불용어를 제거하고, headlines는 불용어를 제거하지 않을 것이므로 따로 호출해서 진행하겠습니다. \n",
    "* 멀티프로세싱을 활용하여 별도의 프로세스를 생성하여 병렬처리하면 CPU 수에 비례하여 획기적으로 소요시간을 줄일 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "annual-ballot",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "184.33993268013  seconds\n",
      "['saurav kant alumnus upgrad iiit pg program machine learning artificial intelligence sr systems engineer infosys almost years work experience program upgrad degree career support helped transition data scientist tech mahindra salary hike upgrad online power learning powered lakh careers'\n",
      " 'kunal shah credit card bill payment platform cred gave users chance win free food swiggy one year pranav kaushik delhi techie bagged reward spending cred coins users get one cred coin per rupee bill paid used avail rewards brands like ixigo bookmyshow ubereats cult fit'\n",
      " 'new zealand defeated india wickets fourth odi hamilton thursday win first match five match odi series india lost international match rohit sharma captaincy consecutive victories dating back march match witnessed india getting seventh lowest total odi cricket history'\n",
      " ...\n",
      " 'according reports new version science fiction film matrix development michael jordan reportedly play lead role film screenwriter zak penn talks write script film reports added actor keanu reeves starred original film followed two sequels'\n",
      " 'new music video shows rapper snoop dogg aiming toy gun clown character parodying us president donald trump video also shows tv airing news conference headline ronald klump wants deport doggs airing live clown house video remixed version song lavender'\n",
      " 'madhesi morcha alliance seven political parties withdrawn support pm pushpa kamal dahal led nepal government failed meet seven day ultimatum fulfil demands including endorsement revised constitution amendment bill morcha seats parliament despite withdrawal support immediate threat government']\n",
      "5.7809834480285645  seconds\n",
      "['upgrad learner switches to career in ml al with salary hike'\n",
      " 'delhi techie wins free food from swiggy for one year on cred'\n",
      " 'new zealand end rohit sharma led india match winning streak' ...\n",
      " 'the matrix film to get reboot reports'\n",
      " 'snoop dogg aims gun at clown dressed as trump in new video'\n",
      " 'madhesi morcha withdraws support to nepalese government']\n"
     ]
    }
   ],
   "source": [
    "import multiprocessing as mp   # 멀티 프로세싱으로 전처리 속도를 획기적으로 줄여봅시다\n",
    "from multiprocessing import Pool\n",
    "import numpy as np\n",
    "import time\n",
    "from functools import partial  # map을 할 때 함수에 여러 인자를 넣어줄 수 있도록 합니다\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "# num_cores 만큼 쪼개진 데이터를 전처리하여 반환합니다\n",
    "def appendTexts(sentences, remove_stopwords):\n",
    "    texts = []\n",
    "    for s in sentences:\n",
    "        texts += preprocess_sentence(s, remove_stopwords),\n",
    "    return texts\n",
    "\n",
    "def preprocess_data(data, remove_stopwords=True):\n",
    "    start_time = time.time()\n",
    "    num_cores = mp.cpu_count()  # 컴퓨터의 코어 수를 구합니다\n",
    "\n",
    "    text_data_split = np.array_split(data, num_cores)  # 코어 수만큼 데이터를 배분하여 병렬적으로 처리할 수 있게 합니다\n",
    "    pool = Pool(num_cores)\n",
    "\n",
    "    processed_data = np.concatenate(pool.map(partial(appendTexts, remove_stopwords=remove_stopwords), text_data_split))  # 각자 작업한 데이터를 하나로 합쳐줍니다\n",
    "    pool.close()\n",
    "    pool.join()\n",
    "    print(time.time() - start_time, \" seconds\")\n",
    "    return processed_data\n",
    "\n",
    "clean_text = preprocess_data(data['text'])  # 클라우드 기준으로 3~4분 정도 소요 됩니다\n",
    "print(clean_text)\n",
    "\n",
    "clean_headlines = preprocess_data(data['headlines'], remove_stopwords=False) # 클라우드 기준 1분정도 소요됩니다.\n",
    "print(clean_headlines)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interstate-fellow",
   "metadata": {},
   "source": [
    "* headlines에 전처리 함수 호출 시, 불용어 제거를 수행하지 않는다는 의미에서 두번째 인자로 False를 넣어주었습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "aerial-section",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['text'] = clean_text\n",
    "data['headlines'] = clean_headlines\n",
    "\n",
    "#- 빈 값을 Null 값으로 반환\n",
    "\n",
    "data.replace('', np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "characteristic-bunch",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "headlines    0\n",
       "text         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "committed-consistency",
   "metadata": {},
   "source": [
    "* 정제과정에서 모든 단어가 제거되어 빈 샘플이 생길 수 있기 때문에 확인해야 합니다. \n",
    "* NULL 값이 없음을 확인했습니다. \n",
    "* 아래는 만약 Null 값이 있는 경우, Null 값을 제거하는 코드입니다. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controlling-blink",
   "metadata": {},
   "source": [
    "data.dropna(axis=0, inplace=True)\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sensitive-vaccine",
   "metadata": {},
   "source": [
    "## 훈련데이터와 테스트데이터 나누기\n",
    "학습을 진행하기 위해서는 학습에 사용할 데이터의 크기를 결정하고, 문장의 시작과 끝을 표시해 줘야합니다.  \n",
    "샘플의 최대 길이를 정하기 위해, text와 headlines의 최소, 최대, 평균 길이를 구하고 길이분포를 시각화해보겠습니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "powerful-clear",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "텍스트의 최소 길이 : 1\n",
      "텍스트의 최대 길이 : 60\n",
      "텍스트의 평균 길이 : 35.09968483123221\n",
      "헤드라인의 최소 길이 : 1\n",
      "헤드라인의 최대 길이 : 16\n",
      "헤드라인의 평균 길이 : 9.299532330215534\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAca0lEQVR4nO3dfXRV9Z3v8fcnEYNYKlIyXCqlcd36kIapOmb6cHWmRVFo6xLvXWhltb1UU5noNNO5cqdRs3pb1xqYMlP7MLSLDA5cXFMn6rUPcrs6BYFoL07HabTaCrHVOqViVaKAdXCgGL73j7Ohh5gQcnKy9845n9daZ+Xsh3P2F+XHJ7+9f/u3FRGYmZnlTU3WBZiZmQ3GAWVmZrnkgDIzs1xyQJmZWS45oMzMLJccUGZmlksOqBRI+qWkuWN8jAZJIemEZPkBSZ9M3n9U0saxPL6ZWbk5oKpARNwZEZdmXYdZliT9e9HrkKT/KFr+aAnf9wFJO8eiVis4IesCzMzSEBFvOvxe0i+BT0bEpuwqsuG4B5WecyX9RNIrku6WNBFA0mWSHpO0V9I/S3rX4Q9IuknSLyS9Kmm7pP9atK1W0hclvSTpGeDDQx1Y0ickbS1aDkmtkp5Kjvt1SSrafq2kXkl7JG2Q9PZkvSR9WdIuSb+R9FNJs8v838ksVZJqitray5LukTQ12bZK0jeL9l0habOkk4F/At5a1At7a1Z/hkrlgErPVcB84HTgXcAnJJ0HrAX+BHgL8HfAekl1yWd+AfwRcApwK/ANSTOSbdcBlwHnAc3AwhHWcxnwh0ktVwHzACQtAG4B/htQD/w/oCv5zKXAHwNnJjVdBbw8wuOa5U0bcAXwfuCtwB7g68m2pcDvJ7/k/RHQAiyOiH3AB4FfR8Sbktev0y+9sjmg0vO3EfHriNgN/F/gXGAJ8HcR8XBE9EfEHcAB4L0AEfF/ks8cioi7gaeAdyffdxXwlYh4NvnOvxphPV+IiL0R8SugO6kHoBX4q4jojYjXgeUUen9vBw4Ck4GzASX7PF/KfwyzHGkFOiJiZ0QcAD4PLJR0QkS8Bnwc+BLwDaAtInzdKSUOqPS8UPT+NeBNwNuBpclptr2S9gJvo/BbHJL+e9Hpv73AbGBa8h1vBZ4t+s4dZaiHpKavFh1zNyDgtIjYAnyNwm+XuyStlvTmER7XLG/eDny76O98L9APTAeIiIeBZyi0g3uyKrIaOaCy9SywLCKmFL0mRURX0mO5HfgU8JaImAI8QaGRADxPIcwOm1XGmv5kQE0nRcQ/A0TE30bE+cA7KZzq+4syHdcsK88CHxzwd35iRDwHIOlPgTrg18Bnij7nR0GMMQdUtm4HWiW9JxmAcLKkD0uaDJxMoQH0AUi6hkIP6rB7gD+TNFPSqcBNZaqpE7hZUlNy3FMkXZm8/8Ok1gnAPmA/cKhMxzXLSiewrGgwUH1yLRZJZwJ/CXyMwqm+z0g6N/nci8BbJJ2SfsnVwQGVoYjooTDY4WsULsw+DXwi2bYduA34IYWG8PvAQ0Ufvx3YADwOPAp8q0w1fRtYAdwl6TcUem0fTDa/OTnuHgqnFF8G/qYcxzXL0FeB9cBGSa8C/wK8J7np/RvAioh4PCKeojCA6B8k1UXEkxQGED2TnB70KL4ykx9YaGZmeeQelJmZ5ZIDyszMcskBZWZmueSAMjOzXEp1sthp06ZFQ0NDmoc0GzOPPPLISxFRn/Zx3Y6s0gzVllINqIaGBnp6etI8pNmYkTTS2TvKwu3IKs1Qbcmn+MzMLJccUGZmlksOKDMzyyUHlJmZ5ZIDyszMcskBZWZmuTRsQElaK2mXpCcGrG+T9KSkbZL+euxKtOM1b948ampqkERNTQ3z5s3LuiQbQNIUSfcmbadX0vskTZV0v6Snkp+nZl1ntevq6mL27NnU1tYye/Zsurq6si6pKh1PD2odML94haQ5wALgnIhoAr5Y/tJsJObNm8fGjRtpbW1l7969tLa2snHjRodU/nwV+H5EnA2cQ+HprTcBmyPiDGAz5Xu2l5Wgq6uLjo4OVq5cyf79+1m5ciUdHR0OqSxExLAvoAF4omj5HmDu8Xy2+HX++eeHjQ1Jcf311x+17vrrrw9JGVVU+YCeGMHff+AU4N9IHnNTtP5nwIzk/QzgZ8f6HrejsdXU1BRbtmw5at2WLVuiqakpo4oq31Bt6bieByWpAfhuRMxOlh8D7qPQs9oP/M+I+NEQn10CLAGYNWvW+Tt2ZHLzfcWTxN69eznllN893POVV15hypQpHM//Yxs5SY9ERPMI9j8XWA1sp9B7egT4NPBcRExJ9hGw5/By0WfdjlJSW1vL/v37mTBhwpF1Bw8eZOLEifT392dYWeUaqi2VOkjiBGAq8F7gL4B7kob1BhGxOiKaI6K5vj71acuqhiRuvvnmo9bdfPPNDPG/xbJxAvAHwKqIOA/Yx4DTeclvk2/4jcLtKD2NjY1s3br1qHVbt26lsbExo4qqV6kBtRP4VtI7+1fgEDCtfGXZSF1yySWsWrWKG264gVdeeYUbbriBVatWcckll2Rdmv3OTmBnRDycLN9LIbBelDQDIPm5K6P6DOjo6KClpYXu7m4OHjxId3c3LS0tdHR0ZF1a1Sl1stjvAHOAbklnAicCL5WrKBu5DRs2MG/ePDo7O1m1ahWSuPTSS9mwYUPWpVkiIl6Q9KyksyLiZ8DFFE73bQcWA19Ift6XYZlVb9GiRQC0tbXR29tLY2Mjy5YtO7Le0jNsQEnqAj4ATJO0E/gcsBZYmww9/y2wOHyhI3MOo3GhDbhT0onAM8A1FM5k3COpBdgBXJVhfUYhpBxI2Rs2oCJiqP9LHytzLWYVLyIeAwYbWHFxyqWY5Z5nkjAzs1xyQJmZWS45oMzMLJccUGZmlksOKDMzy6VS74OyHBps1giP/jez8co9qApRHE533XXXoOvNzMYTB1SFiQg+8pGPuOdkZuOeA6qCFPecBls2MxtPHFAV5Oqrrz7mspkdHz9RNx8cUBVGEnfffbevPZmVyE/UzQ8HVIUovuZU3HPytSizkVm2bBlr1qxhzpw5TJgwgTlz5rBmzRqWLVuWdWlVx8PMK4jDyGz0ent7ufDCC49ad+GFF9Lb25tRRdXLPSgzsyKNjY3ceuutR12DuvXWW/1E3Qw4oMzMisyZM4cVK1Zw7bXX8uqrr3LttdeyYsUK5syZk3VpVccBZWZWpLu7m/b2dtauXcvkyZNZu3Yt7e3tdHd3Z11a1fE1KDOzIr29vcyYMYPt27cTEWzfvp0ZM2b4GlQG3IMyMyty0kknsWnTJlpbW9m7dy+tra1s2rSJk046KevSqo4DysysyL59+5g8eTJXXnklkyZN4sorr2Ty5Mns27cv69KqzrABJWmtpF2Snhhk21JJIWna2JRnIyHpDS8zG7nbbruNtrY2Jk6cSFtbG7fddlvWJVWl4+lBrQPmD1wp6W3ApcCvylyTlWCoMHJImY2MJNrb29m2bRuHDh1i27ZttLe3uy1lYNiAiogfALsH2fRl4DOA7w7NkYg48jKzkZs0aRJ79uyhoaGBp59+moaGBvbs2cOkSZOyLq3qlDSKT9IC4LmIeHy43yokLQGWAMyaNauUw5mZpWbfvn1MmzaNHTt28I53vANJTJs2jZdeeinr0qrOiAdJSJoE3AL8r+PZPyJWR0RzRDTX19eP9HBmZqmrr68/chYiIvC/XdkoZRTffwZOBx6X9EtgJvCopP9UzsKsNB4gYTZ6vb29XH755fT19XH55Zf7HqiMjPgUX0T8FPi9w8tJSDVHhPu/GYqIQUPJ16LMbLwaNqAkdQEfAKZJ2gl8LiLWjHVhNnIOI7PyOPvss1m/fv2RU3tnn302Tz75ZMZVVZ9hAyoiFg2zvaFs1ZhVuOSMw6tAP/B6RDRLmgrcDTQAvwSuiog9WdVovCGMHE7Z8EwSZumbExHnRkRzsnwTsDkizgA2J8uWA/fee2/WJVQ1B5RZ9hYAdyTv7wCuyK4UK7Zw4cKsS6hqDiizdAWwUdIjyT2CANMj4vnk/QvA9IEfkrREUo+knr6+vrRqrVqbNm066qb3TZs2ZV1SVfLjNszSdWFEPCfp94D7JR11cSMiQtIbRrtExGpgNUBzc7NHw4yxuXPnZl2C4R6UWaoi4rnk5y7g28C7gRclzQBIfu7KrkIrtmLFiqxLqGoOKLOUSDpZ0uTD7ylMtvwEsB5YnOy2GLgvmwptoPb29qxLqGo+xWeWnunAt5Mbqk8A/jEivi/pR8A9klqAHcBVGdZolhvuQZmlJCKeiYhzkldTRCxL1r8cERdHxBkRMTciBnt6gGXgs5/9bNYlVDUH1Dg12MMJj/dlZsOrqanh/e9/PzU1/mcyKz7FN04da1ojSZ72yGyUDh065NF8GfOvBmZmlksOKDMzyyUHlJmZ5ZIDyszMcskBZWZmueSAMjMbwvTpb5i311LkgDIzG8KLL76YdQlVzfdBmZkNovheQt/gng0HlJnZIBxK2Rv2FJ+ktZJ2SXqiaN3fSHpS0k8kfVvSlDGt0swsJUPNwuLZWdJ3PNeg1gHzB6y7H5gdEe8Cfg7cXOa6zMxScbzzVXpOy/QNG1AR8QNg94B1GyPi9WTxX4CZY1CbmdmYK360+8DXsbbb2CvHKL5rgX8qw/eYmZkdMaqAktQBvA7ceYx9lkjqkdTT19c3msOZmVkVKTmgJH0CuAz4aByjvxsRqyOiOSKa6+vrSz2cmZlVmZKGmUuaD3wGeH9EvFbekszMzI5vmHkX8EPgLEk7JbUAXwMmA/dLekxS5xjXaWZmVWbYHlRELBpk9ZoxqMXMzOwIz8VnZma55IAyM7NcckCZmVkuOaDMzCyXHFBmZpZLDigzM8slB5SZmeWSA8osRZJqJf1Y0neT5dMlPSzpaUl3Szox6xrN8sIBZZauTwO9RcsrgC9HxDuAPUBLJlWZ5ZADyiwlkmYCHwb+PlkWcBFwb7LLHcAVmRRnlkMOKLP0fIXCJMuHkuW3AHuLHv65EzhtsA/6sTVWjRxQZimQdBmwKyIeKeXzfmyNVaOSHrdhZiN2AXC5pA8BE4E3A18Fpkg6IelFzQSey7BGs1xxD8osBRFxc0TMjIgG4GpgS0R8FOgGFia7LQbuy6hEs9xxQJllqx24UdLTFK5J+VE2Zgmf4jNLWUQ8ADyQvH8GeHeW9ZjllXtQZmaWSw4oM6t4U6dORdKIX8CIPzN16tSM/7SVw6f4zKzi7dmzh4hI5ViHg81Gzz0oMzPLpWEDStJaSbskPVG0bqqk+yU9lfw8dWzLNDOzanM8Pah1wPwB624CNkfEGcDmZNnMzKxshg2oiPgBsHvA6gUUJrYET3BpZmZjoNRrUNMj4vnk/QvA9KF29CSXpUtz5JFHH5lZ3ox6FF9EhKQhh8dExGpgNUBzc3M6w2gqRJojj8Cjj8wsX0rtQb0oaQZA8nNX+UoyMzMrPaDWU5jYEjzBpZmZjYHjGWbeBfwQOEvSTkktwBeASyQ9BcxNls3MzMpm2GtQEbFoiE0Xl7kWM7MxEZ97M3z+lPSOZWXhqY7MrOLp1t+kOtVRfD6VQ1U8T3VkZma55IAyM7NcckCZmVkuOaDMzCyXHFBmZpZLHsVnZlUhram8Tj3VTx8qFweUmVW8UoeYS0p1Pkw7mgMqx9K8ufDI8czMcsIBlWNp3lwIvsHQzPLFgyTMzCyXHFBmZpZLDigzM8slB5SZmeWSA8osJZImSvpXSY9L2ibp1mT96ZIelvS0pLslnZh1rWZ54IAyS88B4KKIOAc4F5gv6b3ACuDLEfEOYA/Qkl2JZvnhgDJLSRT8e7I4IXkFcBFwb7L+DuCK9Kszyx8HlFmKJNVKegzYBdwP/ALYGxGvJ7vsBE4b5HNLJPVI6unr60utXrMsOaDMUhQR/RFxLjATeDdw9nF+bnVENEdEc319/ViWaJYbowooSf8judj7hKQuSRPLVZhZJYuIvUA38D5giqTDs7rMBJ7Lqi6zPCk5oCSdBvwZ0BwRs4Fa4OpyFWZWaSTVS5qSvD8JuATopRBUC5PdFgP3ZVKgWc6Mdi6+E4CTJB0EJgG/Hn1JZhVrBnCHpFoKvxzeExHflbQduEvSXwI/BtZkWaRZXpQcUBHxnKQvAr8C/gPYGBEbB+4naQmwBGDWrFmlHq5qpfUMG/BzbMZaRPwEOG+Q9c9QuB5lZkVGc4rvVGABcDrwVuBkSR8buJ8v7pYuIkp6lfrZ3bt3Z/wnNjP7ndEMkpgL/FtE9EXEQeBbwH8pT1lmZlbtRhNQvwLeK2mSCuehLqZwwdfMzGzUSg6oiHiYwt3vjwI/Tb5rdZnqMjOzKjeqUXwR8Tngc2WqxczM7AjPJGFmZrnkgDIzs1xyQJmZWS6NdiYJM7Nxbbib4YfafvieQxs7Digzq2qDBc1goeRASp9P8ZmZFRmqx5TmtGNW4B6UmdkgintMDqdsOKDMzAbhUMqeT/GZmVkuOaDMzCyXHFBmZpZLDigzM8slB5SZmeWSA8rMzHLJAWVmNsCCBQuIiCOvBQsWZF1SVfJ9UGZmA9x3332+DyoH3IMyMxvCOeeck3UJVc0BZWY2hMcffzzrEqqaA8rMzHJpVAElaYqkeyU9KalX0vvKVZiZWZZqa2t54IEHqK2tzbqUqjXaQRJfBb4fEQslnQhMKkNNZmaZ6+/v56WXXqK/vz/rUqpWyQEl6RTgj4FPAETEb4HflqcsM7PsLVy4MOsSqtpoTvGdDvQB/1vSjyX9vaSTB+4kaYmkHkk9fX19ozic2fgm6W2SuiVtl7RN0qeT9VMl3S/pqeTnqVnXapYHowmoE4A/AFZFxHnAPuCmgTtFxOqIaI6I5vr6+lEczmzcex1YGhHvBN4L/Kmkd1JoN5sj4gxgM4O0I8vGd77znaxLqGqjCaidwM6IeDhZvpdCYJnZICLi+Yh4NHn/KtALnAYsAO5IdrsDuCKTAu0NrrjiiqxLqGolB1REvAA8K+msZNXFwPayVGVW4SQ1AOcBDwPTI+L5ZNMLwPRB9vep8hRdc8011NXVAVBXV8c111yTcUXVabT3QbUBd0r6CXAusHzUFZlVOElvAr4J/HlE/KZ4W0QEEAM/41Pl6Vq3bh3Lly9n3759LF++nHXr1mVdUlUaVUBFxGNJo3lXRFwREXvKVZhZJZI0gUI43RkR30pWvyhpRrJ9BrArq/oMJBERPPjgg7z22ms8+OCDRITn5suAZ5IwS4kK/8KtAXoj4ktFm9YDi5P3i4H70q7NficiaGpqYv369dTX17N+/XqampoodG4tTQ4os/RcAHwcuEjSY8nrQ8AXgEskPQXMTZYtI3V1dUyZMuWoa1DFy5YeB5RZSiJia0QoOSV+bvL6XkS8HBEXR8QZETE3InZnXWs1O/PMM3nooYeYN28efX19zJs3j4ceeogzzzwz69Kqjp8HZWZW5Oc//zkXXHABGzZsoL6+nrq6Oi644AJ6enqyLq3qOKDMzIocOHCAjRs3MmnS76YWfe211zj55DdMlGNjzKf4zMyK1NXV0dnZedS6zs5OX4PKgHtQZmZFrrvuOtrb2wFobW2ls7OT9vZ2WltbM66s+jigzMyKrFy5EoBbbrmFpUuXUldXR2tr65H1lh4HlJnZACtXrnQg5YADapwa7q72Y233DYdmNh44oMYph4yZVTqP4jMzs1xyQJmZWS45oMzMLJccUGZmlksOKDMzyyUHlJmZ5ZIDyszMcskBZWZmueSAMjOzXBp1QEmqlfRjSd8tR0FWOklveJmZjVfl6EF9Gugtw/fYKBwOo5qaGjZt2kRNTc1R683MxptRzcUnaSbwYWAZcGNZKrKS1dTU0N/fD0B/fz+1tbUcOnQo46rMzEoz2h7UV4DPAEP+KyhpiaQeST19fX2jPJwdy8aNG4+5bGY2npQcUJIuA3ZFxCPH2i8iVkdEc0Q019fXl3o4Ow6XXnrpMZfNzMaT0fSgLgAul/RL4C7gIknfKEtVVpJDhw5RW1vL5s2bfXrPzMa9kgMqIm6OiJkR0QBcDWyJiI+VrTIbkcPPhzp06BBz5849Ek5+bpSZjVd+YGEFcRiZWSUpS0BFxAPAA+X4LjMzM/BMEmZmllMOKLOUSForaZekJ4rWTZV0v6Snkp+nZlmjWZ44oMzSsw6YP2DdTcDmiDgD2JwsmxkOKLPURMQPgN0DVi8A7kje3wFckWZNZnnmgDLL1vSIeD55/wIwfbCdPCOLVSMHVAVpa2tj4sSJSGLixIm0tbVlXZKNQBTuExj0XgHPyGLVyAFVIdra2ujs7GT58uXs27eP5cuX09nZ6ZDKvxclzQBIfu7KuB6z3HBAVYjbb7+dFStWcOONNzJp0iRuvPFGVqxYwe233551aXZs64HFyfvFwH0Z1mKWKw6oCnHgwAFaW1uPWtfa2sqBAwcyqsgGktQF/BA4S9JOSS3AF4BLJD0FzE2WzQwHVMWoq6ujs7PzqHWdnZ3U1dVlVJENFBGLImJGRExI5rFcExEvR8TFEXFGRMyNiIGj/MyqlufiqxDXXXcd7e3tQKHn1NnZSXt7+xt6VWZm44UDqkKsXLkSgFtuuYWlS5dSV1dHa2vrkfVmZuONA6qCrFy50oFkZhXD16DMzCyXHFBmZpZLDigzM8slB5SZmeWSA8rMzHLJAWVmZrlUckBJepukbknbJW2T9OlyFmZmZtVtNPdBvQ4sjYhHJU0GHpF0f0RsL1NtZmZWxUruQUXE8xHxaPL+VaAXOK1chZmZWXUryzUoSQ3AecDDg2zzk0DNzGzERh1Qkt4EfBP484j4zcDtfhKomZmVYlQBJWkChXC6MyK+VZ6SzMzMRjeKT8AaoDcivlS+kszMzEbXg7oA+DhwkaTHkteHylSXmZlVuZKHmUfEVkBlrMXMzOwIzyRhZma55IAyM7NcckCZmVkuOaDMzCyXHFBmZpZLDqgK0tXVxezZs6mtrWX27Nl0dXVlXZLZuOS2lA+jmc3ccqSrq4uOjg7WrFnDhRdeyNatW2lpaQFg0aJFGVdnNn64LeVIRKT2Ov/888PGRlNTU2zZsuWodVu2bImmpqaMKqp8QE+k2H7C7SgVbkvpG6otqbAtHc3NzdHT05Pa8apJbW0t+/fvZ8KECUfWHTx4kIkTJ9Lf359hZZVL0iMR0Zz2cd2OxpbbUvqGaku+BlUhGhsb2bp161Hrtm7dSmNjY0YV2UhImi/pZ5KelnRT1vVUM7el/HBAVYiOjg5aWlro7u7m4MGDdHd309LSQkdHR9al2TAk1QJfBz4IvBNYJOmd2VZVvdyW8sODJCrE4Yu3bW1t9Pb20tjYyLJly3xRd3x4N/B0RDwDIOkuYAGwPdOqqpTbUn74GpRZicp1DUrSQmB+RHwyWf448J6I+FTRPkuAJQCzZs06f8eOHaM9rFlu+BqU2TgWfjK1VSEHlFn2ngPeVrQ8M1lnVtUcUGbZ+xFwhqTTJZ0IXA2sz7gms8x5kIRZxiLidUmfAjYAtcDaiNiWcVlmmXNAmeVARHwP+F7WdZjliU/xmZlZLqU6zFxSH+DxsWNvGvBS1kVUgbdHROpD6tyOUuW2lI5B21KqAWXpkNSTxRxxZpXGbSlbPsVnZma55IAyM7NcckBVptVZF2BWIdyWMuRrUGZmlkvuQZmZWS45oMzMLJccUBVE0lpJuyQ9kXUtZuOV21F+OKAqyzpgftZFmI1z63A7ygUHVAWJiB8Au7Ouw2w8czvKDweUmZnlkgPKzMxyyQFlZma55IAyM7NcckBVEEldwA+BsyTtlNSSdU1m443bUX54qiMzM8sl96DMzCyXHFBmZpZLDigzM8slB5SZmeWSA8rMzHLJAWVmZrnkgDIzs1z6/16i1IR6xxEbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeuUlEQVR4nO3de7xXdZ3v8dc7UDNDwSQPcnFj0gVNUbdKJ+toJuLlhM4x0y6imXTRtDnmhNVJs5zoVNrYxcSRgcokxzSZpJBjmDmlAkpyMQ87xIBQTK7qRIKf+WN997j68dubxWL/bu738/FYj99an3X7/IDNZ6/1/a7vUkRgZmZWxqsanYCZmbUuFxEzMyvNRcTMzEpzETEzs9JcRMzMrDQXETMzK81FxKwLkpZLeneNz9EmKST1Tcv3SvpImv+ApLtreX6zneUiYtakIuLmiBjT6DzMuuMiYmZmpbmImHVvlKRHJW2Q9GNJrwaQdKqkBZLWS/qNpEM6d5A0UdIfJG2StETS6bl1fSR9XdKfJS0DTunqxJLOlXR/bjkkfUzS0nTe70hSbv2HJT0maZ2kWZL2T3FJulbSGkkbJS2UdHAP/zlZL+UiYta9M4GxwHDgEOBcSYcBU4CPAq8DbgBmSNot7fMH4B3AXsAXgR9KGpTWXQCcChwGtANn7GA+pwJHplzOBE4EkDQO+Czwd8BA4NfALWmfMcA7gTemnM4Ent3B85pV5SJi1r3rIuJPEbEW+DdgFDABuCEiHoyIrRExDdgMjAaIiH9N+7wUET8GlgJHpeOdCXwzIlakY35lB/OZFBHrI+KPwJyUD8DHgK9ExGMRsQX4R7KrqP2BF4F+wJsBpW1Wl/nDMKvkImLWvady8y8ArwX2By5Nt5TWS1oPDAX2A5B0Tu5W13rgYGCfdIz9gBW5Yz7ZA/mQcvqn3DnXAgIGR8QvgW8D3wHWSJosac8dPK9ZVS4iZjtuBXB1RPTPTa+JiFvSb/43AhcBr4uI/sAisv/QAVaTFZxOw3owp49W5LR7RPwGICKui4gjgJFkt7Uu66HzWi/nImK2424EPibp6NRovYekUyT1A/YAAngGQNJ5ZFcinW4FLpY0RNIAYGIP5fQ94HJJB6Xz7iXpvWn+yJTrLsDzwF+Al3rovNbLuYiY7aCImEfWQP5tYB3QAZyb1i0BvgH8FngaeCvw77ndbwRmAb8DHgZu76Gc7gC+CkyXtJHs6uektHrPdN51ZLfPngW+1hPnNZNfSmVmZmX5SsTMzEpzETEzs9JqVkQkvVrSQ5J+J2mxpC+m+HBJD0rqSE8A75riu6XljrS+LXesy1P8cUkn5uJjU6xDUk81UJqZWUG1vBLZDLwrIg4leyBqrKTRZI1/10bEgWQNfeen7c8H1qX4tWk7JI0EzgIOInty+Ltp6Ig+ZP3eTyLrtnh22tbMzOqkb60OHFmL/XNpcZc0BfAu4P0pPg24ErgeGJfmAW4Dvp3GBRoHTI+IzcATkjp4+enfjohYBiBpetp2SXd57bPPPtHW1raT387MrHeZP3/+nyNiYGW8ZkUEssHmgPnAgWRXDX8A1qdhGQBWAoPT/GDSk7wRsUXSBrJxiQYDD+QOm99nRUX86C7ymEA2VAXDhg1j3rx5O/fFzMx6GUlVR1eoacN6GldoFDCE7OrhzbU8Xzd5TI6I9ohoHzhwm0JqZmYl1aV3VkSsJxss7m1A/863uJEVl1VpfhVpOIi0fi+yh6L+K16xT1dxMzOrk1r2zhooqX+a3x04AXiMrJh0Dn89Hrgzzc9Iy6T1v0ztKjOAs1LvreHACOAhYC4wIvX22pWs8X1Grb6PmZltq5ZtIoOAaald5FXArRHxM0lLyIZm+DLwCHBT2v4m4Aep4XwtWVEgIhZLupWswXwLcGFEbAWQdBHZEBJ9gCkRsbiG38fMzCr0umFP2tvbww3rZmY7RtL8iGivjPuJdTMzK81FxMzMSnMRMTOz0lxEzMystJo+sW5mPadt4l1drls+6ZQ6ZmL2Ml+JmJlZaS4iZmZWmouImZmV5iJiZmaluYiYmVlpLiJmZlaai4iZmZXmImJmZqW5iJiZWWkuImZmVpqLiJmZleYiYmZmpbmImJlZaS4iZmZWmouImZmV5iJiZmaluYiYmVlpLiJmZlaai4iZmZXmImJmZqW5iJiZWWkuImZmVlrNioikoZLmSFoiabGkS1L8SkmrJC1I08m5fS6X1CHpcUkn5uJjU6xD0sRcfLikB1P8x5J2rdX3MTOzbdXySmQLcGlEjARGAxdKGpnWXRsRo9I0EyCtOws4CBgLfFdSH0l9gO8AJwEjgbNzx/lqOtaBwDrg/Bp+HzMzq1CzIhIRqyPi4TS/CXgMGNzNLuOA6RGxOSKeADqAo9LUERHLIuKvwHRgnCQB7wJuS/tPA06ryZcxM7Oq6tImIqkNOAx4MIUukvSopCmSBqTYYGBFbreVKdZV/HXA+ojYUhGvdv4JkuZJmvfMM8/0xFcyMzPqUEQkvRb4CfCpiNgIXA+8ARgFrAa+UescImJyRLRHRPvAgQNrfTozs16jby0PLmkXsgJyc0TcDhART+fW3wj8LC2uAobmdh+SYnQRfxboL6lvuhrJb29mZnVQsyKS2ixuAh6LiGty8UERsTotng4sSvMzgB9JugbYDxgBPAQIGCFpOFmROAt4f0SEpDnAGWTtJOOBO2v1fcxeydom3tXluuWTTqljJtZqankl8nbgQ8BCSQtS7LNkvatGAQEsBz4KEBGLJd0KLCHr2XVhRGwFkHQRMAvoA0yJiMXpeJ8Bpkv6MvAIWdEyM7M6qVkRiYj7ya4iKs3sZp+rgaurxGdW2y8ilpH13jIzswbwE+tmZlaai4iZmZXmImJmZqW5iJiZWWkuImZmVpqLiJmZleYiYmZmpbmImJlZaS4iZmZWmouImZmV5iJiZmaluYiYmVlpLiJmZlaai4iZmZXmImJmZqW5iJiZWWkuImZmVpqLiJmZleYiYmZmpbmImJlZadstIpLeK6lfmv+8pNslHV771MzMrNkVuRL5PxGxSdIxwLuBm4Dra5uWmZm1giJFZGv6PAWYHBF3AbvWLiUzM2sVRYrIKkk3AO8DZkrareB+Zmb2ClekGJwJzAJOjIj1wN7AZbVMyszMWsN2i0hEvACsAY5JoS3A0lomZWZmraFI76wrgM8Al6fQLsAPa5mUmZm1hiK3s04H3gM8DxARfwL6bW8nSUMlzZG0RNJiSZek+N6SZktamj4HpLgkXSepQ9Kj+W7Eksan7ZdKGp+LHyFpYdrnOknasa9vZmY7o0gR+WtEBBAAkvYoeOwtwKURMRIYDVwoaSQwEbgnIkYA96RlgJOAEWmaQOpGLGlv4ArgaOAo4IrOwpO2uSC339iCuZmZWQ8oUkRuTb2z+ku6APh/wI3b2ykiVkfEw2l+E/AYMBgYB0xLm00DTkvz44DvR+aBdL5BwInA7IhYGxHrgNnA2LRuz4h4IBW57+eOZWZmddB3extExNclnQBsBN4EfCEiZu/ISSS1AYcBDwL7RsTqtOopYN80PxhYkdttZYp1F19ZJV7t/BPIrm4YNmzYjqRuZmbd2G4RAUhFY4cKRydJrwV+AnwqIjbmmy0iIiRFmePuiIiYDEwGaG9vr/n5zMx6iy5vZ0naJGljlWmTpI1FDi5pF7ICcnNE3J7CT6dbUaTPNSm+Chia231IinUXH1IlbmZmddJlEYmIfhGxZ5WpX0Tsub0Dp55SNwGPRcQ1uVUzgM4eVuOBO3Pxc1IvrdHAhnTbaxYwRtKA1KA+BpiV1m2UNDqd65zcsczMrA4K3c5K3W2PIeuhdX9EPFJgt7cDHwIWSlqQYp8FJpE11p8PPEn2RDzATOBkoAN4ATgPICLWSvoSMDdtd1VErE3znwCmArsDP0+TmZnVyXaLiKQvAO8FOm9HTZX0rxHx5e72i4j7ga6e2zi+yvYBXNjFsaYAU6rE5wEHd5eHmZnVTpErkQ8Ah0bEXwAkTQIWAN0WETMze+Ur8pzIn4BX55Z3ww3YZmZGsSuRDcBiSbPJ2kROAB6SdB1ARFxcw/zMzKyJFSkid6Sp0721ScXMzFpNkSfWp21vGzMz652KDAV/qqRHJK3d0YcNzczsla3I7axvAn8HLEzdcM2sC20T7+py3fJJp9QxE7P6KNI7awWwyAXEzMwqFbkS+QdgpqRfAZs7gxVDmZiZWS9UpIhcDTxH9qzIrrVNx8zMWkmRIrJfRHhoETMz20aRNpGZksbUPBMzM2s5RYrIx4FfSPoPd/E1M7O8Ig8b9qtHImZm1nqKvk9kADCC3ECMEXFfrZIyM7PWUOR9Ih8BLiF7/ewCYDTwW+BdNc3MzMyaXpE2kUuAI4EnI+I44DBgfS2TMjOz1lCkiPwl90Kq3SLi98CbapuWmZm1giJtIisl9Qd+CsyWtI7s3ehmZtbLFemddXqavVLSHGAv4Bc1zcrMzFpCkaHg3yBpt85FoA14TS2TMjOz1lCkTeQnwFZJBwKTgaHAj2qalZmZtYQiReSliNgCnA58KyIuAwbVNi0zM2sFRYrIi5LOBsYDP0uxXWqXkpmZtYoiReQ84G3A1RHxhKThwA9qm5aZmbWCIr2zlgAX55afAL5ay6TMzKw1FLkSMTMzq6pmRUTSFElrJC3Kxa6UtErSgjSdnFt3uaQOSY9LOjEXH5tiHZIm5uLDJT2Y4j+W5LcumpnVWZdFRNIP0uclJY89FRhbJX5tRIxK08x0jpHAWcBBaZ/vSuojqQ/wHeAkYCRwdtoWsltq10bEgcA64PySeZqZWUndXYkcIWk/4MOSBkjaOz9t78BpqPi1BfMYB0yPiM2pzaUDOCpNHRGxLCL+CkwHxkkS2SjCt6X9pwGnFTyXmZn1kO4a1r8H3AMcAMwne1q9U6R4GRdJOgeYB1waEeuAwcADuW1WphjAior40cDrgPXp+ZXK7bchaQIwAWDYsGEl0zYzs0pdXolExHUR8RZgSkQcEBHDc1PZAnI98AZgFLAa+EbJ4+yQiJgcEe0R0T5w4MB6nNLMrFco0sX345IOBd6RQvdFxKNlThYRT3fOS7qRlx9eXEU2nEqnISlGF/Fngf6S+qarkfz2ZmZWJ0UGYLwYuBl4fZpulvTJMieTlB8u5XSgs+fWDOAsSbulhxlHAA8Bc4ERqSfWrmSN7zMiIoA5wBlp//HAnWVyMjOz8oq8T+QjwNER8TyApK+SvR73W93tJOkW4FhgH0krgSuAYyWNImtTWQ58FCAiFku6FVgCbAEujIit6TgXAbOAPmS31hanU3wGmC7py8AjwE3FvrKZmfWUIkVEwNbc8lb+tpG9qog4u0q4y//oI+Jq4Ooq8ZnAzCrxZWS9t8zMrEGKFJF/AR6UdEdaPg3/1m9mZhRrWL9G0r3AMSl0XkQ8UtOszMysJRS5EiEiHgYernEuZmbWYjwAo5mZleYiYmZmpXVbRNIgiHPqlYyZmbWWbotIelbjJUl71SkfMzNrIUUa1p8DFkqaDTzfGYyIi7vexczMeoMiReT2NJmZmf2NIs+JTJO0OzAsIh6vQ05mZtYiigzA+D+BBcAv0vIoSTNqnJeZmbWAIrezriQbo+pegIhYIKns+0TM7BWmbeJdXa5bPumUOmZijVDkOZEXI2JDReylWiRjZmatpciVyGJJ7wf6SBoBXAz8prZpmZlZKyhyJfJJ4CBgM3ALsBH4VA1zMjOzFlGkd9YLwOfSy6giIjbVPi0zM2sFRXpnHSlpIfAo2UOHv5N0RO1TMzOzZlekTeQm4BMR8WsASceQvajqkFomZmZmza9Im8jWzgICEBH3k70H3czMerkur0QkHZ5mfyXpBrJG9QDeR3pmxMzMerfubmd9o2L5itx81CAXMzNrMV0WkYg4rp6JmJlZ69luw7qk/sA5QFt+ew8Fb2ZmRXpnzQQeABbi4U7MzCynSBF5dUT875pnYmZmLadIF98fSLpA0iBJe3dONc/MzMyaXpErkb8CXwM+x8u9sgLwcPBmZr1ckSuRS4EDI6ItIoanabsFRNIUSWskLcrF9pY0W9LS9DkgxSXpOkkdkh7NPaOCpPFp+6WSxufiR0hamPa5TpJ27KubmdnOKlJEOoAXShx7KjC2IjYRuCciRgD3pGWAk4ARaZoAXA9Z0SF7PuVoshdjXdFZeNI2F+T2qzyXmZnVWJHbWc8DCyTNIRsOHth+F9+IuE9SW0V4HHBsmp9G9uT7Z1L8+xERwAOS+ksalLadHRFrASTNBsZKuhfYMyIeSPHvA6cBPy/wfczMrIcUKSI/TVNP2DciVqf5p4B90/xgYEVuu5Up1l18ZZV4VZImkF3hMGzYsJ1I38zM8oq8T2RaLU4cESGpLsOnRMRkYDJAe3u7h2wxM+shRZ5Yf4IqY2UVaVyv4mlJgyJidbpdtSbFVwFDc9sNSbFVvHz7qzN+b4oPqbK9mZnVUZGG9XbgyDS9A7gO+GHJ880AOntYjQfuzMXPSb20RgMb0m2vWcAYSQNSg/oYYFZat1HS6NQr65zcsczMrE6K3M56tiL0TUnzgS90t5+kW8iuIvaRtJKsl9Uk4FZJ5wNPAmemzWcCJ/NyT7Dz0rnXSvoSMDdtd1VnIzvwCbIeYLuTNai7Ud3MrM6K3M46PLf4KrIrkyLF5+wuVh1fZdsALuziOFOAKVXi84CDt5eHmZnVTpHeWfn3imwBlvPyFYSZmfViRa4o/F4RMzOrqsjtrN2A/8W27xO5qnZpmZlZKyhyO+tOYAMwn9wT62ZmZkWKyJCI8LhUZma2jSLPifxG0ltrnomZmbWcIlcixwDnpifXNwMi65V7SE0zMzOzplekiJxU8yzMzKwlFeni+2Q9EjEzs9ZTpE3EzMysKhcRMzMrzUXEzMxKcxExM7PSXETMzKw0FxEzMyvNRcTMzEpzETEzs9JcRMzMrLQiw56Y9SptE+/qct3ySafUMROz5ucrETMzK81FxMzMSnMRMTOz0lxEzMysNBcRMzMrzUXEzMxKcxExM7PSXETMzKy0hhQRScslLZS0QNK8FNtb0mxJS9PngBSXpOskdUh6VNLhueOMT9svlTS+Ed/FzKw3a+SVyHERMSoi2tPyROCeiBgB3JOWAU4CRqRpAnA9ZEUHuAI4GjgKuKKz8JiZWX000+2sccC0ND8NOC0X/35kHgD6SxoEnAjMjoi1EbEOmA2MrXPOZma9WqOKSAB3S5ovaUKK7RsRq9P8U8C+aX4wsCK378oU6ypuZmZ10qgBGI+JiFWSXg/MlvT7/MqICEnRUydLhWoCwLBhw3rqsGZmvV5DrkQiYlX6XAPcQdam8XS6TUX6XJM2XwUMze0+JMW6ilc73+SIaI+I9oEDB/bkVzEz69XqXkQk7SGpX+c8MAZYBMwAOntYjQfuTPMzgHNSL63RwIZ022sWMEbSgNSgPibFzMysThpxO2tf4A5Jnef/UUT8QtJc4FZJ5wNPAmem7WcCJwMdwAvAeQARsVbSl4C5aburImJt/b6GmZnVvYhExDLg0CrxZ4Hjq8QDuLCLY00BpvR0jmZmVozfbGhmTctvmWx+zfSciJmZtRgXETMzK81FxMzMSnMRMTOz0lxEzMysNBcRMzMrzUXEzMxKcxExM7PSXETMzKw0FxEzMyvNRcTMzEpzETEzs9JcRMzMrDQXETMzK81FxMzMSnMRMTOz0lxEzMysNBcRMzMrza/HtZbk16aaNQdfiZiZWWkuImZmVpqLiJmZleYiYmZmpblh3cx6ne46ZoA7Z+wIX4mYmVlpLiJmZlaai4iZmZXW8kVE0lhJj0vqkDSx0fmYmfUmLd2wLqkP8B3gBGAlMFfSjIhY0tjMDNx4adYbtHQRAY4COiJiGYCk6cA4wEXEzGrGw+68TBHR6BxKk3QGMDYiPpKWPwQcHREXVWw3AZiQFt8EPF7XRLu2D/DnRiexHc2eY7PnB86xJzR7ftD8Oe5sfvtHxMDKYKtfiRQSEZOByY3Oo5KkeRHR3ug8utPsOTZ7fuAce0Kz5wfNn2Ot8mv1hvVVwNDc8pAUMzOzOmj1IjIXGCFpuKRdgbOAGQ3Oycys12jp21kRsUXSRcAsoA8wJSIWNzitHdF0t9iqaPYcmz0/cI49odnzg+bPsSb5tXTDupmZNVar384yM7MGchExM7PSXEQaQNJQSXMkLZG0WNIljc6pGkl9JD0i6WeNzqUaSf0l3Sbp95Iek/S2RueUJ+nv09/vIkm3SHp1E+Q0RdIaSYtysb0lzZa0NH0OaMIcv5b+nh+VdIek/g1MsWqOuXWXSgpJ+zQit5RD1fwkfTL9OS6W9H974lwuIo2xBbg0IkYCo4ELJY1scE7VXAI81ugkuvFPwC8i4s3AoTRRrpIGAxcD7RFxMFnHj7MamxUAU4GxFbGJwD0RMQK4Jy030lS2zXE2cHBEHAL8f+DyeidVYSrb5oikocAY4I/1TqjCVCryk3Qc2Ygeh0bEQcDXe+JELiINEBGrI+LhNL+J7D+/wY3N6m9JGgKcAvxzo3OpRtJewDuBmwAi4q8Rsb6hSW2rL7C7pL7Aa4A/NTgfIuI+YG1FeBwwLc1PA06rZ06VquUYEXdHxJa0+ADZM2EN08WfI8C1wD8ADe2x1EV+HwcmRcTmtM2anjiXi0iDSWoDDgMebHAqlb5J9sPwUoPz6Mpw4BngX9Itt3+WtEejk+oUEavIftP7I7Aa2BARdzc2qy7tGxGr0/xTwL6NTKaADwM/b3QSlSSNA1ZFxO8anUsX3gi8Q9KDkn4l6cieOKiLSANJei3wE+BTEbGx0fl0knQqsCYi5jc6l270BQ4Hro+Iw4DnafxtmP+S2hXGkRW7/YA9JH2wsVltX2R9/pu237+kz5HdDr650bnkSXoN8FngC43OpRt9gb3JbqFfBtwqSTt7UBeRBpG0C1kBuTkibm90PhXeDrxH0nJgOvAuST9sbErbWAmsjIjOK7jbyIpKs3g38EREPBMRLwK3A/+9wTl15WlJgwDSZ4/c5uhpks4FTgU+EM33gNsbyH5h+F36uRkCPCzpvzU0q7+1Erg9Mg+R3WXY6cZ/F5EGSNX/JuCxiLim0flUiojLI2JIRLSRNQb/MiKa6rfoiHgKWCHpTSl0PM31CoA/AqMlvSb9fR9PEzX8V5gBjE/z44E7G5hLVZLGkt1efU9EvNDofCpFxMKIeH1EtKWfm5XA4enfabP4KXAcgKQ3ArvSA6MOu4g0xtuBD5H9hr8gTSc3OqkW9EngZkmPAqOAf2xsOi9LV0i3AQ8DC8l+1ho+LIakW4DfAm+StFLS+cAk4ARJS8muoCY1YY7fBvoBs9PPy/eaMMem0UV+U4ADUrff6cD4nrii87AnZmZWmq9EzMysNBcRMzMrzUXEzMxKcxExM7PSXETMzKw0FxF7xZL0XA2OOSrfHVvSlZI+vRPHe28agXhOz2RYOo/ljRx11lqXi4jZjhkF9OQzPecDF0TEcT14TLO6cRGxXkHSZZLmpvdRfDHF2tJVwI3p/Qp3S9o9rTsybbsgvctikaRdgauA96X4+9LhR0q6V9IySRd3cf6zJS1Mx/lqin0BOAa4SdLXKrYfJOm+dJ5Fkt6R4tdLmpfy/WJu++WSvpK2nyfpcEmzJP1B0sfSNsemY94l6XFJ35O0zf8Bkj4o6aF0rBuUvVemj6SpKZeFkv5+J/9K7JUiIjx5ekVOwHPpcwzZ0+Ii+8XpZ2TDyLeRDeY3Km13K/DBNL8IeFuanwQsSvPnAt/OneNK4DfAbmTjED0L7FKRx35kw6AMJBsE75fAaWndvWTvHKnM/VLgc2m+D9Avze+di90LHJKWlwMfT/PXAo+SPeE9EHg6xY8F/gIckPafDZyR238f4C3Av3V+B+C7wDnAEcDsXH79G/3366k5Jl+JWG8wJk2PkA1D8mZgRFr3REQsSPPzgTZlb83rFxG/TfEfbef4d0XE5oj4M9nghZVDqR8J3BvZYIydI9C+czvHnAucJ+lK4K2RvXcG4ExJD6fvchCQf5nZjPS5EHgwIjZFxDPAZr38JsCHImJZRGwFbiG7Eso7nqxgzJW0IC0fACwjGzLjW2kcq6YZddoaq2+jEzCrAwFfiYgb/iaYvctlcy60Fdi9xPErj7HTP1cRcZ+kd5K9GGyqpGuAXwOfBo6MiHWSpgL5V+525vFSRU4v5XKqHOeoclnAtIjY5s2Bkg4FTgQ+BpxJ9l4P6+V8JWK9wSzgw+n9LUgaLOn1XW0c2RsSN0k6OoXyr7XdRHabaEc8BPwPSftI6gOcDfyqux0k7U92G+pGsrdLHg7sSfbelA2S9gVO2sE8AI6SNDy1hbwPuL9i/T3AGZ1/Psrev75/6rn1qoj4CfB5mmvYfWsgX4nYK15E3C3pLcBvs1HZeQ74INlVQ1fOB26U9BLZf/gbUnwOMDHd6vlKwfOvljQx7Suy21/bG279WOAySS+mfM+JiCckPQL8HlgB/HuR81eYSzYi7oEpnzsqcl0i6fPA3anQvAhcCPwH2VskO3/xbPQ7zq1JeBRfsyokvTYinkvzE4FBEXFJg9PaKZKOBT4dEac2OBV7BfGViFl1p0i6nOxn5EmyXllmVsFXImZmVpob1s3MrDQXETMzK81FxMzMSnMRMTOz0lxEzMystP8EjGM+0BHzMG8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAf5UlEQVR4nO3df7hdVX3n8feHoGgVBSTmCYSYoFGLViNEwEd0UCoEsAU7FqFVIlJSKihOrU6wjjBU2jC2WG1tNJaUYBFkRCQjUYwpSJ0KJEBK+CFDCKEkhiQSIEFsNOEzf+x1ZXO59+Zk555z7sn9vJ5nP3fv7/61Frnkm7322mvJNhEREU3s1u0CRERE70oSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGINpH0ZG15WtIvatt/2OB6R0pa3Y6yRjS1e7cLELGrsv3ivnVJq4A/sv2D7pUoYvjlSSSiwyTtJmmWpAckPSrpKkn7lH1zJF1dO/YiSYslvQj4LrBf7Wlmv27VIaJPkkhE530EOBH4L8B+wGPAl8q+jwO/JemDkt4GnA7MsP1z4Fjgp7ZfXJafdr7oEc+W5qyIzjsTONv2agBJ5wP/IekDtp+S9AGqp47NwEf6josYiZJEIjrvFcA1kp6uxbYB44A1tm+RtBJ4OXBVNwoY0ao0Z0V03sPAsbb3qi0vsL0GQNJZwB7AT4FP1s7LkNsx4iSJRHTel4ELJb0CQNJYSSeU9VcDnwXeD3wA+KSkqeW8dcDLJL2080WOGFiSSETnfQFYAHxf0mbgZuAwSbsD/wxcZPvfbd8PfAr4mqQ9bP8EuAJYKenx9M6KkUCZlCoiIprKk0hERDSWJBIREY0liURERGNJIhER0dio+9hw33339aRJk7pdjIiInnLbbbf9zPbY/vFRl0QmTZrE0qVLu12MiIieIumhgeJpzoqIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjG2vbFuqQDgMuo5o02MNf2FyTtA3wDmASsAk6y/ZgkUU3WcxzwFPBB27eXa80APl0u/Vnb80v8EOBS4IXAQuAcZ4KUiOeYNOu6Ifevmn18h0oSu5p2PolsBT5u+yDgcOAsSQcBs4DFtqcAi8s2wLHAlLLMBOYAlKRzHnAYcChwnqS9yzlzgDNq501vY30iIqKftiUR22v7niRsbwbuBfYHTgDml8PmAyeW9ROAy1y5GdhL0njgGGCR7Y22HwMWAdPLvpfYvrk8fVxWu1ZERHRAR96JSJoEvAm4BRhne23Z9QhVcxdUCebh2mmrS2yo+OoB4gPdf6akpZKWbtiwYecqExERv9b2JCLpxcDVwMdsb6rvK08QbX+HYXuu7Wm2p40d+5yRjCMioqG2JhFJz6NKIJfb/lYJrytNUZSf60t8DXBA7fQJJTZUfMIA8YiI6JC2JZHS2+oS4F7bF9d2LQBmlPUZwLW1+KmqHA48UZq9rgeOlrR3eaF+NHB92bdJ0uHlXqfWrhURER3Qzkmp3gp8AFguaVmJfQqYDVwl6XTgIeCksm8hVffeFVRdfE8DsL1R0l8AS8pxF9jeWNY/zDNdfL9bloiI6JC2JRHbPwI0yO6jBjjewFmDXGseMG+A+FLg9TtRzIiI2An5Yj0iIhpLEomIiMaSRCIiorEkkYiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGIiGgsSSQiIhpLEomIiMaSRCIiorEkkYiIaKyd0+POk7Re0l212DckLSvLqr4ZDyVNkvSL2r4v1845RNJySSskfbFMhYukfSQtknR/+bl3u+oSEREDa+eTyKXA9HrA9vtsT7U9Fbga+FZt9wN9+2yfWYvPAc4AppSl75qzgMW2pwCLy3ZERHRQ25KI7ZuAjQPtK08TJwFXDHUNSeOBl9i+uUyfexlwYtl9AjC/rM+vxSMiokO69U7kbcA62/fXYpMl3SHph5LeVmL7A6trx6wuMYBxtteW9UeAcW0tcUREPMfuXbrvKTz7KWQtMNH2o5IOAb4t6XWtXsy2JXmw/ZJmAjMBJk6c2LDIERHRX8efRCTtDvwe8I2+mO0tth8t67cBDwCvBtYAE2qnTygxgHWluauv2Wv9YPe0Pdf2NNvTxo4dO5zViYgY1brRnPXbwE9s/7qZStJYSWPK+oFUL9BXluaqTZIOL+9RTgWuLactAGaU9Rm1eEREdEg7u/heAfwYeI2k1ZJOL7tO5rkv1N8O3Fm6/H4TONN230v5DwP/CKygekL5bonPBt4l6X6qxDS7XXWJiIiBte2diO1TBol/cIDY1VRdfgc6finw+gHijwJH7VwpIyJiZ+SL9YiIaCxJJCIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGujV2VkTsoEmzrht036rZx3ewJBHPyJNIREQ0liQSERGNJYlERERjSSIREdFYkkhERDSWJBIREY0liURERGNJIhER0ViSSERENNbO6XHnSVov6a5a7HxJayQtK8txtX3nSloh6T5Jx9Ti00tshaRZtfhkSbeU+DckPb9ddYmIiIFtN4lI+n1Je5b1T0v6lqSDW7j2pcD0AeKftz21LAvLdQ+imnv9deWcf5A0RtIY4EvAscBBwCnlWICLyrVeBTwGnN7/RhER0V6tPIn8D9ubJR0B/DZwCTBneyfZvgnY2GI5TgCutL3F9oPACuDQsqywvdL2L4ErgRMkCXgn8M1y/nzgxBbvFRERw6SVJLKt/DwemGv7OmBnmo7OlnRnae7au8T2Bx6uHbO6xAaLvwx43PbWfvEBSZopaamkpRs2bNiJokdERF0rSWSNpK8A7wMWStqjxfMGMgd4JTAVWAv8TcPr7BDbc21Psz1t7NixnbhlRMSo0EoyOAm4HjjG9uPAPsAnmtzM9jrb22w/DXyVqrkKYA1wQO3QCSU2WPxRYC9Ju/eLR0REB203idh+ClgPHFFCW4H7m9xM0vja5nuAvp5bC4CTJe0haTIwBbgVWAJMKT2xnk/18n2BbQM3AO8t588Arm1SpoiIaG67k1JJOg+YBrwG+CfgecA/A2/dznlXAEcC+0paDZwHHClpKmBgFfDHALbvlnQVcA9VkjrL9rZynbOpnoTGAPNs311u8d+BKyV9FriD6oV/RER0UCszG74HeBNwO4Dtn/Z1+R2K7VMGCA/6F73tC4ELB4gvBBYOEF/JM81hERHRBa28E/llaT4ygKQXtbdIERHRK1pJIleV3ll7SToD+AHVS/GIiBjlttucZfuvJb0L2ET1XuQzthe1vWQRETHitfJOhJI0kjgiIuJZBk0ikjZT3oP03wXY9kvaVqqIiOgJgyYR29vtgRUREaNbS81ZZdTeI6ieTH5k+462lioiRoxJs64bcv+q2cd3qCQxErUyFPxnqEbJfRmwL3CppE+3u2ARETHytfIk8ofAG23/J4Ck2cAy4LNtLFdERPSAVr4T+Snwgtr2HmSww4iIoLUnkSeAuyUtonon8i7gVklfBLD90TaWLyIiRrBWksg1ZelzY3uKEhERvaaVL9bnd6IgERHRe1rpnfVuSXdI2ihpk6TNkjZ1onARETGytdKc9bfA7wHLy2i+ERERQGu9sx4G7koCiYiI/lp5EvkksFDSD4EtfUHbFw91kqR5wLuB9bZfX2KfA34H+CXwAHCa7cclTQLuBe4rp99s+8xyziHApcALqSanOse2Je0DfAOYRDVL4km2H2uhPhERMUxaeRK5EHiK6luRPWvL9lwKTO8XWwS83vYbgP8HnFvb94DtqWU5sxafA5xBNe/6lNo1ZwGLbU8BFpftiIjooFaeRPbre5LYEbZvKk8Y9dj3a5s3A+8d6hqSxgMvsX1z2b4MOBH4LnAC1RzuUA3LciPVvOsREdEhrTyJLJR0dBvu/SGqZNBncukF9kNJbyux/YHVtWNWlxjAONtry/ojwLjBbiRppqSlkpZu2LBhmIofERGtJJE/Ab4n6RfD1cVX0p8DW4HLS2gtMNH2m4A/Bb4uqeX5SupzwA+yf67tabanjR07didKHhERda18bDis84pI+iDVC/ej+np82d5CeWlv+zZJDwCvphqja0Lt9Ak8M27XOknjba8tzV7rh7OcERGxfa08iSBpb0mHSnp739LkZpKmU/X2+l3bT9XiYyWNKesHUr1AX1maqzZJOlySgFOBa8tpC4AZZX1GLR4RER2y3ScRSX8EnEP1FLAMOBz4MfDO7Zx3BdWL730lrQbOo+qNtQewqMoJv+7K+3bgAkm/Ap4GzrS9sVzqwzzTxfe7PPMeZTZwlaTTgYeAk1qpcEREDJ9WemedA7yZ6i/8d0h6LfCX2zvJ9ikDhC8Z5NirgasH2bcUeE7vMNuPAkdtrxwREdE+rTRn/WdtQqo9bP8EeE17ixUREb2glSeR1ZL2Ar5N1Qz1GFXzUUREjHKt9M56T1k9X9INwEuB77W1VBER0RNaGQr+lZL26NukGqvqN9pZqIiI6A2tvBO5Gtgm6VXAXOAA4OttLVVERPSEVpLI07a3Au8B/s72J4Dx7S1WRET0glaSyK8knUL1Qd93Sux57StSRET0ilaSyGnAW4ALbT8oaTLwtfYWKyIiekErvbPuAT5a234QuKidhYqIiN7Q0thZERERA0kSiYiIxgZNIpK+Vn6e07niRERELxnqSeQQSfsBHypDwe9TXzpVwIiIGLmGerH+ZWAxcCBwG9XX6n1c4hERMYoN+iRi+4u2fxOYZ/tA25NrSxJIRES01MX3TyS9EXhbCd1k+872FisiInpBKwMwfhS4HHh5WS6X9JF2FywiIka+Vrr4/hFwmO3P2P4M1fS4Z7RycUnzJK2XdFctto+kRZLuLz/3LnFJ+qKkFZLulHRw7ZwZ5fj7Jc2oxQ+RtLyc88UyD3tERHRIK0lEwLba9jae/ZJ9KJcC0/vFZgGLbU+henE/q8SPBaaUZSYwB6qkQzU/+2HAocB5fYmnHHNG7bz+94qIiDZqJYn8E3CLpPMlnQ/czCBzpfdn+yZgY7/wCcD8sj4fOLEWv8yVm4G9JI0HjgEW2d5o+zFgETC97HuJ7ZttG7isdq2IiOiAVl6sXyzpRuCIEjrN9h07cc9xtteW9UeAcWV9f+Dh2nGrS2yo+OoB4s8haSbV0w0TJ07ciaJHjEyTZl3X7SLEKNXKHOvYvh24fbhvbtuSPNzXHeA+c6km1GLatGltv19ExGjRjbGz1pWmKMrP9SW+hmrWxD4TSmyo+IQB4hER0SHdSCILqCa4ovy8thY/tfTSOhx4ojR7XQ8cXYZe2Rs4Gri+7Nsk6fDSK+vU2rUiIqIDhmzOkjQG+IHtdzS5uKQrgCOBfSWtpuplNRu4StLpwEPASeXwhcBxwArgKarJsLC9UdJfAEvKcRfY7ntZ/2GqHmAvBL5bloiI6JAhk4jtbZKelvRS20/s6MVtnzLIrqMGONbAWYNcZx4wb4D4UuD1O1quiIgYHq28WH8SWC5pEfDzvqDtjw5+SkREjAatJJFvlSUidlHpIhxNtfKdyHxJLwQm2r6vA2WKiIge0coAjL8DLAO+V7anSlrQ5nJFREQPaKWL7/lUY1Y9DmB7GZmQKiIiaC2J/GqAnllPt6MwERHRW1p5sX63pD8AxkiaAnwU+Lf2FisiInpBK08iHwFeB2wBrgA2AR9rY5kiIqJHtNI76yngzyVdVG16c/uLFRERvaCV3llvlrQcuJPqo8N/l3RI+4sWEREjXSvvRC4BPmz7XwEkHUE1UdUb2lmwiIgY+Vp5J7KtL4EA2P4RsLV9RYqIiF4x6JOIpIPL6g8lfYXqpbqB9wE3tr9oEREx0g3VnPU3/bbPq61ndsCIiBg8iTSdQyQiIkaP7b5Yl7QX1ayBk+rHZyj4iIho5cX6QqoEshy4rbY0Iuk1kpbVlk2SPibpfElravHjauecK2mFpPskHVOLTy+xFZJmNS1TREQ000oX3xfY/tPhumEZTn4q/Hr63TXANVTT4X7e9l/Xj5d0EHAy1Vfz+wE/kPTqsvtLwLuA1cASSQts3zNcZY2IiKG1kkS+JukM4DtUQ58A1dznw3D/o4AHbD8kabBjTgCutL0FeFDSCqpRhQFW2F4JIOnKcmySSEREh7TSnPVL4HPAj3mmKWvpMN3/ZKquw33OlnSnpHmS9i6x/YGHa8esLrHB4s8haaakpZKWbtiwYZiKHhERrSSRjwOvsj3J9uSy7PR8IpKeD/wu8L9LaA7wSqqmrrU8t4txY7bn2p5me9rYsWOH67IREaNeK81ZK4Cn2nDvY4Hbba8D6PsJIOmrVM1nUL0zOaB23oQSY4h4RER0QCtJ5OfAMkk38Ox3IjvbxfcUak1ZksbbXls23wPcVdYXAF+XdDHVi/UpwK2AgCmSJlMlj5OBP9jJMkVExA5oJYl8uyzDRtKLqHpV/XEt/L8kTaX6Gn5V3z7bd0u6iuqF+VbgLNvbynXOBq4HxgDzbN89nOWMiIihtTKfyPzhvqntnwMv6xf7wBDHXwhcOEB8IdV3LBER0QWtfLH+IAOMlTUcL9cjIqK3tdKcNa22/gLg94F92lOciIjoJdvt4mv70dqyxvbfAse3v2gRETHStdKcdXBtczeqJ5NWnmAiImIX10oyqH/0t5Wq59RJbSlNRET0lFZ6Z2VekYiIGFArzVl7AP+V584nckH7ihUREb2gleasa4EnqAZe3LKdYyMiYhRpJYlMsD297SWJiIie08oovv8m6bfaXpKIiOg5rTyJHAF8sHy5voVq4EPbfkNbSxYRESNeK0nk2LaXIiIielIrXXwf6kRBIka7SbOu63YRInZYK+9EIiIiBpQkEhERjSWJREREY0kiERHRWNeSiKRVkpZLWiZpaYntI2mRpPvLz71LXJK+KGmFpDvrIwtLmlGOv1/SjG7VJyJiNOr2k8g7bE+13Tfx1Sxgse0pwOKyDVU34yllmQnMgSrpAOcBhwGHAuf1JZ6IiGi/bieR/k4A+uZ0nw+cWItf5srNwF6SxgPHAItsb7T9GLAIyBAtEREd0s3JpQx8X5KBr9ieC4yzvbbsfwQYV9b3Bx6unbu6xAaLP4ukmVRPMEycOHE46xARQ9jety+rZmeS1F7XzSRyhO01kl4OLJL0k/pO2y4JZqeVBDUXYNq0acNyzYiI6GJzlu015ed64BqqdxrrSjMV5ef6cvga4IDa6RNKbLB4RER0QFeeRCS9CNjN9uayfjRwAbAAmAHMLj+vLacsAM6WdCXVS/QnbK+VdD3wl7WX6UcD53awKhHPkuabGG261Zw1DrhGUl8Zvm77e5KWAFdJOh14iGfmcl8IHAesAJ4CTgOwvVHSXwBLynEX2N7YuWpERIxuXUkitlcCbxwg/ihw1ABxA2cNcq15wLzhLmNEtCYDR45uI62Lb0RE9JAkkYiIaCxJJCIiGuvmdyIRo07eH8SuJk8iERHRWJJIREQ0liQSERGNJYlERERjSSIREdFYkkhERDSWJBIREY0liURERGNJIhER0ViSSERENJYkEhERjSWJREREYx1PIpIOkHSDpHsk3S3pnBI/X9IaScvKclztnHMlrZB0n6RjavHpJbZC0qxO1yUiYrTrxii+W4GP275d0p7AbZIWlX2ft/3X9YMlHQScDLwO2A/4gaRXl91fAt4FrAaWSFpg+56O1CIiIjqfRGyvBdaW9c2S7gX2H+KUE4ArbW8BHpS0Aji07FtRptpF0pXl2CSRiIgO6eo7EUmTgDcBt5TQ2ZLulDRP0t4ltj/wcO201SU2WHyg+8yUtFTS0g0bNgxnFSIiRrWuJRFJLwauBj5mexMwB3glMJXqSeVvhutetufanmZ72tixY4frshERo15XZjaU9DyqBHK57W8B2F5X2/9V4Dtlcw1wQO30CSXGEPGIiOiAbvTOEnAJcK/ti2vx8bXD3gPcVdYXACdL2kPSZGAKcCuwBJgiabKk51O9fF/QiTpERESlG08ibwU+ACyXtKzEPgWcImkqYGAV8McAtu+WdBXVC/OtwFm2twFIOhu4HhgDzLN9d+eqERER3eid9SNAA+xaOMQ5FwIXDhBfONR5ERHRXvliPSIiGksSiYiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkQiIqKxJJGIiGisK2NnRUQATJp13ZD7V80+vkMliaaSRCJ2wPb+0osYbZJEIvpJohg5hvqzyFPKyJB3IhER0ViSSERENJYkEhERjSWJREREY0kiERHRWJJIREQ01vNJRNJ0SfdJWiFpVrfLExExmvT0dyKSxgBfAt4FrAaWSFpg+57ulixGsnwHsmvI1+4jQ08nEeBQYIXtlQCSrgROAJJERrkkikiS6YxeTyL7Aw/XtlcDh/U/SNJMYGbZfFLSfS1ce1/gZztdwpFhV6oLpD4jWc/URRe1dFjP1KcFO1uXVwwU7PUk0hLbc4G5O3KOpKW2p7WpSB21K9UFUp+RbFeqC+xa9WlXXXr9xfoa4IDa9oQSi4iIDuj1JLIEmCJpsqTnAycDC7pcpoiIUaOnm7Nsb5V0NnA9MAaYZ/vuYbr8DjV/jXC7Ul0g9RnJdqW6wK5Vn7bURbbbcd2IiBgFer05KyIiuihJJCIiGksS6afXh1GRNE/Sekl31WL7SFok6f7yc+9ulrFVkg6QdIOkeyTdLemcEu/V+rxA0q2S/r3U53+W+GRJt5TfuW+UTiI9QdIYSXdI+k7Z7uW6rJK0XNIySUtLrCd/1wAk7SXpm5J+IuleSW9pR32SRGpqw6gcCxwEnCLpoO6WaoddCkzvF5sFLLY9BVhctnvBVuDjtg8CDgfOKn8evVqfLcA7bb8RmApMl3Q4cBHweduvAh4DTu9eEXfYOcC9te1ergvAO2xPrX1P0au/awBfAL5n+7XAG6n+nIa/PrazlAV4C3B9bftc4Nxul6tBPSYBd9W27wPGl/XxwH3dLmPDel1LNU5az9cH+A3gdqoRFn4G7F7iz/odHMkL1XdZi4F3At8B1Kt1KeVdBezbL9aTv2vAS4EHKZ2n2lmfPIk820DDqOzfpbIMp3G215b1R4Bx3SxME5ImAW8CbqGH61Oaf5YB64FFwAPA47a3lkN66Xfub4FPAk+X7ZfRu3UBMPB9SbeVoZKgd3/XJgMbgH8qzY3/KOlFtKE+SSKjjKt/gvRUv25JLwauBj5me1N9X6/Vx/Y221Op/hV/KPDa7paoGUnvBtbbvq3bZRlGR9g+mKo5+yxJb6/v7LHftd2Bg4E5tt8E/Jx+TVfDVZ8kkWfbVYdRWSdpPED5ub7L5WmZpOdRJZDLbX+rhHu2Pn1sPw7cQNXks5ekvg9/e+V37q3A70paBVxJ1aT1BXqzLgDYXlN+rgeuoUryvfq7thpYbfuWsv1NqqQy7PVJEnm2XXUYlQXAjLI+g+rdwognScAlwL22L67t6tX6jJW0V1l/IdX7nXupksl7y2E9UR/b59qeYHsS1f8n/2L7D+nBugBIepGkPfvWgaOBu+jR3zXbjwAPS3pNCR1FNUXGsNcnX6z3I+k4qrbevmFULuxuiXaMpCuAI6mGfV4HnAd8G7gKmAg8BJxke2OXitgySUcA/wos55l2909RvRfpxfq8AZhP9bu1G3CV7QskHUj1r/l9gDuA99ve0r2S7hhJRwJ/ZvvdvVqXUu5ryubuwNdtXyjpZfTg7xqApKnAPwLPB1YCp1F+7xjG+iSJREREY2nOioiIxpJEIiKisSSRiIhoLEkkIiIaSxKJiIjGkkRilybpyTZcc2rpCt63fb6kP9uJ6/1+GWX1huEpYeNyrJK0bzfLEL0nSSRix00FjtveQTvgdOAM2+8YxmtGdESSSIwakj4haYmkO2tzeUwqTwFfLXN8fL98TY6kN5djl0n6nKS7ykgGFwDvK/H3lcsfJOlGSSslfXSQ+59S5qu4S9JFJfYZ4AjgEkmf63f8eEk3lfvcJeltJT5H0lLV5iQp8VWS/qpvPgxJB0u6XtIDks4sxxxZrnmdqnlzvizpOX8PSHq/qrlPlkn6Shk4coykS0tZlkv6bzv5RxK7gm4PWZwlSzsX4Mny82hgLtVw5btRDV3+dqph87cCU8txV1F9ZQ3VsBdvKeuzKcPrAx8E/r52j/OBfwP2oBop4FHgef3KsR/wH8BYqi+i/wU4sey7EZg2QNk/Dvx5WR8D7FnW96nFbgTeULZXAX9S1j8P3AnsWe65rsSPBP4TOLCcvwh4b+38fYHfBP5PXx2AfwBOBQ4BFtXKt1e3/3yzdH/Jk0iMFkeX5Q6qeTxeC0wp+x60vays3wZMKmNc7Wn7xyX+9e1c/zrbW2z/jGpQu/5DbL8ZuNH2BldDpV9OlcSGsgQ4TdL5wG/Z3lziJ0m6vdTldVQTqPXpG+ttOXCL7c22NwBb+sbtAm61vdL2NuAKqiehuqOoEsaSMmz9UVRJZyVwoKS/kzQd2ESMertv/5CIXYKAv7L9lWcFq3lK6mM7bQNe2OD6/a+x0/9v2b6pDEd+PHCppIupxhL7M+DNth+TdCnwggHK8XS/Mj1dK1P/sY76bwuYb/vc/mWS9EbgGOBM4CTgQztar9i15EkkRovrgQ+VuUmQtL+klw92sKuh2jdLOqyETq7t3kzVTLQjbgX+i6R9VU3DfArww6FOkPQKqmaor1INpHcw8BKquSGekDSOau6LHXVoGal6N+B9wI/67V8MvLfvv4+qeblfUXpu7Wb7auDTpTwxyuVJJEYF29+X9JvAj6sR5nkSeD/VU8NgTge+Kulpqr/wnyjxG4BZpannr1q8/1pJs8q5omr+2t4w3EcCn5D0q1LeU20/KOkO4CdUs3D+31bu388S4O+BV5XyXFPfafseSZ+mmuVvN+BXwFnAL6hmyuv7x+dznlRi9MkovhGDkPRi20+W9VlUc1Of0+Vi7ZT6sO1dLkrsIvIkEjG44yWdS/X/yUNUvbIioiZPIhER0VherEdERGNJIhER0ViSSERENJYkEhERjSWJREREY/8fU5BFBanEMtgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "text_len = [len(s.split()) for s in data['text']]\n",
    "headlines_len = [len(s.split()) for s in data['headlines']]\n",
    "\n",
    "print('텍스트의 최소 길이 : {}'.format(np.min(text_len)))\n",
    "print('텍스트의 최대 길이 : {}'.format(np.max(text_len)))\n",
    "print('텍스트의 평균 길이 : {}'.format(np.mean(text_len)))\n",
    "print('헤드라인의 최소 길이 : {}'.format(np.min(headlines_len)))\n",
    "print('헤드라인의 최대 길이 : {}'.format(np.max(headlines_len)))\n",
    "print('헤드라인의 평균 길이 : {}'.format(np.mean(headlines_len)))\n",
    "\n",
    "plt.subplot(1,2,1)\n",
    "plt.boxplot(headlines_len)\n",
    "plt.title('headlines')\n",
    "plt.subplot(1,2,2)\n",
    "plt.boxplot(text_len)\n",
    "plt.title('Text')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.title('headlines')\n",
    "plt.hist(headlines_len, bins = 40)\n",
    "plt.xlabel('length of samples')\n",
    "plt.ylabel('number of samples')\n",
    "plt.show()\n",
    "\n",
    "plt.title('Text')\n",
    "plt.hist(text_len, bins = 40)\n",
    "plt.xlabel('length of samples')\n",
    "plt.ylabel('number of samples')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faced-london",
   "metadata": {},
   "source": [
    "* 위에서부터 차례대로 그래프는 각각 요약과 실제 텍스트의 길이 분포, 요약본 샘플 길이별 개수, 실제 텍스트 셈플 길이별 개수를 나타냅니다. \n",
    "* text의 경우 최소 길이가 1, 최대 길이가 60 로 차이가 큰 편이지만, 평균 길이는 35로 시각화된 그래프로 봤을 땐 대체적으로 35 내외의 길이를 가진다는 것을 확인할 수 있습니다.\n",
    "* headlines의 경우 최소 길이가 1, 최대 길이가 16, 평균 길이가 9로 text 에 비해 상대적으로 길이가 매우 짧습니다. 그래프로 봤을 때에도 대체적으로 9 내외의 길이를 가집니다.\n",
    "\n",
    "* **위 정보로부터 text의 최대 길이와 headlines의 적절한 최대 길이를 임의로 각각 38, 10으로 정했습니다.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "accurate-height",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_max_len = 38\n",
    "headlines_max_len = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "least-dinner",
   "metadata": {},
   "outputs": [],
   "source": [
    "def below_threshold_len(max_len, nested_list):\n",
    "  cnt = 0\n",
    "  for s in nested_list:\n",
    "    if(len(s.split()) <= max_len):\n",
    "        cnt = cnt + 1\n",
    "  print('전체 샘플 중 길이가 %s 이하인 샘플의 비율: %s'%(max_len, (cnt / len(nested_list))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "hindu-institution",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플 중 길이가 38 이하인 샘플의 비율: 0.8163379422529483\n",
      "전체 샘플 중 길이가 10 이하인 샘플의 비율: 0.8162972753151687\n"
     ]
    }
   ],
   "source": [
    "below_threshold_len(text_max_len, data['text'])\n",
    "below_threshold_len(headlines_max_len,  data['headlines'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "preliminary-speaking",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "전체 샘플수 : 65295\n"
     ]
    }
   ],
   "source": [
    "data = data[data['text'].apply(lambda x: len(x.split()) <= text_max_len)]\n",
    "data = data[data['headlines'].apply(lambda x: len(x.split()) <= headlines_max_len)]\n",
    "print('전체 샘플수 :', (len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-reason",
   "metadata": {},
   "source": [
    "## 시작 토큰과 종료 토큰 추가하기\n",
    "* 디코더는 시작 토큰을 입력받아 문장을 생성하기 시작하고, 종료 토큰을 예측한 순간에 문장 생성을 멈추는 것입니다.\n",
    "* seq2seq 훈련을 위해서는 디코더의 입력과 레이블에 시작 토큰과 종료 토큰을 추가해야 합니다.\n",
    "* 시작 토큰은 sostoken, 종료 토큰은 eostoken이라 임의로 명명하고 앞뒤로 추가하겠습니다. \n",
    "* 디코더의 입력에 해당하면서 시작 토큰이 맨 앞에 있는 문장의 이름을 decoder_input, 디코더의 출력 또는 레이블에 해당되면서 종료 토큰이 맨 뒤에 붙는 문장의 이름을 decoder_target이라 이름을 정하겠습니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "changing-student",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "      <th>decoder_input</th>\n",
       "      <th>decoder_target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>new zealand end rohit sharma led india match w...</td>\n",
       "      <td>new zealand defeated india wickets fourth odi ...</td>\n",
       "      <td>sostoken new zealand end rohit sharma led indi...</td>\n",
       "      <td>new zealand end rohit sharma led india match w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aegon life iterm insurance plan helps customer...</td>\n",
       "      <td>aegon life iterm insurance plan customers enjo...</td>\n",
       "      <td>sostoken aegon life iterm insurance plan helps...</td>\n",
       "      <td>aegon life iterm insurance plan helps customer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>rahat fateh ali khan denies getting notice for...</td>\n",
       "      <td>pakistani singer rahat fateh ali khan denied r...</td>\n",
       "      <td>sostoken rahat fateh ali khan denies getting n...</td>\n",
       "      <td>rahat fateh ali khan denies getting notice for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>cong wins ramgarh bypoll in rajasthan takes to...</td>\n",
       "      <td>congress candidate shafia zubair ramgarh assem...</td>\n",
       "      <td>sostoken cong wins ramgarh bypoll in rajasthan...</td>\n",
       "      <td>cong wins ramgarh bypoll in rajasthan takes to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>up cousins fed human excreta for friendship wi...</td>\n",
       "      <td>two minor cousins uttar pradesh gorakhpur alle...</td>\n",
       "      <td>sostoken up cousins fed human excreta for frie...</td>\n",
       "      <td>up cousins fed human excreta for friendship wi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            headlines  \\\n",
       "2   new zealand end rohit sharma led india match w...   \n",
       "3   aegon life iterm insurance plan helps customer...   \n",
       "5   rahat fateh ali khan denies getting notice for...   \n",
       "9   cong wins ramgarh bypoll in rajasthan takes to...   \n",
       "10  up cousins fed human excreta for friendship wi...   \n",
       "\n",
       "                                                 text  \\\n",
       "2   new zealand defeated india wickets fourth odi ...   \n",
       "3   aegon life iterm insurance plan customers enjo...   \n",
       "5   pakistani singer rahat fateh ali khan denied r...   \n",
       "9   congress candidate shafia zubair ramgarh assem...   \n",
       "10  two minor cousins uttar pradesh gorakhpur alle...   \n",
       "\n",
       "                                        decoder_input  \\\n",
       "2   sostoken new zealand end rohit sharma led indi...   \n",
       "3   sostoken aegon life iterm insurance plan helps...   \n",
       "5   sostoken rahat fateh ali khan denies getting n...   \n",
       "9   sostoken cong wins ramgarh bypoll in rajasthan...   \n",
       "10  sostoken up cousins fed human excreta for frie...   \n",
       "\n",
       "                                       decoder_target  \n",
       "2   new zealand end rohit sharma led india match w...  \n",
       "3   aegon life iterm insurance plan helps customer...  \n",
       "5   rahat fateh ali khan denies getting notice for...  \n",
       "9   cong wins ramgarh bypoll in rajasthan takes to...  \n",
       "10  up cousins fed human excreta for friendship wi...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['decoder_input'] = data['headlines'].apply(lambda x : 'sostoken ' + x)\n",
    "data['decoder_target'] = data['headlines'].apply(lambda x : x + ' eostoken')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "promising-safety",
   "metadata": {},
   "source": [
    "인코더의 입력, 디코더의 입력과 레이블을 각각 Numpy 타입으로 저장하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "narrow-literacy",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input = np.array(data['text']) #- 인코더의 입력\n",
    "decoder_input = np.array(data['decoder_input']) #- 디코더의 입력\n",
    "decoder_target = np.array(data['decoder_target']) #-디코더의 레이블"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "invisible-pittsburgh",
   "metadata": {},
   "source": [
    "훈련 데이터와 테스트 데이터를 직접 분리하겠습니다.   \n",
    "아래 랜덤한 정수 시퀀스를 이용해서 데이터의 샘플 순서를 정의해주면 잘 섞인 샘플이 될 것입니다.   \n",
    "잘 섞인 데이터를 8:2의 비율로 훈련 데이터와 테스트 데이터로 분리하겠습니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "micro-bookmark",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18115 10309 39214 ... 35104 58166 52801]\n"
     ]
    }
   ],
   "source": [
    "indices = np.arange(encoder_input.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "print(indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "hybrid-discount",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input = encoder_input[indices]\n",
    "decoder_input = decoder_input[indices]\n",
    "decoder_target = decoder_target[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "aggressive-australian",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "테스트 데이터의 수 : 13059\n"
     ]
    }
   ],
   "source": [
    "n_of_val = int(len(encoder_input)*0.2)\n",
    "print('테스트 데이터의 수 :', n_of_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "continental-headquarters",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "훈련 데이터의 개수 : 52236\n",
      "훈련 레이블의 개수 : 52236\n",
      "테스트 데이터의 개수 : 13059\n",
      "테스트 레이블의 개수 : 13059\n"
     ]
    }
   ],
   "source": [
    "encoder_input_train = encoder_input[:-n_of_val]\n",
    "decoder_input_train = decoder_input[:-n_of_val]\n",
    "decoder_target_train = decoder_target[:-n_of_val]\n",
    "\n",
    "encoder_input_test = encoder_input[-n_of_val:]\n",
    "decoder_input_test = decoder_input[-n_of_val:]\n",
    "decoder_target_test = decoder_target[-n_of_val:]\n",
    "\n",
    "print('훈련 데이터의 개수 :', len(encoder_input_train))\n",
    "print('훈련 레이블의 개수 :', len(decoder_input_train))\n",
    "print('테스트 데이터의 개수 :', len(encoder_input_test))\n",
    "print('테스트 레이블의 개수 :', len(decoder_input_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "finnish-reservation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['sostoken oneplus announces exciting oneplus launch offers starting may',\n",
       "       'sostoken veteran actor dilip kumar diagnosed with pneumonia',\n",
       "       'sostoken centre slams un expert who criticised swachh bharat mission',\n",
       "       ...,\n",
       "       'sostoken us has lost mediator role in middle east turkish prez',\n",
       "       'sostoken chose computer science over becoming mayor twitter ceo',\n",
       "       'sostoken yr old computer built by steve jobs sells for crore'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_input_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-cinema",
   "metadata": {},
   "source": [
    "## 정수 인코딩 \n",
    "단어 집합(vocabulary) 만들기 및 정수 인코딩의 작업을 하겠습니다.   \n",
    "기계가 텍스트를 숫자로 처리할 수 있도록 훈련 데이터와 테스트 데이터의 단어들을 모두 정수로 바꾸어 주어야 합니다.   \n",
    "\n",
    "각 단어에 고유한 정수를 맵핑하는 작업이 필요한데,이 과정을 단어집합(Vocabulary)를 만든다고 표현합니다. \n",
    "\n",
    "* 훈련 데이터에 대한 단어집합을 만들어보겠습니다.  \n",
    "* 우선적으로 원문에 해당하는 encoder_input_train에 대한 단어 집합을 만들겠습니다. \n",
    "* Keras의 토크나이저를 사용하면, 입력된 훈련 데이터로부터 단어 집합을 만들 수 있습니다. \n",
    "\n",
    "### encoder_input_train 단어집합"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "complete-cleaning",
   "metadata": {},
   "outputs": [],
   "source": [
    "src_tokenizer = Tokenizer() #- 토크나이저 정의\n",
    "src_tokenizer.fit_on_texts(encoder_input_train) #- 입력된 데이터로부터 단어 집합 생성"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unlike-effects",
   "metadata": {},
   "source": [
    "단어 집합이 생성되는 동시에 각 단어에 고유한 정수가 부여되었습니다.  \n",
    "현재 생성된 단어 집합은 src_tokenizer.word_index에 저장되어 있습니다.   \n",
    "그런데 우리는 단어 집합에 있느 모든 단어를 사용하는 것이 아니라, 빈도수가 낮은 단어들을 훈련 데이터에서 제외하고 진행하려고 합니다. \n",
    "  \n",
    "* 등장 빈도수가 7회 미만은 단어들이 이 데이터에서 얼마만큼의 비중을 차지하는지 확인해보겠습니다.   \n",
    "* src_tokenizer.word_counts.items()에는 단어와 각 단어의 등장 빈도수가 저장돼 있는데, 이를 통해서 통계적인 정보를 얻을 수 있습니다.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "going-newton",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어 집합(vocabulary)의 크기 : 57367\n",
      "등장 빈도가 6번 이하인 희귀 단어의 수: 39449\n",
      "단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 17918\n",
      "단어 집합에서 희귀 단어의 비율: 68.76601530496626\n",
      "전체 등장 빈도에서 희귀 단어 등장 빈도 비율: 4.537751485166954\n"
     ]
    }
   ],
   "source": [
    "threshold = 7\n",
    "total_cnt = len(src_tokenizer.word_index) #- 단어의 수\n",
    "rare_cnt = 0 #- 등장 빈도수가 threshold보다 작은 단어의 개수를 카운트\n",
    "total_freq = 0 #- 훈련 데이터의 전체 단어 빈도수의 총 합 \n",
    "rare_freq = 0 #- 등장 빈도수가 threshold 보다 작은 단어의 등장 빈도수의 합 \n",
    "\n",
    "#- 단어와 빈도수의 쌍(pair)를 key와 value로 받는다\n",
    "\n",
    "for key, value in src_tokenizer.word_counts.items():\n",
    "    total_freq = total_freq + value\n",
    "    \n",
    "    #- 단어의 등장 빈도수가 threshold보다 작으면\n",
    "    if (value < threshold):\n",
    "        rare_cnt = rare_cnt + 1\n",
    "        rare_freq = rare_freq + value\n",
    "        \n",
    "print('단어 집합(vocabulary)의 크기 :', total_cnt)\n",
    "print('등장 빈도가 %s번 이하인 희귀 단어의 수: %s'%(threshold - 1, rare_cnt))\n",
    "print('단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 %s'%(total_cnt - rare_cnt))\n",
    "print(\"단어 집합에서 희귀 단어의 비율:\", (rare_cnt / total_cnt)*100)\n",
    "print(\"전체 등장 빈도에서 희귀 단어 등장 빈도 비율:\", (rare_freq / total_freq)*100)        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "annual-louis",
   "metadata": {},
   "source": [
    "* encoder_input_train에는 약 5만7천개의 단어가 있습니다.\n",
    "* 등장 빈도가 threshold 값은 7회 미만, 즉 6회 이하인 단어들은 단어 집합에서 70% 에 육박합니다. \n",
    "* 하지만 실제로 훈련 데이터에서 등장 빈도로 차지하는 비중은 상대적으로 적은 수치인 4.54% 밖에 되지 않습니다. \n",
    "* 이러한 단어들은 훈련에 큰 영향을 주지 않으면서 연산량을 차지하기 때문에, 등장 빈도가 6회 이하인 단어들은 정수 인코딩 과정에서 빼고, 훈련 데이터에서 제거하겠습니다. \n",
    "* 위에서 이를 제외한 단어 집합의 크기를 17863개로 계산했는데 이와 비슷한 값으로 어림잡아 1만7천으로 제한해보겠습니다. \n",
    "* 토크나이저 정의 시 num_words 값을 정해주면, 단어 집합의 크기를 제한할 수 있습니다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "regular-portsmouth",
   "metadata": {},
   "outputs": [],
   "source": [
    "src_vocab = 17000\n",
    "src_tokenizer = Tokenizer(num_words=src_vocab) # 단어 집합의 크기를 17,000으로 제한\n",
    "src_tokenizer.fit_on_texts(encoder_input_train) # 단어 집합 재생성."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bronze-borough",
   "metadata": {},
   "source": [
    "* texts_to_sequences()는 생성된 단어 집합에 기반하여 입력으로 주어진 텍스트 데이터의 단어들을 모두 정수로 변환하는 정수 인코딩을 수행합니다.\n",
    "\n",
    "* 현재 단어 집합의 크기를 17,000으로 제한했으니까 이제 17,000이 넘는 숫자들은 정수 인코딩 후에는 데이터에 존재하지 않습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "secondary-singer",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1991, 79, 1464, 5396, 1751, 583, 1991, 3913, 116, 4402, 4265, 4786, 1981, 1441, 11, 171, 260, 526, 199, 914, 1902, 1346, 4849, 5, 1309, 2611, 1051, 520, 191, 662, 7688, 520, 16650], [1843, 48, 4719, 206, 578, 72, 13229, 130, 8024, 3125, 14655, 10352, 185, 12581, 15557, 1, 66, 663, 441, 25, 156, 849, 7054, 10352, 1457, 3826, 825, 819, 1, 2, 14, 48, 185], [192, 392, 525, 3950, 12066, 361, 4556, 1287, 1134, 10007, 14656, 314, 508, 2221, 1, 30, 1956, 7373, 161, 4557, 3711, 1134, 314, 508, 7373, 9, 1, 2352, 689, 1752, 507, 254, 549]]\n"
     ]
    }
   ],
   "source": [
    "# 텍스트 시퀀스를 정수 시퀀스로 변환\n",
    "encoder_input_train = src_tokenizer.texts_to_sequences(encoder_input_train) \n",
    "encoder_input_test = src_tokenizer.texts_to_sequences(encoder_input_test)\n",
    "\n",
    "# 잘 진행되었는지 샘플 출력\n",
    "print(encoder_input_train[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ethical-haven",
   "metadata": {},
   "source": [
    "### decoder_input_train 단어집합\n",
    "* Summary 데이터에 대해서도 동일한 작업을 수행합니다.\n",
    "* 케라스의 토크나이저를 사용하여 decoder_input_train을 입력으로 전체 단어 집합과 각 단어에 대한 빈도수를 계산하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "rapid-stephen",
   "metadata": {},
   "outputs": [],
   "source": [
    "tar_tokenizer = Tokenizer()\n",
    "tar_tokenizer.fit_on_texts(decoder_input_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "apart-distance",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어 집합(vocabulary)의 크기 : 25873\n",
      "등장 빈도가 6번 이하인 희귀 단어의 수: 18325\n",
      "단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 7548\n",
      "단어 집합에서 희귀 단어의 비율: 70.82673056854635\n",
      "전체 등장 빈도에서 희귀 단어 등장 빈도 비율: 7.492913293784155\n"
     ]
    }
   ],
   "source": [
    "threshold = 7\n",
    "total_cnt = len(tar_tokenizer.word_index) # 단어의 수\n",
    "rare_cnt = 0 # 등장 빈도수가 threshold보다 작은 단어의 개수를 카운트\n",
    "total_freq = 0 # 훈련 데이터의 전체 단어 빈도수 총 합\n",
    "rare_freq = 0 # 등장 빈도수가 threshold보다 작은 단어의 등장 빈도수의 총 합\n",
    "\n",
    "# 단어와 빈도수의 쌍(pair)을 key와 value로 받는다.\n",
    "for key, value in tar_tokenizer.word_counts.items():\n",
    "    total_freq = total_freq + value\n",
    "\n",
    "    # 단어의 등장 빈도수가 threshold보다 작으면\n",
    "    if(value < threshold):\n",
    "        rare_cnt = rare_cnt + 1\n",
    "        rare_freq = rare_freq + value\n",
    "\n",
    "print('단어 집합(vocabulary)의 크기 :', total_cnt)\n",
    "print('등장 빈도가 %s번 이하인 희귀 단어의 수: %s'%(threshold - 1, rare_cnt))\n",
    "print('단어 집합에서 희귀 단어를 제외시킬 경우의 단어 집합의 크기 %s'%(total_cnt - rare_cnt))\n",
    "print(\"단어 집합에서 희귀 단어의 비율:\", (rare_cnt / total_cnt)*100)\n",
    "print(\"전체 등장 빈도에서 희귀 단어 등장 빈도 비율:\", (rare_freq / total_freq)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "indian-building",
   "metadata": {},
   "source": [
    "* 등장 빈도가 6회 이하인 단어들은 단어 집합에서 약 71%를 차지합니다.  \n",
    "* 이하는 encoder_input_train의 단어집합 생성 과정과 동일합니다.\n",
    "* 단어집합의 크기는 7000으로 정했습니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "numeric-romantic",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input\n",
      "input  [[1, 1201, 298, 6403, 1201, 224, 340, 3220, 97], [1, 2200, 184, 3112, 373, 3923, 9], [1, 198, 200, 207, 4935, 50, 2257, 3630, 823, 1155], [1, 2379, 3221, 1857, 747, 30, 4, 2755], [1, 3924, 1003, 142, 13, 140, 414, 768]]\n",
      "target\n",
      "decoder  [[1201, 298, 6403, 1201, 224, 340, 3220, 97, 2], [2200, 184, 3112, 373, 3923, 9, 2], [198, 200, 207, 4935, 50, 2257, 3630, 823, 1155, 2], [2379, 3221, 1857, 747, 30, 4, 2755, 2], [3924, 1003, 142, 13, 140, 414, 768, 2]]\n"
     ]
    }
   ],
   "source": [
    "tar_vocab = 7000\n",
    "tar_tokenizer = Tokenizer(num_words=tar_vocab) \n",
    "tar_tokenizer.fit_on_texts(decoder_input_train)\n",
    "tar_tokenizer.fit_on_texts(decoder_target_train)\n",
    "\n",
    "# 텍스트 시퀀스를 정수 시퀀스로 변환\n",
    "decoder_input_train = tar_tokenizer.texts_to_sequences(decoder_input_train) \n",
    "decoder_target_train = tar_tokenizer.texts_to_sequences(decoder_target_train)\n",
    "decoder_input_test = tar_tokenizer.texts_to_sequences(decoder_input_test)\n",
    "decoder_target_test = tar_tokenizer.texts_to_sequences(decoder_target_test)\n",
    "\n",
    "# 잘 변환되었는지 확인\n",
    "print('input')\n",
    "print('input ',decoder_input_train[:5])\n",
    "print('target')\n",
    "print('decoder ',decoder_target_train[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "understood-invite",
   "metadata": {},
   "source": [
    "* 전체 데이터에서 빈도수가 낮은 단어가 삭제되었다는 것은 빈도수가 낮은 단어만으로 구성되었던 샘플들은 이제 빈(empty) 샘플이 되었을 가능성이 있습니다.\n",
    "* 이 현상은 길이가 상대적으로 길었던 원문(Text)의 경우에는 문제가 별로 없겠지만, 애초에 평균 길이가 4밖에 되지 않았던 요약문(Summary)의 경우에는 이 현상이 굉장히 두드러졌을 가능성이 높습니다.\n",
    "* 요약문인 decoder_input에는 sostoken 또는 decoder_target에는 eostoken이 추가된 상태이고, 이 두 토큰은 모든 샘플에서 등장하므로 빈도수가 샘플 수와 동일하게 매우 높으므로 단어 집합 제한에도 삭제되지 않습니다. 그래서 이제 길이가 0이 된 요약문의 실제 길이는 1로 나올 것이라는 것을 주의해야 합니다. \n",
    "* 훈련 데이터와 테스트 데이터에 대해서 요약문의 길이가 1인 경우의 인덱스를 각각 drop_train과 drop_test에 라는 변수에 저장해보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "respiratory-organic",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "삭제할 훈련 데이터의 개수 : 1\n",
      "삭제할 테스트 데이터의 개수 : 0\n",
      "훈련 데이터의 개수 : 52235\n",
      "훈련 레이블의 개수 : 52235\n",
      "테스트 데이터의 개수 : 13059\n",
      "테스트 레이블의 개수 : 13059\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  return array(a, dtype, copy=False, order=order)\n"
     ]
    }
   ],
   "source": [
    "drop_train = [index for index, sentence in enumerate(decoder_input_train) if len(sentence) == 1]\n",
    "drop_test = [index for index, sentence in enumerate(decoder_input_test) if len(sentence) == 1]\n",
    "\n",
    "print('삭제할 훈련 데이터의 개수 :', len(drop_train))\n",
    "print('삭제할 테스트 데이터의 개수 :', len(drop_test))\n",
    "\n",
    "encoder_input_train = np.delete(encoder_input_train, drop_train, axis=0)\n",
    "decoder_input_train = np.delete(decoder_input_train, drop_train, axis=0)\n",
    "decoder_target_train = np.delete(decoder_target_train, drop_train, axis=0)\n",
    "\n",
    "encoder_input_test = np.delete(encoder_input_test, drop_test, axis=0)\n",
    "decoder_input_test = np.delete(decoder_input_test, drop_test, axis=0)\n",
    "decoder_target_test = np.delete(decoder_target_test, drop_test, axis=0)\n",
    "\n",
    "print('훈련 데이터의 개수 :', len(encoder_input_train))\n",
    "print('훈련 레이블의 개수 :', len(decoder_input_train))\n",
    "print('테스트 데이터의 개수 :', len(encoder_input_test))\n",
    "print('테스트 레이블의 개수 :', len(decoder_input_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-individual",
   "metadata": {},
   "source": [
    "## 패딩하기 \n",
    "패딩작업은 서로 다른 길이의 샘플들을 병렬 처리하기 위해 같은 길이로 맞춰줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "received-donna",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input_train = pad_sequences(encoder_input_train, maxlen=text_max_len, padding='post')\n",
    "encoder_input_test = pad_sequences(encoder_input_test, maxlen=text_max_len, padding='post')\n",
    "decoder_input_train = pad_sequences(decoder_input_train, maxlen=headlines_max_len, padding='post')\n",
    "decoder_target_train = pad_sequences(decoder_target_train, maxlen=headlines_max_len, padding='post')\n",
    "decoder_input_test = pad_sequences(decoder_input_test, maxlen=headlines_max_len, padding='post')\n",
    "decoder_target_test = pad_sequences(decoder_target_test, maxlen=headlines_max_len, padding='post')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amateur-alexandria",
   "metadata": {},
   "source": [
    "# 1-2. 모델 설계하기\n",
    "함수형 API를 이용해서 인코더를 설계합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "improved-serve",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer lstm_1 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n",
      "WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "\n",
    "# 인코더 설계 시작\n",
    "embedding_dim = 128\n",
    "hidden_size = 256\n",
    "\n",
    "# 인코더\n",
    "encoder_inputs = Input(shape=(text_max_len,))\n",
    "\n",
    "# 인코더의 임베딩 층\n",
    "enc_emb = Embedding(src_vocab, embedding_dim)(encoder_inputs)\n",
    "\n",
    "# 인코더의 LSTM 1\n",
    "encoder_lstm1 = LSTM(hidden_size, return_sequences=True, return_state=True ,dropout = 0.4, recurrent_dropout = 0.4)\n",
    "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
    "\n",
    "# 인코더의 LSTM 2\n",
    "encoder_lstm2 = LSTM(hidden_size, return_sequences=True, return_state=True, dropout=0.4, recurrent_dropout=0.4)\n",
    "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
    "\n",
    "# 인코더의 LSTM 3\n",
    "encoder_lstm3 = LSTM(hidden_size, return_state=True, return_sequences=True, dropout=0.4, recurrent_dropout=0.4)\n",
    "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vocal-possession",
   "metadata": {},
   "source": [
    "* 임베딩 벡터의 차원은 128로 정의하고, hidden state의 크기를 256으로 정의했습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "instrumental-shooting",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernel since it doesn't meet the cuDNN kernel criteria. It will use generic GPU kernel as fallback when running on GPU\n"
     ]
    }
   ],
   "source": [
    "# 디코더 설계\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "\n",
    "# 디코더의 임베딩 층\n",
    "dec_emb_layer = Embedding(tar_vocab, embedding_dim)\n",
    "dec_emb = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "# 디코더의 LSTM\n",
    "decoder_lstm = LSTM(hidden_size, return_sequences=True, return_state=True, dropout=0.4, recurrent_dropout=0.2)\n",
    "decoder_outputs, _, _ = decoder_lstm(dec_emb, initial_state=[state_h, state_c])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "widespread-latest",
   "metadata": {},
   "source": [
    "* 디코더의 임베딩 층과 LSTM을 설계하는 것은 인코더와 거의 동일하지만 LSTM의 입력을 정의할 때, initial_state의 인자값으로 인코더의 hidden state와 cell state의 값을 넣어줘야합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "collect-necessity",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 38)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, 38, 128)      2176000     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 38, 256), (N 394240      embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   [(None, 38, 256), (N 525312      lstm[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, None, 128)    896000      input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm_2 (LSTM)                   [(None, 38, 256), (N 525312      lstm_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "lstm_3 (LSTM)                   [(None, None, 256),  394240      embedding_1[0][0]                \n",
      "                                                                 lstm_2[0][1]                     \n",
      "                                                                 lstm_2[0][2]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, None, 7000)   1799000     lstm_3[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 6,710,104\n",
      "Trainable params: 6,710,104\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 디코더의 출력층\n",
    "decoder_softmax_layer = Dense(tar_vocab, activation='softmax')\n",
    "decoder_softmax_outputs = decoder_softmax_layer(decoder_outputs) \n",
    "\n",
    "# 모델 정의\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_softmax_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "minor-motion",
   "metadata": {},
   "source": [
    "# 어텐션 메커니즘\n",
    "일반적인 seq2seq보다는 어텐션 메커니즘을 사용한 seq2seq를 사용하는 것이 더 나은 성능을 얻을 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fifty-antenna",
   "metadata": {},
   "outputs": [],
   "source": [
    "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/thushv89/attention_keras/master/src/layers/attention.py\", filename=\"attention.py\")\n",
    "from attention import AttentionLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "endangered-repair",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 38)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, 38, 128)      2176000     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 38, 256), (N 394240      embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   [(None, 38, 256), (N 525312      lstm[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, None, 128)    896000      input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm_2 (LSTM)                   [(None, 38, 256), (N 525312      lstm_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "lstm_3 (LSTM)                   [(None, None, 256),  394240      embedding_1[0][0]                \n",
      "                                                                 lstm_2[0][1]                     \n",
      "                                                                 lstm_2[0][2]                     \n",
      "__________________________________________________________________________________________________\n",
      "attention_layer (AttentionLayer ((None, None, 256),  131328      lstm_2[0][0]                     \n",
      "                                                                 lstm_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "concat_layer (Concatenate)      (None, None, 512)    0           lstm_3[0][0]                     \n",
      "                                                                 attention_layer[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, None, 7000)   3591000     concat_layer[0][0]               \n",
      "==================================================================================================\n",
      "Total params: 8,633,432\n",
      "Trainable params: 8,633,432\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 어텐션 층(어텐션 함수)\n",
    "attn_layer = AttentionLayer(name='attention_layer')\n",
    "# 인코더와 디코더의 모든 time step의 hidden state를 어텐션 층에 전달하고 결과를 리턴\n",
    "attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])\n",
    "\n",
    "# 어텐션의 결과와 디코더의 hidden state들을 연결\n",
    "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
    "\n",
    "# 디코더의 출력층\n",
    "decoder_softmax_layer = Dense(tar_vocab, activation='softmax')\n",
    "decoder_softmax_outputs = decoder_softmax_layer(decoder_concat_input)\n",
    "\n",
    "# 모델 정의\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_softmax_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "capital-throw",
   "metadata": {},
   "source": [
    "* 인코더의 hidden state들과 디코더의 hidden state들을 어텐션 함수의 입력으로 사용하고, 어텐션 함수가 리턴한 값을 예측 시에 디코더의 hidden state와 함께 활용하는 형태로 작동합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "satisfied-sessions",
   "metadata": {},
   "source": [
    "## 모델 훈련"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "alpha-nicholas",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "205/205 [==============================] - 141s 632ms/step - loss: 6.4488 - val_loss: 5.7010\n",
      "Epoch 2/50\n",
      "205/205 [==============================] - 129s 628ms/step - loss: 5.6636 - val_loss: 5.4542\n",
      "Epoch 3/50\n",
      "205/205 [==============================] - 130s 632ms/step - loss: 5.3396 - val_loss: 5.2073\n",
      "Epoch 4/50\n",
      "205/205 [==============================] - 130s 635ms/step - loss: 5.0555 - val_loss: 5.0543\n",
      "Epoch 5/50\n",
      "205/205 [==============================] - 130s 633ms/step - loss: 4.8236 - val_loss: 4.8468\n",
      "Epoch 6/50\n",
      "205/205 [==============================] - 130s 636ms/step - loss: 4.6128 - val_loss: 4.6914\n",
      "Epoch 7/50\n",
      "205/205 [==============================] - 129s 631ms/step - loss: 4.4403 - val_loss: 4.6074\n",
      "Epoch 8/50\n",
      "205/205 [==============================] - 129s 631ms/step - loss: 4.2881 - val_loss: 4.5496\n",
      "Epoch 9/50\n",
      "205/205 [==============================] - 130s 633ms/step - loss: 4.1489 - val_loss: 4.4463\n",
      "Epoch 10/50\n",
      "205/205 [==============================] - 129s 629ms/step - loss: 4.0221 - val_loss: 4.3559\n",
      "Epoch 11/50\n",
      "205/205 [==============================] - 130s 633ms/step - loss: 3.9021 - val_loss: 4.3272\n",
      "Epoch 12/50\n",
      "205/205 [==============================] - 129s 628ms/step - loss: 3.7926 - val_loss: 4.2653\n",
      "Epoch 13/50\n",
      "205/205 [==============================] - 128s 627ms/step - loss: 3.6962 - val_loss: 4.2212\n",
      "Epoch 14/50\n",
      "205/205 [==============================] - 127s 620ms/step - loss: 3.5977 - val_loss: 4.1939\n",
      "Epoch 15/50\n",
      "205/205 [==============================] - 126s 615ms/step - loss: 3.5241 - val_loss: 4.2207\n",
      "Epoch 16/50\n",
      "205/205 [==============================] - 127s 618ms/step - loss: 3.4358 - val_loss: 4.1570\n",
      "Epoch 17/50\n",
      "205/205 [==============================] - 125s 609ms/step - loss: 3.3589 - val_loss: 4.1523\n",
      "Epoch 18/50\n",
      "205/205 [==============================] - 126s 615ms/step - loss: 3.2988 - val_loss: 4.1417\n",
      "Epoch 19/50\n",
      "205/205 [==============================] - 125s 610ms/step - loss: 3.2266 - val_loss: 4.1329\n",
      "Epoch 20/50\n",
      "205/205 [==============================] - 126s 615ms/step - loss: 3.1541 - val_loss: 4.1267\n",
      "Epoch 21/50\n",
      "205/205 [==============================] - 126s 614ms/step - loss: 3.1006 - val_loss: 4.0924\n",
      "Epoch 22/50\n",
      "205/205 [==============================] - 125s 609ms/step - loss: 3.0465 - val_loss: 4.0880\n",
      "Epoch 23/50\n",
      "205/205 [==============================] - 127s 618ms/step - loss: 3.0003 - val_loss: 4.0778\n",
      "Epoch 24/50\n",
      "205/205 [==============================] - 127s 619ms/step - loss: 2.9413 - val_loss: 4.0828\n",
      "Epoch 25/50\n",
      "205/205 [==============================] - 126s 613ms/step - loss: 2.8883 - val_loss: 4.0875\n",
      "Epoch 00025: early stopping\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy')\n",
    "es = EarlyStopping(monitor='val_loss', patience=2, verbose=1)\n",
    "history = model.fit(x=[encoder_input_train, decoder_input_train], y=decoder_target_train, \\\n",
    "          validation_data=([encoder_input_test, decoder_input_test], decoder_target_test), \\\n",
    "          batch_size=256, callbacks=[es], epochs=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "aging-active",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt0UlEQVR4nO3deXhU1f3H8feZ7AkhIRuQjSQQ9j0hBAXEhVUEFKGAVKkVXOrSjUptrXWrtbZ1rxUUf1gVtYgWARVUEFS2EFnCGpYEskBCVrIvc35/3AFCJCEJk0xm5vt6nnnmztybO9/bsZ+5nHvuOUprjRBCCMdjsnUBQgghWocEvBBCOCgJeCGEcFAS8EII4aAk4IUQwkG52uqDg4KCdFRUlK0+Xggh7NLOnTvPaK2Dm7KtzQI+KiqKpKQkW328EELYJaVUelO3lSYaIYRwUBLwQgjhoCTghRDCQdmsDV4IIVqiurqajIwMKioqbF1Kq/L09CQ8PBw3N7cW70MCXghhVzIyMvD19SUqKgqllK3LaRVaa/Ly8sjIyCA6OrrF+5EmGiGEXamoqCAwMNBhwx1AKUVgYOAV/ytFAl4IYXccOdzPscYx2l3AH8kp4fFP91FVY7Z1KUII0a7ZXcCfzC/jre/SWL//tK1LEUI4ocLCQv71r381++8mTZpEYWGh9QtqhN0F/OiewYT5e/HutibfzCWEEFbTUMDX1NQ0+ndr167F39+/laq6NLsLeBeTYs7wSL4/msfR3BJblyOEcDKLFi3i6NGjDB48mGHDhjFq1CimTJlC3759AZg2bRpxcXH069ePxYsXn/+7qKgozpw5Q1paGn369GH+/Pn069ePcePGUV5e3iq1NqmbpFLKH3gD6A9o4E6t9ZY66xXwIjAJKAPmaa2TrV6txYz4cJ5ff5jl207wx8l9W+tjhBDt3OOf7mN/VrFV99k3tCOP3dSvwfV//etfSUlJYdeuXWzcuJEbb7yRlJSU890Zly5dSkBAAOXl5QwbNozp06cTGBh40T5SU1NZvnw5S5YsYebMmXz00UfMnTvXqscBTT+DfxH4XGvdGxgEHKi3fiIQa3ksAF6zWoWXEOLryfh+XViRnEFFdW1rfpQQQjQqISHhor7qL730EoMGDSIxMZGTJ0+Smpr6o7+Jjo5m8ODBAMTFxZGWltYqtV32DF4p5QeMBuYBaK2rgKp6m00F3tbGDN5blVL+SqmuWutsK9d73m2JkazZm82aPdlMjwtvrY8RQrRjjZ1ptxUfH5/zyxs3buTLL79ky5YteHt7M2bMmEv2Zffw8Di/7OLi0mpNNE05g48GcoG3lFI/KKXeUEr51NsmDDhZ53WG5b2LKKUWKKWSlFJJubm5LS4aYERMIDFBPnKxVQjRpnx9fTl79uwl1xUVFdGpUye8vb05ePAgW7dubePqLtaUgHcFhgKvaa2HAKXAopZ8mNZ6sdY6XmsdHxzcpPHqG6SUcbE1+USh1dvghBCiIYGBgVx99dX079+fhQsXXrRuwoQJ1NTU0KdPHxYtWkRiYqKNqjQoo1WlkQ2U6gJs1VpHWV6PAhZprW+ss83rwEat9XLL60PAmMaaaOLj4/WVTvhRWFZFwl++YmZ8OE9NG3BF+xJC2IcDBw7Qp08fW5fRJi51rEqpnVrr+Kb8/WXP4LXWp4CTSqlelreuB/bX22wVcLsyJAJFrdn+fo6/tzuTB3bl4+RMSiob74MqhBDOpqm9aB4A3lVK7QEGA39RSt2jlLrHsn4tcAw4AiwB7rN2oQ25bXg3SqtqWbUrq60+Uggh7EKT+sFrrXcB9f9J8O866zXwC+uV1XRDI/3p3cWXd7elMzshwikGIRJCiKawuztZ61NKcVtiN/ZlFbM7o8jW5QghRLth9wEPMG1wKN7uLry7VbpMCiHEOQ4R8L6ebkwbEsane7IoKqu2dTlCCNEuOETAA8xJiKSi2sxHyRm2LkUI4cBaOlwwwAsvvEBZWZmVK2qYwwR8/zA/Bkf48+62dC7Xt18IIVrKngLeoSbdvm14JAtX7GHb8XwSYwIv/wdCCNFMdYcLHjt2LCEhIXz44YdUVlZy88038/jjj1NaWsrMmTPJyMigtraWRx99lNOnT5OVlcW1115LUFAQGzZsaPVaHSrgJw8M5cnV+3l32wkJeCGcwWeL4NRe6+6zywCY+NcGV9cdLnjdunWsWLGC7du3o7VmypQpbNq0idzcXEJDQ1mzZg1gjFHj5+fHP//5TzZs2EBQUJB1a26AwzTRAHi5uzA9LpzPU7I5U1Jp63KEEA5u3bp1rFu3jiFDhjB06FAOHjxIamoqAwYMYP369Tz88MNs3rwZPz8/m9TnUGfwYDTTvPVdGv9NyuDeMd1tXY4QojU1cqbdFrTW/P73v+fuu+/+0brk5GTWrl3LH//4R66//nr+9Kc/tXl9DnUGD9AjxJfh0QG8tz0ds1kutgohrKvucMHjx49n6dKllJQY04dmZmaSk5NDVlYW3t7ezJ07l4ULF5KcnPyjv20LDncGD3BbYjceXP4Dm4+c4ZqeVzYssRBC1FV3uOCJEycyZ84cRowYAUCHDh145513OHLkCAsXLsRkMuHm5sZrrxmT3C1YsIAJEyYQGhraJhdZLztccGu5ouGCa2vApeHfpqoaMyOe+Yq4bp1YfHuTRtUUQtgJGS7YisMFtzvHNsKrCVB4ssFN3F1NzBwWwVcHc8guap2psIQQor2zv4D3DYXSXHh/NlSVNrjZ7GGRmLXmgx0N/xAIIYQjs7+AD+4Jty6FUynwyX3QQBNTZKA3o2ODeX/7SWpqzW1cpBCiNTnD3erWOEb7C3iA2LEw9gnY/wlseq7BzW4bHsmp4gq+OpjTdrUJIVqVp6cneXl5Dh3yWmvy8vLw9PS8ov3Yby+aqx6AnP2w4WkI7g19p/xok+t6h9CloyfvbjvB+H5dbFCkEMLawsPDycjIIDc319altCpPT0/Cw8OvaB/2G/BKweQX4EwqfHw3BEQbtxjX4epiYlZCBC98mcqJvDIiA71tU6sQwmrc3NyIjo62dRl2wT6baM5x84RZ74KnPyyfAyU//kWfNSwSF5Pive0n2r4+IYSwIfsOeADfLkbIl+bAh7dDTdVFq7v4eTK2T2fe3ZrOqaIKGxUphBBtz/4DHiBsKEx9FU58D2t/+6OeNYsm9qaq1syj/0tx6AszQghRl2MEPMCAW2HUbyB5GWxfctGqqCAffj22J+v3n2bt3lM2KlAIIdpWkwJeKZWmlNqrlNqllPrR+AJKqTFKqSLL+l1KqbYfNg3g2j9Cr0nw+SLjjtc6fj4ymgFhfjy2KoWC0qpL/70QQjiQ5pzBX6u1HtzIGAibLesHa62fsEZxzWYywS2LIagnfHgH5B09v8rVxcSz0wdSWFbNU2sO2KQ8IYRoS47TRHOOhy/MXg7KBMtnQ0XR+VV9QztyzzXd+Sg5g28OO3YfWiGEaGrAa2CdUmqnUmpBA9uMUErtVkp9ppTqZ6X6WiYgGma+DflH4aO7wFx7ftX91/UgJtiHR1bupbSyxoZFCiFE62pqwI/UWg8FJgK/UEqNrrc+GeimtR4EvAx8cqmdKKUWKKWSlFJJrX4XWvQomPgspK6Drx4//7anmwt/mz6QrKJynvviUOvWIIQQNtSkgNdaZ1qec4CPgYR664u11iWW5bWAm1LqR7PKaq0Xa63jtdbxwcFtMBHHsLsg/k747kXY/f75t+OjArg9sRvLtqSxMz2/9esQQggbuGzAK6V8lFK+55aBcUBKvW26KKWUZTnBst8865fbAhP/BlGjYNWDkHGhA9DCCb3p2tGThz/aS2VNbSM7EEII+9SUM/jOwLdKqd3AdmCN1vpzpdQ9Sql7LNvcCqRYtnkJmKXbyx1FLm4wY5lxx+v7t0FxFgAdPFx5+pYBHMkp4dWvj9i4SCGEsD77nLKvJU7vhzfHQlAs/OwzcPMC4Ncf7GLV7iw+fWAkfbp2bLt6hBCiBRx7yr6W6tzX6COftQtWPXB+OINHJ/fFz8uNhz/aIxODCCEcivMEPEDvG+G6P8Le/8J3LwDQycedP0/px56MIt76Ls2m5QkhhDU5V8CDMV5N/+nw5eNw6DMAJg/syg19QvjH+kOknWl4nlchhLAnzhfwSsGUV6DrIOMmqJwDKKV4clp/3Ewmfr9yr4w4KYRwCM4X8ADu3jDrPXDzhuWzoCyfrn5e/H5SH7Ycy+ODHSdtXaEQQlwx5wx4AL8wI+SLs+C/d0BtNbOGRTA8OoCn1xyQyUGEEHbPeQMeIGIY3PQiHN8EXzyCyaT46/SBMjmIEMIhOHfAAwyeAyPuh+2LIektoutMDvLpnmxbVyeEEC0mAQ8w9gnocYMx3V/ad/x8ZDRDIv15ZOVejuaW2Lo6IYRoEQl4AJMLTH8TOkXDhz/Ftfgkr84ZiruriXv+s1OGFRZC2CUJ+HO8/GH2+2CugffnEOpVy0uzhnA0t4RF0nVSCGGHJODrCuoBt74FOfvh47sZ2T2A34zrxae7s1j2fZqtqxNCiGaRgK+vx/Uw/i9wcDWseoB7rwrlhj4hPLXmgIwdL4SwKxLwlzL8Hhj9O9j1Dqal43j+hg6E+ntx37vJ5J6ttHV1QgjRJBLwl6IUXPcHuG0FFGfiu+wG3h2RQWFZNQ8sT5ZRJ4UQdkECvjGxY+GezdC5HxFf3c/a7h/zw7FTPLdO5nIVQrR/EvCX4xcO89bAVQ/SPf0Dvu70Fz7btIXPU07ZujIhhGiUBHxTuLjBuCdh9vuE6hw+8/wDX/z3dY7L0MJCiHZMAr45ek1E3bMZt869eV79k12L76asTEJeCNE+ScA3l38k7nd9QUavedxc9Sm5L12Hzj9u66qEEOJHJOBbwtWd8NkvsqbPc3QqP0HVv0bBwTW2rkoIIS4iAX8FJs6Yz9Ph/+ZwVRC8Pwc+fwRqZdwaIUT7IAF/BUwmxSO3TeKXPn/jv6aJsPVV+OReMNfaujQhhMC1KRsppdKAs0AtUKO1jq+3XgEvApOAMmCe1jrZuqW2T37ebrz00+Hc8q9aagO6MGvvW0avmymvgEl+P4UQttOkgLe4Vmt9poF1E4FYy2M48Jrl2Sn0C/XjqWn9WbjCTECkYtyupWByhckvSMgLIWymOQHfmKnA29oYU3erUspfKdVVa+00UyLNiI/gZEE5C77SvBtj5urk/zPO5Cf93Rj6QAgh2lhTA14D65RSGnhda7243vow4GSd1xmW9y4KeKXUAmABQGRkZIsKbs9+dUMsBaVV3LZ1LJ/Emhm84w0wucGEZyTkhRBtrqkBP1JrnamUCgHWK6UOaq03NffDLD8MiwHi4+MdbgYNpRSPT+lHYXk103aPZ10fTc9tr4GLK4x9UkJeCNGmmhTwWutMy3OOUupjIAGoG/CZQESd1+GW95yOyaT4x4xBFJVXM+HgBDb2NRP5/cvg4g7XPSohL4RoM5e9AqiU8lFK+Z5bBsYBKfU2WwXcrgyJQJEztb/X5+5q4t9zhzIoohNjD93IqdhZsPkf8M2zti5NCOFEmtLFozPwrVJqN7AdWKO1/lwpdY9S6h7LNmuBY8ARYAlwX6tUa0e83V15a94wugV2YOzhaeT3nAkbn4FNf7d1aUIIJ6FsNZl0fHy8TkpKsslnt6VTRRVMf+17Kquq2NDjA3wPrzTa469+0NalCSHskFJqZ/17kRoinbRbWRc/T965azhauXBj+hzKe06F9Y/C1tdsXZoQwsFJwLeB6CAflt2ZQEGFmWnZd1DVczJ8vgi2L7F1aUIIByYB30b6h/mx5I54jhdUMTt/PjWxE2Dtb2HHm7YuTQjhoCTg21BiTCCvzhnKrqwy5pffT22PsbDm17D611BTaevyhBAORgK+jY3t25lnpw9kw5FifsnDmEc8CElvwtLxUJBu6/KEEA5EAt4Gbo0L5w+T+vBpSg5/KJ2JeeY7kHcMXh8Nh7+wdXlCCAchAW8j80fH8Itru7N8+wl+ty+S2vkbwT8S3psJXz4uE4cIIa6YtUaTFC3w23G9cHMx8cKXqZRXd+X5eV/gvm4RfPtPyNgB098E3862LlMIYack4G1IKcUvb+iJj7srT689QEVVLa/e9jyekYnGhdfXR8Gtb0HU1bYuVQhhh6SJph2YPzqGp6b15+tDOfx82Q5K+8yE+V+BewdYdhN8+wLY6I5jIYT9koBvJ+YmduMfMwax5Wgety/dTrFfT1iwEfrcBF8+ZkzqXV5o6zKFEHZEAr4duWVoOK/OGcqejELmLNlKfq0nzPg/mPAspK4zetlk7bJ1mUIIOyEB385MHNCVxT+NJ/V0CbMWbyHnbCUk3gM/+wzMNfDmOPj6aSjOsnWpQoh2TgK+Hbq2dwhv/WwYGQXlzHx9CxkFZRCRAHdvhp7jYNNz8Hx/+OCncOwbaZ8XQlySDBfcjiWfKOCOpdvx9XDl3fmJRAf5GCvyj0HSW/DDf6C8AIJ6QvzPYdAs8PK3ac1CiNbVnOGCJeDbuZTMIm5fuh0Xk+Kdnw+nVxffCyurK2Dfx7DjDchMAjdvGDADht0FXQfarmghRKuRgHcwR3LOMmfJNqprzbx953AGhPv9eKOsH4yRKfeugJpyCE8wgr7vVHDzbPuihRCtQgLeAaXnlTJnyTaKy6t54454hscEXnrD8gLYtdw4q88/Ct6BMGQu9JkCoUPA5NK2hQshrEoC3kFlFZYz981tnMwv46lp/fnJsMiGNzab4fg3RtAfWgvaDF4B0ON66DHWePYJarvihRBWIQHvwIrKqrl/eTKbU89w59XRPDKpN64ul+kMVZYPR7+G1PVw5EsoOwMo44w+dqwR+GFD5exeCDsgAe/gamrNPL32AG99l8ao2CBemTMUPy+3pv2x2QzZu4ygT11vXJzVZvDqBN2vu3B23yGkVY9BCNEyEvBO4oMdJ/jjJylEdPJmyR3xdA/u0PydlOXDsQ2Q+qUR+qU5xvsRiTB6oRH2Slm3cCFEi7VKwCulXIAkIFNrPbneunnAc0Cm5a1XtNZvNLY/CXjr2H48n3vf2UlVrZlX5wxldM/glu/MbIZTe+DIetj5NhSdMHrjXPsIxIyRoBeiHWhOwDfnTtaHgAONrP9Aaz3Y8mg03IX1JEQH8L/7rybM34t5b23nzW+P0+J/lZlMEDrYOHN/YCdMfh6KM+E/0+D/boS0b61ZuhCilTUp4JVS4cCNgAR3OxTeyZuP7r2KsX078+Tq/Tz80R4qa2qvbKeu7hB/JzyQDBOfg7yjRsgvuwlObLVO4UKIVtXUM/gXgN8B5ka2ma6U2qOUWqGUirjUBkqpBUqpJKVUUm5ubjNLFY3x8XDltdviePC6HnyYlMHcN7ZxpqTyynfs5gnDF8BDu2D8XyDngDFB+H9ugQxpYhOiPbtswCulJgM5WuudjWz2KRCltR4IrAeWXWojrfVirXW81jo+OPgK2orFJZlMil+P68XLs4ewN7OIqa98x/6sYuvs3M0LRvwCHtoNY58w7px943p4d6axLIRody57kVUp9QzwU6AG8AQ6Aiu11nMb2N4FyNdaX+J++gvkImvr2ptRxPy3kygqr+b5nwxmQv8u1v2AyrOwfTF89xJUFELvyUbbfehg636OEOIirdZNUik1BvjtJXrRdNVaZ1uWbwYe1lonNrYvCfjWl1NcwYL/7GTXyULuHh3Db8cbk3xbVUURbP03bHkVKosgapRxph873rhoK4SwqtbqRVP/Q55QSk2xvHxQKbVPKbUbeBCY19L9CusJ6ejJ+wsSmTM8ktc3HWPGv7dwMr/Muh/i6QdjHoZf7YVxT0H+cVg+C16JN4ZJqCq17ucJIZpMbnRyEmv2ZLPooz2g4NnpA5k0oGvrfFBtNRxYBd+/AlnJxh2y8XfCsPnQsZU+UwgnIneyiks6mV/G/ct/YPfJQm4bHsmjk/vi6dZK489oDSe3wfcvw8E1YHKFAbdC4n0yVr0QV0ACXjSoutbM3784xOubjtG7iy+vzBlCjxDfy//hlcg/ZrTT//AOVJdC9GgYcb8x7o200wvRLBLw4rI2HsrhNx/upqyqlsen9GNGfDiqtYciKC+Anctg2+twNgsCukPEcAiIgcAY4zkgxmjXF0JckgS8aJLTxRX86oNdfH80j6mDQ3lqWn98PZs4KuWVqK2GfZ/Arncg97AR9nV5BxrhH1An9M/9AHh1av36hGjHJOBFk9WaNf/acITnvzxMRIA3r8weeukpAVtTVRkUpBkzUOUfu/DIOwbFGRdv69sVYsdBr0kQc41xA5YQTkQCXjTb9uP5PPT+D5wpqWTRxD7ceXVU6zfZNEV1ORSkXwj/jCQ48hVUnQVXL2MM+14Toed4GcNeOAUJeNEiBaVVLFyxhy8PnOa63iE8c8sAOndshxN211RB+rdw6DPjUXQSUBA+zAj7XpMguJcMbywckgS8aDGtNcu+T+OZzw7i7mrijzf2YWZ8RPs4m78UreF0iiXs114YF6dTtBH0vSZCZCK4tMG1BSHagAS8uGLHz5Ty8Ed72H48n5E9gnjmlgFEBHjbuqzLK86Cw58bgX/sG6itBFdPY/7ZsDjjLD98GPiF2bpSIVpEAl5YhdmseW/7CZ5ZewCzht9N6MUdI6Iwmdrp2Xx9lSXGdITpWyBjB2TvNgIfjIu14fEQFm8EfuhgcPexablCNIUEvLCqzMJyHlm5l28O5xLXrRPPTh9Ij5AWzP9qazVVcHqvcaE2Y4fxXHDcWKdcoHNfI+zD4iGoJ3SKAp8g67blm2vhbLbRa8g7EIJ7y7UC0SwS8MLqtNasTM7kidX7Ka+u5aHrY1kwOsb6o1O2tdIzkLnzQuBn7oTKOmPou3cwgr7uIyDaaOP3izBmvqqvosgI8POP9AvLhSfAXH1h24AY6H0j9L7J+HGRO3vFZUjAi1aTc7aCx/63j89STtEvtCN/u3Ug/UId6M5Ts7lOf/zjlmA+fiGgayoubKtM0DEcOnUDL38oPGlsU1F48T69Ol34cfDvZlnuZmx7YDUc32SEvk+IcVG4z03GcA6uHm1xxMLOSMCLVvfZ3mwe/d8+CsuquOea7jxwfQ88XFtp4LL2wmyGktMXh/65H4GKQuOM/qKz/W5GoHv5N77fiiJIXQ8HVxvPVSXg7guxY42z+9ixMnyDOE8CXrSJwrIqnli9n5XJmfQI6cCz0wcQ1y3A1mXZt5pKo/fPwdVGt8/SXDC5GWf0vSdBxzCjy6fJDVzcLQ+3es/nlt2MHkTSRdShSMCLNrXhUA5/WLmXrKIKZsSFs2hibwI7SPPCFTPXGtcGDq42mnLOXRBuLhcP8OgAHr7Gvww8fOu8tjzXXXb3MR5u3sZ77t6WZct7bl5yYdiGJOBFmyutrOGlr1N5c/NxfDxcWTi+F7MTInGxly6V7Z3WRsBXFBmDtdVWWZ7PLVddermm0hjWobLEmEe3yvJ80XKJMYxzkylL4J8LfR/jeoGrp+XZ48JrF/d671ve8/A1eiqF9DF6Kokmk4AXNpN6+iyP/i+FrcfyGRjux5NT+zMowt/WZYnLMddeCPyqUuNRXWYMBFddankuM7Y5v1x64bmm0rgAXVNp3GtQ9/X5Rzlo848/2yfY6C4a0sd4BPeBkN4ycmgDJOCFTWmtWbU7i6fWHOBMSSWzEyL53fhe+HtfokuhcC61NcYPQHkB5B6C3IOQsx9yDhrLVSUXtvXtagn+vkbgdwy7+PqCi7vlWkRjrx3vwr8EvGgXzlZU8/z6VJZtSaOjpyuLJvZmRlyE/dwJK9qW1lCUATkHIPeAEfo5+40fgprylu3TxcNoRvLoYLn+0MFyXcGnznUHy2t3X+P6grmmTlNXlXGDXG39R7XlXysNNI2Zq+s1pdVbf9UDcP2fWnRIEvCiXTmQXcyjn6SQlF7A0Eh/npzW37H6zovWZTZDYbrRRbW2uk54ngvTGn50TcJcbQTzuWalyhJL81Kd5cqSC9cndG3jNZjq9FRy9bhE7yWPi//lcKleTXX3ETUKYm9o0f8cEvCi3TGbNSt/yOSZtQcoKKvi9hFR/GpsT/y8pAufsDGtLRejS4wfBNO5EK/T7NOO7jBuTsC7NmOnLkASkKm1nlxvnQfwNhAH5AE/0VqnNbli4fBMJsWtceGM7dOZv687xLItaazek83C8T25NS5CetsI21EK3DyNh4Npzs/SQ8CBBtb9HCjQWvcAngeevdLChGPy83bjyWn9WfWLkUQGePHwR3uZ/PK3fH/kjK1LE8LhNCnglVLhwI3AGw1sMhVYZlleAVyv2u0MEaI9GBDux0f3XsXLs4dQXF7NnDe2cdeyJI7lllz+j4UQTdLUM/gXgN8Bl+jECkAYcBJAa10DFAGB9TdSSi1QSiUppZJyc3ObX61wKEopbhoUyle/uYbfTejF1mN5jHt+E49/aoxxI4S4MpcNeKXUZCBHa73zSj9Ma71Yax2vtY4PDg6+0t0JB+Hp5sJ9Y3qw4bdjmBEfwbLv07jmuY0s/fY4VTUNnVMIIS6nKWfwVwNTlFJpwPvAdUqpd+ptkwlEACilXAE/jIutQjRZsK8Hz9wygLUPjWJguB9PrN7P+Bc2sX7/aWzV20sIe3bZgNda/15rHa61jgJmAV9rrefW22wVcIdl+VbLNvL/SNEivbt05O07E3hr3jBMCua/ncScJdvYl1Vk69KEsCst7typlHpCKTXF8vJNIFApdQT4NbDIGsUJ56WU4treIXz+y9E8MbUfB08VM/nlb1n4391kFJTZujwh7ILc6CTsQlFZNS9/ncrbW9IBmJ0QwS+u7UFIR8fruyxEY+ROVuGwsgrLefnrVD5MysDNRXHHiCjuvqY7AT4ykJlwDhLwwuGlnSnlxa9S+WRXJj7urtw5Mpq7RkXT0VOGPhCOTQJeOI3U02d5/svDrN17Cj8vNxaMjuFnV0fh7d7kUTiEsCsS8MLppGQW8c/1h/n6YA5BHdy5d0wPbhseiaeb440HLpybBLxwWjvT8/nHusN8fzSPLh09eeD6HsyIi8Ddtf2MBijElZCAF07v+yNn+Pu6QySfKCTM34t7rolhRnyEnNELuycBLwTG1IEbD+Xy8tepJJ8oJNjXg/mjorlteDd8PKSNXtgnCXgh6tBas+VYHq9uOMJ3R/Lw93bjZ1dFM++qKPy8pdeNsC8S8EI0IPlEAf/acIQvD+TQwcOVuYnduGtUNEEdPGxdmhBNIgEvxGXszyrm1Y1HWLs3G3cXE7MTIlkwOoZQfy9blyZEoyTghWiio7klvLbxKJ/8kIlSMH1oOPdc052oIB9blybEJUnAC9FMGQVlvP7NMT5IOklNrZmJA7qyYFQMgyL8bV2aEBeRgBeihXKKK3jzu+O8t/UEZytrSIgOYMGoGK7rHYJJJgYX7YAEvBBX6GxFNR/sOMlb36WRWVhOTLAP80fFcPOQMOlLL2xKAl4IK6muNbN2bzZLNh8jJbOYoA7u3D4iirmJ3WQES2ETEvBCWNm5vvRLNh1jw6FcPN1MzIiL4Ocjo+WCrGhTzQl4uZ1PiCZQSnFV9yCu6h5E6umzvLH5OB/sOMk729IZ37cLd42KJq5bJ5SSdnrRfsgZvBAtlHO2gre/T+c/W9MpKq+mdxdf5gyPZNqQMBmXXrQaaaIRog2VVtbwv11ZvLc9nZTMYrzcXLhpUFfmDO/GoHA/OasXViUBL4SN7Mko5L1tJ1i1O4uyqlr6du14/qy+gwxwJqxAAl4IGztbUc0nu7J4b9sJDmQX4+3uwtTBocxJ6MaAcD9blyfsmAS8EO2E1ppdJ42z+k/3ZFFRbWZguB+zEyKZMihUhi0WzWbVgFdKeQKbAA+MXjcrtNaP1dtmHvAckGl56xWt9RuN7VcCXjibovJqPvkhk/e2neDQ6bN4u7swaUBXbo0LJyEqQO6UFU1i7YBXgI/WukQp5QZ8Czyktd5aZ5t5QLzW+v6mFikBL5yV1prkEwX8NymD1XuyKamsISLAi+lDw5k+NJyIAG9blyjaMav2g9fGL0CJ5aWb5WGbdh0hHIBSirhuAcR1C+Cxm/rxxb5TrNiZwYtfpfLCl6kkxgRwa1wEE/t3kSYccUWa1AavlHIBdgI9gFe11g/XWz8PeAbIBQ4Dv9Jan7zEfhYACwAiIyPj0tPTr7R+IRxGZmE5HydnsGJnBml5ZdKEIy6p1S6yKqX8gY+BB7TWKXXeDwRKtNaVSqm7gZ9ora9rbF/SRCPEpWmt2ZlewIqdFzfh3DLEaMKJDJQmHGfWqr1olFJ/Asq01n9vYL0LkK+1brQvmAS8EJdXXlV7vgnnu6Nn0BoSogK4ZWgYkwZ2lTtmnZC1L7IGA9Va60KllBewDnhWa726zjZdtdbZluWbgYe11omN7VcCXojmySws55MfMlmZnMHR3FLcXU2M69uZ6UPDGRUbhKuLydYlijZg7YAfCCwDXAAT8KHW+gml1BNAktZ6lVLqGWAKUAPkA/dqrQ82tl8JeCFaRmvNnowiViZnsGp3FgVl1QR1cGfq4DBuGRpG364dZXgEByY3OgnhJKpqzGw8lMPK5Ey+Onia6lpN7y6+3DI0jKmDw+jc0dPWJQork4AXwgkVlFaxem82K5Mz+OFEISYFV/cI4qaBoYzr1xl/b5mgxBFIwAvh5I7llvDxD5l8siuTk/nluJoUI2ODmDSgK+P7dsHPWy7O2isJeCEEYLTX780sYs3ebNbsySajoBw3F8XIHkHcODCUsX074+clYW9PJOCFED9y7uLsubDPLDTCflRsMDcO6MrYfp2l26UdkIAXQjRKa83ujCLW7Mli7d5TZBaW4+5iYnRPoxnn+j5yZt9eScALIZrs3JDGa/Zks3ZvNllFFbi5GHPQTuzfhbF9OxPYwcPWZQoLCXghRIuYzZrdGYV8nnKKz1JOcSK/DJOC4dGBTBrQhfH9uhAiXS9tSgJeCHHFtNbszy7m85RTrN2bzdHcUpSCuMhOTOjfhQn9uxDeScbFaWsS8EIIq0s9fZbPLGf2B7KLARgY7meEfb8uxAR3sHGFzkECXgjRqtLOlPJZyik+T8lmd0YRALEhHRjfz2jG6R8mwyW0Fgl4IUSbySwsZ92+U3yx7xTbj+dj1hDm78XYvp0Z168zCVEBMhCaFUnACyFsIr+0iq8OnOaLfafZnJpLZY2ZTt5uXN+nM+P7dWFUbBCebi62LtOuScALIWyutLKGTYdz+WLfKb46mMPZihq83Fy4pmcw4/p15pqewdL9sgWsOierEEK0hI+HKxMHdGXigK5U1ZjZdjyPL/adYt2+03y+7xRKwcAwP67pGcw1vUIYHOGPi0xLaFVyBi+EaFNmszE+zjeHc/nmcC4/nCjArMHPy42RsUGM6RnMNT2Dpb99A6SJRghhNwrLqvj2yBk2HjICP/dsJQB9unZkTC8j7OO6dcJNLtQCEvBCCDt17uaqbw7n8s2hXHamF1Bj1nTwcOWq7oGM7hnM6Nhgp554XAJeCOEQiiuq+f5IHt8czmHT4TNkFpYDEBXozajYYEb3DGZE90A6eDjP5UQJeCGEw9Fac+xMKZsP57Ip9QxbjuZRXl2Lq0kxtFsnRscGMbpnMP1D/TA58MVaCXghhMOrrKllZ3oBm1PPsOlwLvuyjOETOnm7MTI2mFGxQVzdI4gwfy8bV2pdEvBCCKdzpqSSb1PPsCk1l82pZ85frI0M8GZETCAjuhsPe5+IXAJeCOHUtNYcPHWWLUfz2HIsj23H8iiuqAEgJsiHxO6BjIgJJDEmkGBf+7rZSgJeCCHqqDVrDmQXnw/87cfzKak0Aj82pINxdh8TyPCYQAJ83G1cbeOsGvBKKU9gE+CBcefrCq31Y/W28QDeBuKAPOAnWuu0xvYrAS+EsJWaWjMpWRcCPyktn7KqWgB6d/FleHQAiTGBJEQHtLvhFKwd8Arw0VqXKKXcgG+Bh7TWW+tscx8wUGt9j1JqFnCz1vonje1XAl4I0V5U15rZk1HIlqN5bDueT1JaAeXVRuD37NyBxJhAhkcHMjwmgCAbB75Vx6LRxi9AieWlm+VR/1dhKvBny/IK4BWllNK2av8RQohmcHMxEdctgLhuAdwPVNWY2ZtZxNZjRuCv2JnB21vSAegR0oHEmIDzgR/i234v2japDV4p5QLsBHoAr2qtH663PgWYoLXOsLw+CgzXWp+pt90CYAFAZGRkXHp6ulUOQgghWlN1rZmUzCK2Hstn2/E8dhzPp9TSpBMT7MPwaCPwE6IDCG3lbpmtdpFVKeUPfAw8oLVOqfN+kwK+LmmiEULYq3Nt+NssZ/g70vI5a+mlE97Ji4ToABItgd8t0Nuqs1u12nDBWutCpdQGYAKQUmdVJhABZCilXAE/jIutQgjhcFxdTAyO8GdwhD93X9OdWrPm4Klith3LZ/vxfDYeymVlciYAIb4eJEQHMDw6gIToQGJDOrTZnbaXDXilVDBQbQl3L2As8Gy9zVYBdwBbgFuBr6X9XQjhLFxMin6hfvQL9ePOkdForTmaW8JWS+BvO57H6j3ZgHGn7X1jejB/dEyr19WUM/iuwDJLO7wJ+FBrvVop9QSQpLVeBbwJ/EcpdQTIB2a1WsVCCNHOKaXoEeJLjxBf5iZ2Q2vNyfxyth43+uB39mubC7Nyo5MQQtiR5rTBywj6QgjhoCTghRDCQUnACyGEg5KAF0IIByUBL4QQDkoCXgghHJQEvBBCOCgJeCGEcFA2u9FJKZULtHQ4ySCgwYHMnIAzH78zHzs49/HLsRu6aa2Dm/JHNgv4K6GUSmrqnVyOyJmP35mPHZz7+OXYm3/s0kQjhBAOSgJeCCEclL0G/GJbF2Bjznz8znzs4NzHL8feTHbZBi+EEOLy7PUMXgghxGVIwAshhIOyu4BXSk1QSh1SSh1RSi2ydT1tSSmVppTaq5TapZRy+NlSlFJLlVI5lkndz70XoJRar5RKtTx3smWNraWBY/+zUirT8v3vUkpNsmWNrUUpFaGU2qCU2q+U2qeUesjyvrN89w0df7O/f7tqg7dMG3gYY17YDGAHMFtrvd+mhbURpVQaEK+1doqbPZRSo4ES4G2tdX/Le38D8rXWf7X8wHfSWj9syzpbQwPH/megRGv9d1vW1tqUUl2BrlrrZKWUL7ATmAbMwzm++4aOfybN/P7t7Qw+ATiitT6mta4C3gem2rgm0Uq01psw5vitayqwzLK8DOM/fIfTwLE7Ba11ttY62bJ8FjgAhOE8331Dx99s9hbwYcDJOq8zaOGB2ykNrFNK7VRKLbB1MTbSWWudbVk+BXS2ZTE2cL9Sao+lCcchmyjqUkpFAUOAbTjhd1/v+KGZ37+9BbyzG6m1HgpMBH5h+We809JG+6L9tDFeudeA7sBgIBv4h02raWVKqQ7AR8AvtdbFddc5w3d/ieNv9vdvbwGfCUTUeR1uec8paK0zLc85wMcYTVbO5rSljfJcW2WOjetpM1rr01rrWq21GViCA3//Sik3jHB7V2u90vK203z3lzr+lnz/9hbwO4BYpVS0UsodmAWssnFNbUIp5WO54IJSygcYB6Q0/lcOaRVwh2X5DuB/NqylTZ0LN4ubcdDvXymlgDeBA1rrf9ZZ5RTffUPH35Lv36560QBYuga9ALgAS7XWT9u2orahlIrBOGsHcAXec/RjV0otB8ZgDJV6GngM+AT4EIjEGG56ptba4S5GNnDsYzD+ea6BNODuOm3SDkMpNRLYDOwFzJa3H8Foh3aG776h459NM79/uwt4IYQQTWNvTTRCCCGaSAJeCCEclAS8EEI4KAl4IYRwUBLwQgjhoCTghRDCQUnACyGEg/p/ZnUrewEAlEMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reserved-elizabeth",
   "metadata": {},
   "source": [
    "train loss : 2.8883  \n",
    "val loss : 4.0875  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proper-switch",
   "metadata": {},
   "source": [
    "* 테스트 단계에서는 정수 인덱스 행렬로 존재하던 텍스트 데이터를 실제 데이터로 복원해야 하므로, 필요한 3개의 사전을 아래와 같이 미리 준비합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "appropriate-shannon",
   "metadata": {},
   "outputs": [],
   "source": [
    "src_index_to_word = src_tokenizer.index_word # 원문 단어 집합에서 정수 -> 단어를 얻음\n",
    "tar_word_to_index = tar_tokenizer.word_index # 요약 단어 집합에서 단어 -> 정수를 얻음\n",
    "tar_index_to_word = tar_tokenizer.index_word # 요약 단어 집합에서 정수 -> 단어를 얻음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "rational-breathing",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tar_word_to_index['sostoken']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "absolute-sentence",
   "metadata": {},
   "source": [
    "* seq2seq는 훈련할 때와 실제 동작할 때(인퍼런스 단계)의 방식이 다르므로 그에 맞게 모델 설계를 별개로 진행해야 합니다. \n",
    "* 훈련 단계에서는 디코더의 입력부에 정답이 되는 문장 전체를 한꺼번에 넣고 디코더의 출력과 한 번에 비교할 수 있으므로, 인코더와 디코더를 엮은 모델 하나만 준비합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "civil-principal",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 인코더 설계\n",
    "encoder_model = Model(inputs=encoder_inputs, outputs=[encoder_outputs, state_h, state_c])\n",
    "\n",
    "# 이전 시점의 상태들을 저장하는 텐서\n",
    "decoder_state_input_h = Input(shape=(hidden_size,))\n",
    "decoder_state_input_c = Input(shape=(hidden_size,))\n",
    "\n",
    "dec_emb2 = dec_emb_layer(decoder_inputs)\n",
    "\n",
    "# 문장의 다음 단어를 예측하기 위해서 초기 상태(initial_state)를 이전 시점의 상태로 사용. 이는 뒤의 함수 decode_sequence()에 구현\n",
    "# 훈련 과정에서와 달리 LSTM의 리턴하는 은닉 상태와 셀 상태인 state_h와 state_c를 버리지 않음.\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "american-clear",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 어텐션 함수\n",
    "decoder_hidden_state_input = Input(shape=(text_max_len, hidden_size))\n",
    "attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])\n",
    "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
    "\n",
    "# 디코더의 출력층\n",
    "decoder_outputs2 = decoder_softmax_layer(decoder_inf_concat) \n",
    "\n",
    "# 최종 디코더 모델\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
    "    [decoder_outputs2] + [state_h2, state_c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "weighted-sheet",
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # 입력으로부터 인코더의 상태를 얻음\n",
    "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
    "\n",
    "     # <SOS>에 해당하는 토큰 생성\n",
    "    target_seq = np.zeros((1,1))\n",
    "    target_seq[0, 0] = tar_word_to_index['sostoken']\n",
    "\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition: # stop_condition이 True가 될 때까지 루프 반복\n",
    "\n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_token = tar_index_to_word[sampled_token_index]\n",
    "\n",
    "        if (sampled_token!='eostoken'):\n",
    "            decoded_sentence += ' '+sampled_token\n",
    "\n",
    "        #  <eos>에 도달하거나 최대 길이를 넘으면 중단.\n",
    "        if (sampled_token == 'eostoken'  or len(decoded_sentence.split()) >= (headlines_max_len-1)):\n",
    "            stop_condition = True\n",
    "\n",
    "        # 길이가 1인 타겟 시퀀스를 업데이트\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "\n",
    "        # 상태를 업데이트 합니다.\n",
    "        e_h, e_c = h, c\n",
    "\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spatial-edition",
   "metadata": {},
   "source": [
    "## 실제 결과와 요약문 비교\n",
    "* 원래의 요약문(headlines 열)과 학습을 통해 얻은 추상적 요약의 결과를 비교합니다. \n",
    "* 아래는 주어진 정수 시퀀스를 텍스트 시퀀스로 변환하는 함수입니다. \n",
    "* Text의 정수 시퀀스에서는 패딩을 위해 사용되는 숫자 0을 제외하고   \n",
    "  Summary의 정수 시퀀스에서는 숫자 0, 시작 토큰의 인덱스, 종료 토큰의 인덱스를 출력에서 제외하도록 합니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "prompt-letters",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 원문의 정수 시퀀스를 텍스트 시퀀스로 변환\n",
    "def seq2text(input_seq):\n",
    "    temp=''\n",
    "    for i in input_seq:\n",
    "        if (i!=0):\n",
    "            temp = temp + src_index_to_word[i]+' '\n",
    "    return temp\n",
    "\n",
    "# 요약문의 정수 시퀀스를 텍스트 시퀀스로 변환\n",
    "def seq2summary(input_seq):\n",
    "    temp=''\n",
    "    for i in input_seq:\n",
    "        if ((i!=0 and i!=tar_word_to_index['sostoken']) and i!=tar_word_to_index['eostoken']):\n",
    "            temp = temp + tar_index_to_word[i] + ' '\n",
    "    return temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "otherwise-flash",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "원문 : samajwadi party legislator ali monday said muslims first indians later anything islam whether constitution law islam ali said stated muslims follow government order sing national anthem hoist tricolour independence day state \n",
      "실제 요약 : we are muslims first indians later sp mla ali \n",
      "예측 요약 :  rss should be like muslims sp leader\n",
      "\n",
      "\n",
      "원문 : haryana health minister anil vij saturday tweeted even dogs join hands cannot challenge lion bjp win gujarat polls narendra modi wave continues country vij added comes reports claimed patidar leader hardik patel join hands congress gujarat \n",
      "실제 요약 : dogs cannot challenge lion bjp will win polls minister \n",
      "예측 요약 :  haryana minister slams media for hrs over video\n",
      "\n",
      "\n",
      "원문 : actor shia spending month isolated cabin finland part exhibition titled communication outside world via text messages visitors helsinki museum also features partners turner separate cabins cannot talk among \n",
      "실제 요약 : shia to month in finland cabin \n",
      "예측 요약 :  restaurant offers to sell for marijuana\n",
      "\n",
      "\n",
      "원문 : aadhaar governing body uidai sunday said helpline number mobile contact list data stored mobile phone cannot stolen comes google admitted inadvertently coding uidai helpline number android release panic delete number harm caused uidai added \n",
      "실제 요약 : helpline number in mobile contact list cannot steal data uidai \n",
      "예측 요약 :  aadhaar database aadhaar card to prevent id uidai\n",
      "\n",
      "\n",
      "원문 : groom named sanjay became first dalit lead baraat village family protected police officers local allegedly threatened disrupt procession district magistrate said took five months prepare day bride said faced huge obstacles \n",
      "실제 요약 : cops provide security as dalit groom leads in up \n",
      "예측 요약 :  video shows srk as vandalised in rajasthan\n",
      "\n",
      "\n",
      "원문 : lancashire captain liam wore pad left hand protect broken thumb batting number second innings yorkshire county championship today fractured thumb fielding earlier match however could face ball lancashire got two minutes came crease \n",
      "실제 요약 : batsman wears pad on hand to protect broken \n",
      "예측 요약 :  fans slam barcelona to score matches in ipl\n",
      "\n",
      "\n",
      "원문 : cops punjab filed case police station alleged theft electricity notably rules mandate fir lodged living entities punjab state power corporation limited official said surprised charge booked instead police station \n",
      "실제 요약 : cops file case against police station over electricity theft \n",
      "예측 요약 :  police booked for smuggling in police\n",
      "\n",
      "\n",
      "원문 : according reports actor aamir khan cricketer virat kohli appear together diwali special chat show show said promotion aamir upcoming film secret superstar reports added show aamir asked virat share one thing rumoured girlfriend anushka sharma one \n",
      "실제 요약 : aamir virat to appear together on tv show reports \n",
      "예측 요약 :  kohli to get married to pakistan reports\n",
      "\n",
      "\n",
      "원문 : power minister piyush goyal sunday said india aims sell electric cars country going introduce electric vehicles big way single petrol diesel car sold country said goyal added electric car industry driven demand subsidy \n",
      "실제 요약 : india to sell only electric cars by power minister \n",
      "예측 요약 :  govt to develop electric cars in electric cars minister\n",
      "\n",
      "\n",
      "원문 : arrested inx media money laundering case ex finance minister chidambaram son karti told delhi court hindustan like others hindustan arrested chennai airport returned london wednesday court sent karti one day cbi custody cooperating investigation \n",
      "실제 요약 : am not hindustan karti chidambaram after arrest \n",
      "예측 요약 :  cbi arrests chidambaram over bail in case\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(110, 120):\n",
    "    print(\"원문 :\", seq2text(encoder_input_test[i]))\n",
    "    print(\"실제 요약 :\", seq2summary(decoder_input_test[i]))\n",
    "    print(\"예측 요약 :\", decode_sequence(encoder_input_test[i].reshape(1, text_max_len)))\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funded-fever",
   "metadata": {},
   "source": [
    "50개 정도 데이터의 실제 요약과 예측 요약을 비교해 본 결과, 대부분의 경우,   \n",
    "문맥이 맞지 않거나, 전체 내용을 포괄하지 못하는 일부의 내용을 가져오거나, 엉뚱한 내용요약을 하는 등 많이 미흡해보입니다.   \n",
    "\n",
    "[엉뚱한 내용요약 사례]  \n",
    "원문은 '뇌사 소년의 부모가 소년의 장기를 기증하기로 승인한 이후에 소년이 다시 깨어났다'는 기사인데, '심장병으로 소년이 죽었다'라는 엉뚱한 내용을 예측했습니다.   \n",
    "\n",
    "\n",
    "        실제 요약 : brain dead boy wakes up after parents approve donation   \n",
    "        예측 요약 :  year old boy dies after heart attack on\n",
    "\n",
    "다음으로 비교적 난이도가 낮은 추출적 요약을 진행해보겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "innocent-encoding",
   "metadata": {},
   "source": [
    "# 2. 추출적 요약\n",
    "* 동일한 데이터에 대해 Summa을 이용해서 추출적 요약을 해보겠습니다. \n",
    "* 추상적 요약은 추출적 요약과는 달리 문장의 표현력을 다양하게 가져갈 수 있지만, 추출적 요약에 비해서 난이도가 높은 반면,\n",
    "* 추출적 요약은 추상적 요약에 비해 난이도가 낮고 기존 문장에서 문장을 꺼내오는 것이므로 잘못된 요약이 나올 가능성이 낮습니다.  \n",
    "* Summa의 summarize는 문장 토큰화를 별도로 하지 않더라도 내부적으로 문장 토큰화를 수행하기 때문에 문장 구분이 되어 있지 않는 원문을 그래로 입력값으로 넣어줄 수 있습니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "collectible-headquarters",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "summa                         1.2.0\n",
      "\u001b[33mWARNING: You are using pip version 20.3.3; however, version 21.2.4 is available.\n",
      "You should consider upgrading via the '/opt/conda/bin/python3.7 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "! pip list | grep summa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "absolute-macro",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from summa.summarizer import summarize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "after-poker",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>text</th>\n",
       "      <th>news_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>upGrad learner switches to career in ML &amp; Al w...</td>\n",
       "      <td>Saurav Kant, an alumnus of upGrad and IIIT-B's...</td>\n",
       "      <td>upGrad's Online Power Learning has powered 3 l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Delhi techie wins free food from Swiggy for on...</td>\n",
       "      <td>Kunal Shah's credit card bill payment platform...</td>\n",
       "      <td>Users get one CRED coin per rupee of bill paid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New Zealand end Rohit Sharma-led India's 12-ma...</td>\n",
       "      <td>New Zealand defeated India by 8 wickets in the...</td>\n",
       "      <td>The match witnessed India getting all out for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aegon life iTerm insurance plan helps customer...</td>\n",
       "      <td>With Aegon Life iTerm Insurance plan, customer...</td>\n",
       "      <td>Also, customers have options to insure against...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Have known Hirani for yrs, what if MeToo claim...</td>\n",
       "      <td>Speaking about the sexual harassment allegatio...</td>\n",
       "      <td>Speaking about the sexual harassment allegatio...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           headlines  \\\n",
       "0  upGrad learner switches to career in ML & Al w...   \n",
       "1  Delhi techie wins free food from Swiggy for on...   \n",
       "2  New Zealand end Rohit Sharma-led India's 12-ma...   \n",
       "3  Aegon life iTerm insurance plan helps customer...   \n",
       "4  Have known Hirani for yrs, what if MeToo claim...   \n",
       "\n",
       "                                                text  \\\n",
       "0  Saurav Kant, an alumnus of upGrad and IIIT-B's...   \n",
       "1  Kunal Shah's credit card bill payment platform...   \n",
       "2  New Zealand defeated India by 8 wickets in the...   \n",
       "3  With Aegon Life iTerm Insurance plan, customer...   \n",
       "4  Speaking about the sexual harassment allegatio...   \n",
       "\n",
       "                                        news_summary  \n",
       "0  upGrad's Online Power Learning has powered 3 l...  \n",
       "1  Users get one CRED coin per rupee of bill paid...  \n",
       "2  The match witnessed India getting all out for ...  \n",
       "3  Also, customers have options to insure against...  \n",
       "4  Speaking about the sexual harassment allegatio...  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "urllib.request.urlretrieve(\"https://raw.githubusercontent.com/sunnysai12345/News_Summary/master/news_summary_more.csv\", filename=\"news_summary_more.csv\")\n",
    "data = pd.read_csv('news_summary_more.csv', encoding='iso-8859-1')\n",
    "data['news_summary'] = data['text'].apply(lambda x : summarize(x, ratio = 0.5))\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "assigned-thesaurus",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headlines</th>\n",
       "      <th>news_summary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>upGrad learner switches to career in ML &amp; Al with 90% salary hike</td>\n",
       "      <td>upGrad's Online Power Learning has powered 3 lakh+ careers.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Delhi techie wins free food from Swiggy for one year on CRED</td>\n",
       "      <td>Users get one CRED coin per rupee of bill paid, which can be used to avail rewards from brands like Ixigo, BookMyShow, UberEats, Cult.Fit and more.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New Zealand end Rohit Sharma-led India's 12-match winning streak</td>\n",
       "      <td>The match witnessed India getting all out for 92, their seventh lowest total in ODI cricket history.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aegon life iTerm insurance plan helps customers save tax</td>\n",
       "      <td>Also, customers have options to insure against Critical Illnesses, Disability and Accidental Death Benefit Rider with a life cover up to the age of 80 years.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Have known Hirani for yrs, what if MeToo claims are not true: Sonam</td>\n",
       "      <td>Speaking about the sexual harassment allegations against Rajkumar Hirani, Sonam Kapoor said, \"I've known Hirani for many years...What if it's not true, the [#MeToo] movement will get derailed.\" \"In the #MeToo movement, I always believe a woman.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Rahat Fateh Ali Khan denies getting notice for smuggling currency</td>\n",
       "      <td>Pakistani singer Rahat Fateh Ali Khan has denied receiving any notice from the Enforcement Directorate over allegedly smuggling foreign currency out of India.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>India get all out for 92, their lowest ODI total in New Zealand</td>\n",
       "      <td>India's previous lowest ODI total in New Zealand was 108.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Govt directs Alok Verma to join work 1 day before his retirement</td>\n",
       "      <td>Weeks after ex-CBI Director Alok Verma told the Department of Personnel and Training to consider him retired, the Home Ministry asked him to join work on the last day of his fixed tenure as Director on Thursday.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Called PM Modi 'sir' 10 times to satisfy his ego: Andhra CM</td>\n",
       "      <td>Andhra Pradesh CM N Chandrababu Naidu has said, \"When I met then US President Bill Clinton, I addressed him as Mr Clinton, not as 'sir'.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Cong wins Ramgarh bypoll in Rajasthan, takes total to 100 seats</td>\n",
       "      <td>Congress candidate Shafia Zubair won the Ramgarh Assembly seat in Rajasthan, by defeating BJP's Sukhwant Singh with a margin of 12,228 votes in the bypoll.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                             headlines  \\\n",
       "0    upGrad learner switches to career in ML & Al with 90% salary hike   \n",
       "1         Delhi techie wins free food from Swiggy for one year on CRED   \n",
       "2     New Zealand end Rohit Sharma-led India's 12-match winning streak   \n",
       "3             Aegon life iTerm insurance plan helps customers save tax   \n",
       "4  Have known Hirani for yrs, what if MeToo claims are not true: Sonam   \n",
       "5    Rahat Fateh Ali Khan denies getting notice for smuggling currency   \n",
       "6      India get all out for 92, their lowest ODI total in New Zealand   \n",
       "7     Govt directs Alok Verma to join work 1 day before his retirement   \n",
       "8          Called PM Modi 'sir' 10 times to satisfy his ego: Andhra CM   \n",
       "9      Cong wins Ramgarh bypoll in Rajasthan, takes total to 100 seats   \n",
       "\n",
       "                                                                                                                                                                                                                                           news_summary  \n",
       "0                                                                                                                                                                                           upGrad's Online Power Learning has powered 3 lakh+ careers.  \n",
       "1                                                                                                   Users get one CRED coin per rupee of bill paid, which can be used to avail rewards from brands like Ixigo, BookMyShow, UberEats, Cult.Fit and more.  \n",
       "2                                                                                                                                                  The match witnessed India getting all out for 92, their seventh lowest total in ODI cricket history.  \n",
       "3                                                                                         Also, customers have options to insure against Critical Illnesses, Disability and Accidental Death Benefit Rider with a life cover up to the age of 80 years.  \n",
       "4  Speaking about the sexual harassment allegations against Rajkumar Hirani, Sonam Kapoor said, \"I've known Hirani for many years...What if it's not true, the [#MeToo] movement will get derailed.\" \"In the #MeToo movement, I always believe a woman.  \n",
       "5                                                                                        Pakistani singer Rahat Fateh Ali Khan has denied receiving any notice from the Enforcement Directorate over allegedly smuggling foreign currency out of India.  \n",
       "6                                                                                                                                                                                             India's previous lowest ODI total in New Zealand was 108.  \n",
       "7                                   Weeks after ex-CBI Director Alok Verma told the Department of Personnel and Training to consider him retired, the Home Ministry asked him to join work on the last day of his fixed tenure as Director on Thursday.  \n",
       "8                                                                                                              Andhra Pradesh CM N Chandrababu Naidu has said, \"When I met then US President Bill Clinton, I addressed him as Mr Clinton, not as 'sir'.  \n",
       "9                                                                                           Congress candidate Shafia Zubair won the Ramgarh Assembly seat in Rajasthan, by defeating BJP's Sukhwant Singh with a margin of 12,228 votes in the bypoll.  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', None)\n",
    "data[['headlines','news_summary']][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "humanitarian-caution",
   "metadata": {},
   "source": [
    "원문 안의 단어를 추출하여 문장을 생성했기 때문에, 대부분의 출력문장이 (학습이 미흡한)추상적 요약에 비해 맥락에서 벗어나지 않아보입니다.    \n",
    "그러나 새로운 문장을 생성할 수 있는 능력이 없기 때문에, 핵심문장을 센스있게 추상적 단어로 표현하지 못하고 단순한 사실의 나열이 되는 모습도 보입니다.    \n",
    "\n",
    "그럼에도 불구하고, 추상적 요약을 성공적으로 학습시킬만한 적절한 데이터와 시간과 같은 자원이 충분하지 않은 경우, 비용과 성능의 측면을 고려했을 때, 오히려 간단한 추출적 요약 기법이 낫다고 생각됩니다. \n",
    "\n",
    "추상적 요약 모델과 텍스트 요약 모델을 문법의 완성도, 핵심단어의 포함측면으로 비교평가해볼때, 텍스트요약의 성능이 낫다고 판단되기 때문입니다. \n",
    "\n",
    "추상적 요약 모델의 성능을 개선하는 방법으로는\n",
    "* seq2seq와 어텐션의 자체의 조합을 좀 더 좋게 수정하는 방법\n",
    "* 빔 서치(beam search)\n",
    "* 사전 훈련된 워드 임베딩(pre-trained word embedding)\n",
    "* 인코더 - 디코더 자체의 구조를 새로이 변경한 하는 트랜스포머(Transformer)\n",
    "\n",
    "와 같은 여러 개선 방안들이 존재합니다. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}