chatbot_ner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition\n",
    "\n",
    "NER is a subtask of information extraction that seeks to locate and classify named entities in text into predefined categories such as the name of a person, location, time, quantity, etc.\n",
    "\n",
    "Chatbot NER is heuristic based that uses several NLP techniques to extract necessary entities from chat interface. In Chatbot, there are several entities that need to be identified and each entity has to be distinguished based on its type as a different entity has different detection logic.\n",
    "\n",
    "For the first version, we will only support date, time, time_with_range, number and email detector using SageMaker. We will add support for other detectors and languages on SageMaker soon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install boto3 sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import boto3\n",
    "import sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Consider using IAM roles instead\n",
    "\n",
    "AWS_ACCESS_KEY = os.environ.get('AWS_ACCESS_KEY')\n",
    "AWS_SECRET_KEY = os.environ.get('AWS_SECRET_KEY')\n",
    "AWS_REGION = os.environ.get('AWS_REGION')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "boto_session = boto3.session.Session(aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_KEY, region_name=AWS_REGION)\n",
    "sagemaker_session = sagemaker.session.Session(boto_session=boto_session)\n",
    "real_time_predictor = sagemaker.predictor.RealTimePredictor(endpoint='chatbot-ner', sagemaker_session=sagemaker_session, content_type='application/json', accept='application/json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Date Detection\n",
    "\n",
    "Date Detector allows you to detect date. Use the timezone parameter to pass your current timezone to date detection. Check pytz.all_timezones for the list of all valid timezone values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'detection': 'message', 'original_text': '25th december', 'entity_value': {'end_range': False, 'from': True, 'normal': False, 'value': {'mm': 12, 'yy': 2018, 'dd': 25, 'type': 'date'}, 'to': False, 'start_range': False}}]}\n"
     ]
    }
   ],
   "source": [
    "date_detector_payload = {\n",
    "    \"message\": \"Set a reminder for 25th December\", \n",
    "    \"entity_name\": \"date\", \n",
    "    \"entity_type\": \"date\", \n",
    "    \"timezone\": \"UTC\"\n",
    "}\n",
    "date_predictions = real_time_predictor.predict(json.dumps(date_detector_payload))\n",
    "date_predictions = json.loads(date_predictions)\n",
    "print(date_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time Detection\n",
    "\n",
    "Time Detector allows you to detect time. Use the timezone parameter to pass your current timezone to time detection. Check pytz.all_timezones for the list of all valid timezone values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'detection': 'message', 'original_text': '12:30 pm', 'entity_value': {'mm': 30, 'hh': 12, 'nn': 'pm'}, 'language': 'en'}, {'detection': 'message', 'original_text': 'in 15 mins', 'entity_value': {'mm': '15', 'hh': 0, 'nn': 'df'}, 'language': 'en'}, {'detection': 'message', 'original_text': '13:50', 'entity_value': {'mm': 50, 'hh': 13, 'nn': 'hrs'}, 'language': 'en'}]}\n"
     ]
    }
   ],
   "source": [
    "time_detector_payload =  {\n",
    "    \"message\": \"John arrived at the bus stop at 13:50 hrs, expecting the bus to be there in 15 mins. But the bus was scheduled for 12:30 pm\", \n",
    "    \"entity_name\": \"time\", \n",
    "    \"entity_type\": \"time\", \n",
    "    \"timezone\": \"UTC\"\n",
    "}\n",
    "time_predictions = real_time_predictor.predict(json.dumps(time_detector_payload))\n",
    "time_predictions = json.loads(time_predictions)\n",
    "print(time_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time With Range Detection\n",
    "\n",
    "Time with range Detector allows you to detect time ranges. Use the timezone parameter to pass your current timezone to time detection. Check pytz.all_timezones for the list of all valid timezone values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'detection': 'message', 'original_text': '7:00 am to 6:00 pm', 'entity_value': {'mm': 0, 'hh': 7, 'range': 'start', 'nn': 'am', 'time_type': None}, 'language': 'en'}, {'detection': 'message', 'original_text': '7:00 am to 6:00 pm', 'entity_value': {'mm': 0, 'hh': 6, 'range': 'end', 'nn': 'pm', 'time_type': None}, 'language': 'en'}]}\n"
     ]
    }
   ],
   "source": [
    "time_with_range_detector_payload = {\n",
    "    \"message\": \"Set a drink water reminder for tomorrow from 7:00 AM to 6:00 PM\", \n",
    "    \"entity_name\": \"time_with_range\", \n",
    "    \"entity_type\": \"time_with_range\", \n",
    "    \"timezone\": \"UTC\"\n",
    "}\n",
    "time_with_range_predictions = real_time_predictor.predict(json.dumps(time_with_range_detector_payload))\n",
    "time_with_range_predictions = json.loads(time_with_range_predictions)\n",
    "print(time_with_range_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number Detection\n",
    "\n",
    "Number Detector allows you to detector numbers or numerals. You can configure min_digit and max_digit as per your requirement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'detection': 'message', 'original_text': 'three', 'entity_value': {'value': '3'}, 'language': 'en'}]}\n"
     ]
    }
   ],
   "source": [
    "number_detector_payload = {\n",
    "    \"message\": \"I want to reserve a table for three people\", \n",
    "    \"entity_name\": \"number_of_people\", \n",
    "    \"entity_type\": \"number\", \n",
    "    \"min_digit\": 1, \n",
    "    \"max_digit\": 2\n",
    "}\n",
    "number_predictions = real_time_predictor.predict(json.dumps(number_detector_payload))\n",
    "number_predictions = json.loads(number_predictions)\n",
    "print(number_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Email Detection\n",
    "\n",
    "Email Detector allows you to detect email ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': [{'detection': 'message', 'original_text': 'hello@haptik.ai', 'entity_value': {'value': 'hello@haptik.ai'}, 'language': 'en'}]}\n"
     ]
    }
   ],
   "source": [
    "email_detector_payload = {\n",
    "    \"message\": \"my email id is hello@haptik.ai\", \n",
    "    \"entity_name\": \"email\", \n",
    "    \"entity_type\": \"email\"\n",
    "}\n",
    "email_predictions = real_time_predictor.predict(json.dumps(email_detector_payload))\n",
    "email_predictions = json.loads(email_predictions)\n",
    "print(email_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "haptik_sagemaker",
   "language": "python",
   "name": "haptik_sagemaker"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}