Topic Modeling _ Extracting Topics_ Using Genism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for text preprocessing\n",
    "import re\n",
    "import spacy\n",
    "\n",
    "from nltk.corpus import stopwords \n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "import string\n",
    "\n",
    "# import numpy for matrix operation\n",
    "import numpy as np\n",
    "\n",
    "# Importing Gensim\n",
    "import gensim\n",
    "from gensim import corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to suppress warnings\n",
    "from warnings import filterwarnings\n",
    "filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "D1 = 'I want to watch a movie this weekend.'\n",
    "D2 =  'I went shopping yesterday. New Zealand won the World Test Championship by beating India by eight wickets at Southampton.'\n",
    "D3 =  'I don’t watch cricket. Netflix and Amazon Prime have very good movies to watch.'\n",
    "D4 =  'Movies are a nice way to chill however, this time I would like to paint and read some good books. It’s been long!'\n",
    "D5 =  'This blueberry milkshake is so good! Try reading Dr. Joe Dispenza’s books. His work is such a game-changer! His books helped to learn so much about how our thoughts impact our biology and how we can all rewire our brains.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combining all the documents into a list:\n",
    "\n",
    "corpus = [D1, D2, D3, D4, D5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "executionInfo": {
     "elapsed": 6177,
     "status": "ok",
     "timestamp": 1603266387620,
     "user": {
      "displayName": "Aishwarya Singh",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgeJwfn4BdBDCAplWi_kdtB9FRssOpXO7T_aMgg=s64",
      "userId": "01105858832371513140"
     },
     "user_tz": -330
    },
    "id": "fNATC-NcaVrt"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I want to watch a movie this weekend.',\n",
       " 'I went shopping yesterday. New Zealand won the World Test Championship by beating India by eight wickets at Southampton.',\n",
       " 'I don’t watch cricket. Netflix and Amazon Prime have very good movies to watch.',\n",
       " 'Movies are a nice way to chill however, this time I would like to paint and read some good books. It’s been long!',\n",
       " 'This blueberry milkshake is so good! Try reading Dr. Joe Dispenza’s books. His work is such a game-changer! His books helped to learn so much about how our thoughts impact our biology and how we can all rewire our brains.']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the complete corpus as below:\n",
    "\n",
    "corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Text Preprocessing\n",
    "\n",
    "Steps to preprocess text data:\n",
    "\n",
    "1. Convert the text into lowercase\n",
    "2. Split text into words\n",
    "3. Remove the stop loss words\n",
    "3. Remove the Punctuation, any symbols and special characters\n",
    "4. Normalize the word (I'll be using Lemmatization for normalization)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply Preprocessing on the Corpus\n",
    "\n",
    "# stop loss words \n",
    "stop = set(stopwords.words('english'))\n",
    "\n",
    "# punctuation \n",
    "exclude = set(string.punctuation) \n",
    "\n",
    "# lemmatization\n",
    "lemma = WordNetLemmatizer() \n",
    "\n",
    "# One function for all the steps:\n",
    "def clean(doc):\n",
    "    \n",
    "    # convert text into lower case + split into words\n",
    "    stop_free = \" \".join([i for i in doc.lower().split() if i not in stop])\n",
    "    \n",
    "    # remove any stop words present\n",
    "    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)  \n",
    "    \n",
    "    # remove punctuations + normalize the text\n",
    "    normalized = \" \".join(lemma.lemmatize(word) for word in punc_free.split())  \n",
    "    return normalized\n",
    "\n",
    "# clean data stored in a new list\n",
    "clean_corpus = [clean(doc).split() for doc in corpus]   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['want', 'watch', 'movie', 'weekend'],\n",
       " ['went',\n",
       "  'shopping',\n",
       "  'yesterday',\n",
       "  'new',\n",
       "  'zealand',\n",
       "  'world',\n",
       "  'test',\n",
       "  'championship',\n",
       "  'beating',\n",
       "  'india',\n",
       "  'eight',\n",
       "  'wicket',\n",
       "  'southampton'],\n",
       " ['don’t',\n",
       "  'watch',\n",
       "  'cricket',\n",
       "  'netflix',\n",
       "  'amazon',\n",
       "  'prime',\n",
       "  'good',\n",
       "  'movie',\n",
       "  'watch'],\n",
       " ['movie',\n",
       "  'nice',\n",
       "  'way',\n",
       "  'chill',\n",
       "  'however',\n",
       "  'time',\n",
       "  'would',\n",
       "  'like',\n",
       "  'paint',\n",
       "  'read',\n",
       "  'good',\n",
       "  'book',\n",
       "  'it’s',\n",
       "  'long'],\n",
       " ['blueberry',\n",
       "  'milkshake',\n",
       "  'good',\n",
       "  'try',\n",
       "  'reading',\n",
       "  'dr',\n",
       "  'joe',\n",
       "  'dispenza’s',\n",
       "  'book',\n",
       "  'work',\n",
       "  'gamechanger',\n",
       "  'book',\n",
       "  'helped',\n",
       "  'learn',\n",
       "  'much',\n",
       "  'thought',\n",
       "  'impact',\n",
       "  'biology',\n",
       "  'rewire',\n",
       "  'brain']]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Creating Document Term Matrix\n",
    "\n",
    "Using gensim for Document Term Matrix(DTM), we don't need to create the DTM matrix from scratch explicitly. The gensim library has internal mechanism to create the DTM.\n",
    "\n",
    "The only requirement for gensis package is we need to pass the cleaned data in the form of tokenized words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(52 unique tokens: ['movie', 'want', 'watch', 'weekend', 'beating']...)\n"
     ]
    }
   ],
   "source": [
    "# Creating the term dictionary of our courpus that is of all the words (Sepcific to Genism syntax perspective), \n",
    "# where every unique term is assigned an index. \n",
    "\n",
    "dict_ = corpora.Dictionary(clean_corpus)\n",
    "\n",
    "print(dict_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movie\n",
      "want\n",
      "watch\n",
      "weekend\n",
      "beating\n",
      "championship\n",
      "eight\n",
      "india\n",
      "new\n",
      "shopping\n",
      "southampton\n",
      "test\n",
      "went\n",
      "wicket\n",
      "world\n",
      "yesterday\n",
      "zealand\n",
      "amazon\n",
      "cricket\n",
      "don’t\n",
      "good\n",
      "netflix\n",
      "prime\n",
      "book\n",
      "chill\n",
      "however\n",
      "it’s\n",
      "like\n",
      "long\n",
      "nice\n",
      "paint\n",
      "read\n",
      "time\n",
      "way\n",
      "would\n",
      "biology\n",
      "blueberry\n",
      "brain\n",
      "dispenza’s\n",
      "dr\n",
      "gamechanger\n",
      "helped\n",
      "impact\n",
      "joe\n",
      "learn\n",
      "milkshake\n",
      "much\n",
      "reading\n",
      "rewire\n",
      "thought\n",
      "try\n",
      "work\n"
     ]
    }
   ],
   "source": [
    "# The dictionary had 52 unqiue words in the cleaned corpus.\n",
    "\n",
    "for i in dict_.values():\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the next step is to convert the corpus (the list of documents) into a document-term Matrix using the dictionary that we had prepared above. (The vectorizer used here is the Bag of Words).\t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[(0, 1), (1, 1), (2, 1), (3, 1)],\n",
       " [(4, 1),\n",
       "  (5, 1),\n",
       "  (6, 1),\n",
       "  (7, 1),\n",
       "  (8, 1),\n",
       "  (9, 1),\n",
       "  (10, 1),\n",
       "  (11, 1),\n",
       "  (12, 1),\n",
       "  (13, 1),\n",
       "  (14, 1),\n",
       "  (15, 1),\n",
       "  (16, 1)],\n",
       " [(0, 1), (2, 2), (17, 1), (18, 1), (19, 1), (20, 1), (21, 1), (22, 1)],\n",
       " [(0, 1),\n",
       "  (20, 1),\n",
       "  (23, 1),\n",
       "  (24, 1),\n",
       "  (25, 1),\n",
       "  (26, 1),\n",
       "  (27, 1),\n",
       "  (28, 1),\n",
       "  (29, 1),\n",
       "  (30, 1),\n",
       "  (31, 1),\n",
       "  (32, 1),\n",
       "  (33, 1),\n",
       "  (34, 1)],\n",
       " [(20, 1),\n",
       "  (23, 2),\n",
       "  (35, 1),\n",
       "  (36, 1),\n",
       "  (37, 1),\n",
       "  (38, 1),\n",
       "  (39, 1),\n",
       "  (40, 1),\n",
       "  (41, 1),\n",
       "  (42, 1),\n",
       "  (43, 1),\n",
       "  (44, 1),\n",
       "  (45, 1),\n",
       "  (46, 1),\n",
       "  (47, 1),\n",
       "  (48, 1),\n",
       "  (49, 1),\n",
       "  (50, 1),\n",
       "  (51, 1)]]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converting list of documents (corpus) into Document Term Matrix using the dictionary \n",
    "doc_term_matrix = [dict_.doc2bow(i) for i in clean_corpus]\n",
    "doc_term_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output implies: \n",
    "\n",
    "* Document wise we have the index of the word and its frequency.\n",
    "* The 0th word is repeated 1 time, then the 1st word repeated 1 and so on ...\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Implementation of LDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the object for LDA model using gensim library\n",
    "\n",
    "Lda = gensim.models.ldamodel.LdaModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running and Training LDA model on the document term matrix.\n",
    "\n",
    "ldamodel = Lda(doc_term_matrix, num_topics=6, id2word = dict_, passes=1, random_state=0, eval_every=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.136*\"watch\" + 0.084*\"movie\" + 0.060*\"good\" + 0.060*\"amazon\" + 0.060*\"netflix\" + 0.060*\"prime\" + 0.060*\"cricket\" + 0.060*\"don’t\" + 0.029*\"want\" + 0.027*\"weekend\"'),\n",
       " (1,\n",
       "  '0.074*\"weekend\" + 0.070*\"want\" + 0.065*\"movie\" + 0.063*\"watch\" + 0.015*\"don’t\" + 0.015*\"good\" + 0.015*\"book\" + 0.015*\"cricket\" + 0.015*\"prime\" + 0.015*\"new\"'),\n",
       " (2,\n",
       "  '0.052*\"book\" + 0.028*\"good\" + 0.028*\"blueberry\" + 0.028*\"try\" + 0.028*\"helped\" + 0.028*\"reading\" + 0.028*\"joe\" + 0.028*\"work\" + 0.028*\"dispenza’s\" + 0.028*\"brain\"'),\n",
       " (3,\n",
       "  '0.020*\"championship\" + 0.020*\"wicket\" + 0.020*\"southampton\" + 0.020*\"yesterday\" + 0.020*\"went\" + 0.020*\"india\" + 0.020*\"shopping\" + 0.020*\"new\" + 0.020*\"world\" + 0.020*\"zealand\"'),\n",
       " (4,\n",
       "  '0.051*\"movie\" + 0.051*\"book\" + 0.051*\"paint\" + 0.051*\"long\" + 0.051*\"like\" + 0.051*\"read\" + 0.051*\"time\" + 0.051*\"chill\" + 0.051*\"would\" + 0.051*\"however\"'),\n",
       " (5,\n",
       "  '0.019*\"weekend\" + 0.019*\"watch\" + 0.019*\"movie\" + 0.019*\"good\" + 0.019*\"want\" + 0.019*\"don’t\" + 0.019*\"book\" + 0.019*\"cricket\" + 0.019*\"prime\" + 0.019*\"new\"')]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prints the topics with the indexes: 0,1,2 :\n",
    "\n",
    "ldamodel.print_topics()\n",
    "\n",
    "# we need to manually check whethere the topics are different from one another or not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above output means: each of the 52 unique words are given weights based on the topics.This implies which of the words dominate the topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4a. Extracting Topics from the Corpus "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.136*\"watch\" + 0.084*\"movie\" + 0.060*\"good\" + 0.060*\"amazon\" + 0.060*\"netflix\"'), (1, '0.074*\"weekend\" + 0.070*\"want\" + 0.065*\"movie\" + 0.063*\"watch\" + 0.015*\"don’t\"'), (2, '0.052*\"book\" + 0.028*\"good\" + 0.028*\"blueberry\" + 0.028*\"try\" + 0.028*\"helped\"'), (3, '0.020*\"championship\" + 0.020*\"wicket\" + 0.020*\"southampton\" + 0.020*\"yesterday\" + 0.020*\"went\"'), (4, '0.051*\"movie\" + 0.051*\"book\" + 0.051*\"paint\" + 0.051*\"long\" + 0.051*\"like\"'), (5, '0.019*\"weekend\" + 0.019*\"watch\" + 0.019*\"movie\" + 0.019*\"good\" + 0.019*\"want\"')]\n"
     ]
    }
   ],
   "source": [
    "print(ldamodel.print_topics(num_topics=6, num_words=5))\n",
    "\n",
    "# num_topics mean: how many topics want to extract \n",
    "# num_words: the number of words that want per topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This return following six topics (indexed from 0,1,2,3,4,5) with the five words in each of the topics along with their respective weights."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "[(0, '0.136*\"watch\" + 0.084*\"movie\" + 0.060*\"good\" + 0.060*\"amazon\" + 0.060*\"netflix\"'), \n",
    " (1, '0.074*\"weekend\" + 0.070*\"want\" + 0.065*\"movie\" + 0.063*\"watch\" + 0.015*\"don’t\"'), \n",
    " (2, '0.052*\"book\" + 0.028*\"good\" + 0.028*\"blueberry\" + 0.028*\"try\" + 0.028*\"helped\"'), \n",
    " (3, '0.020*\"championship\" + 0.020*\"wicket\" + 0.020*\"southampton\" + 0.020*\"yesterday\" + 0.020*\"went\"'), \n",
    " (4, '0.051*\"movie\" + 0.051*\"book\" + 0.051*\"paint\" + 0.051*\"long\" + 0.051*\"like\"'), \n",
    " (5, '0.019*\"weekend\" + 0.019*\"watch\" + 0.019*\"movie\" + 0.019*\"good\" + 0.019*\"want\"')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4b. Assigning the topics to the documents\n",
    "\n",
    "Pass the doc_term_matrix to the LDA model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "doc :  0 [(0, 0.28222796), (1, 0.5843408), (2, 0.033334), (3, 0.033336144), (4, 0.033424955), (5, 0.033336177)]\n",
      "doc :  1 [(0, 0.011905565), (1, 0.011906154), (2, 0.94046915), (3, 0.011907186), (4, 0.011905455), (5, 0.011906532)]\n",
      "doc :  2 [(0, 0.91660583), (1, 0.016687142), (2, 0.016676303), (3, 0.01666756), (4, 0.016695531), (5, 0.016667573)]\n",
      "doc :  3 [(0, 0.011138699), (1, 0.011119543), (2, 0.01112717), (3, 0.011111963), (4, 0.94439065), (5, 0.011111974)]\n",
      "doc :  4 [(2, 0.9602892)]\n"
     ]
    }
   ],
   "source": [
    "# printing the topic associations with the documents\n",
    "count = 0\n",
    "for i in ldamodel[doc_term_matrix]:\n",
    "    print(\"doc : \",count,i)\n",
    "    count += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion:\n",
    "\n",
    "The five documents are assigned the topics with the weightage that will help to tell which is the dominant topic for the respective document.\n",
    "\n",
    "From above can see:\n",
    "1. Document 1 has the highest weight of 58.4% for Topic 2.\n",
    "2. Topic 3 dominates the document 2 having the weightage of 94%. Similarly, 1st topic is the main topic for document 3 with ~92% weight.\n",
    "3. Document 3 is influenced by the Topic 5 with 94% and Topic 4 rules the document 5. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------------------"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "3. Interpreting Patterns from Text - Topic Modelling.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}