10-A-Sentiment-Analysis-Sample-Script_by-Rachel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2bc9f0cf",
   "metadata": {},
   "source": [
    "# How to do basic sentiment analysis using BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf4f9f1",
   "metadata": {},
   "source": [
    "### Step 1: Import necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43a9f6f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for webscraping\n",
    "import requests\n",
    "from bs4 import BeautifulSoup \n",
    "\n",
    "# for text processing\n",
    "import nltk\n",
    "from nltk import sent_tokenize, word_tokenize\n",
    "nltk.download('punkt')\n",
    "\n",
    "# for the language model\n",
    "# you might need to install transformers first using either !pip install transformers or Qt Console in Anaconda Navigator\n",
    "from transformers import BertTokenizer, BertForSequenceClassification\n",
    "\n",
    "# for formatting\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64367748",
   "metadata": {},
   "source": [
    "### Step 2a: Get your text data (usually through via an API or webscraping)\n",
    "\n",
    "Here, we are just looking at Federal Reserve press releases from Q1 2023. We use BeautifulSoup to webscrape the text through their webpage urls. You can use specific APIs (e.g. Reddit API, Twitter API, Dow Jones API, etc.) to scrape other text though, and your scraping code will look different.\n",
    "\n",
    "### Step 2b: Clean and pre-process the text for putting sentences through model\n",
    "In general, you'll want to make everything lowercase and get rid of excess whitespaces (tabs, new lines, etc.). Other than that, the way you clean text should be specific to the text itself. Your text may repeatedly include phrases or characters that you want to exclude--you can use regular expressions to delete/replace them from your text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ad693a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#manually add urls\n",
    "\n",
    "#2023 q1\n",
    "q1urls2023 = []\n",
    "q1urls2023.append('https://www.federalreserve.gov/newsevents/pressreleases/monetary20230201a.htm')\n",
    "q1urls2023.append('https://www.federalreserve.gov/newsevents/pressreleases/monetary20230322a.htm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da84287c",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#scrape text from urls\n",
    "textlist = [] #to store clean text later\n",
    "\n",
    "for url in q1urls2023: #iterate through urls in list\n",
    "    reqs = requests.get(url)\n",
    "    soup = BeautifulSoup(reqs.text, 'html.parser')\n",
    "    text = soup.get_text()\n",
    "    ##################clean text######################\n",
    "    #delete stuff before relevant text\n",
    "    index = text.find('Recent indicators') \n",
    "    text = text[index:]\n",
    "    #delete stuff after relevant text\n",
    "    index = text.find('Voting for the monetary policy action')\n",
    "    text = text[:index]\n",
    "    #remove white space\n",
    "    text = text.replace('\\n',\"\")\n",
    "    #lowercase\n",
    "    text = text.lower()\n",
    "    #cleaning u.s.\n",
    "    text = text.replace('u.s.',\"united states\")\n",
    "    #remove unicode\n",
    "    text_encode = text.encode(encoding=\"ascii\", errors=\"ignore\")\n",
    "    text_decode = text_encode.decode()\n",
    "    text = \" \".join([word for word in text_decode.split()])\n",
    "    #cleaning fractions\n",
    "    text = text.replace('-1/4',\".25\")\n",
    "    text = text.replace('-1/2',\".5\")\n",
    "    text = text.replace('-3/4',\".75\")\n",
    "    text = text.replace('1/4',\"0.25\")\n",
    "    text = text.replace('1/2',\".5\")\n",
    "    text = text.replace('3/4',\"0.75\")\n",
    "    ##################################################\n",
    "    textlist.append(text)\n",
    "\n",
    "sentences = [] #to store tokenized sentences \n",
    "sentences.append(sent_tokenize(textlist[0])) #tokenize each sentence\n",
    "sentences.append(sent_tokenize(textlist[1])) #tokenize each sentence\n",
    "\n",
    "#using list comprehension to store each sentence as 1 item in a list\n",
    "finalsentences = [item for sublist in sentences for item in sublist]\n",
    "\n",
    "#then each sentence should be 1 string item in a list\n",
    "print('cleaned tokenized sentences:\\n',finalsentences) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69f31d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "#if 'w' it'll write to file / if 'a', it'll append to the file\n",
    "# 'w' is fine for single batch processing, if using multiple batches due to data size then 'a' is more useful\n",
    "txt = open(\"QSSsample.txt\",\"w\",encoding=\"utf-8\") \n",
    "txt.writelines(finalsentences)\n",
    "txt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60c873da",
   "metadata": {},
   "source": [
    "### Step 3: Run the model and record results\n",
    "There are different types of sentiment analysis models. BERT (which stands for \"Bidirectional Encoder Representations from Transformers\") is one of them. There are all kinds of BERTs that are fine-tuned (i.e. specially trained) for specific tasks, like reading Tweets or recognizing hatespeech or detecting specific emotions. Here, I use finbert, a BERT model fine-tuned to understand sentiment from financial news. \n",
    "\n",
    "The BERT model reads each sentence then determines if it is positive, neutral, or negative. We can record these results, along with the counts of positive, neutral, and negative sentences. We might want to convert these counts into percentages to make them more universally comparable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17cec7a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing the model we want to use\n",
    "finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)\n",
    "tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')\n",
    "\n",
    "#feed in the sentences from earlier \n",
    "sentences = finalsentences \n",
    "\n",
    "#BERT has its own way of encoding and processing sentences\n",
    "inputs = tokenizer(sentences, return_tensors=\"pt\", padding=True)\n",
    "outputs = finbert(**inputs)[0]\n",
    "\n",
    "#this BERT tells us if the sentiment of each sentence is positive, neutral, or negative\n",
    "labels = {0:'neutral', 1:'positive',2:'negative'} \n",
    "\n",
    "#open a text file to record our results if we want to manually view each sentence and its sentiment result\n",
    "with open(\"QSSsampleresults.txt\", \"a\") as f:\n",
    "    #get ready to count the number of pos, neu, and neg sentences\n",
    "    poscount=0 \n",
    "    neucount=0\n",
    "    negcount=0\n",
    "\n",
    "    #iterate through all the sentences\n",
    "    for idx, sent in enumerate(sentences):\n",
    "        #if the model says it's positive\n",
    "        if labels[np.argmax(outputs.detach().numpy()[idx])] == 'positive':\n",
    "            #then add 1 to our positive count\n",
    "            poscount+=1\n",
    "            #also record the sentence and its sentiment result into the text file\n",
    "            f.write(sent +\"\\t*positive*\\n\")\n",
    "        elif labels[np.argmax(outputs.detach().numpy()[idx])]=='neutral':\n",
    "            neucount+=1\n",
    "            f.write(sent +\"\\t*neutral*\\n\")\n",
    "        elif labels[np.argmax(outputs.detach().numpy()[idx])]=='negative':\n",
    "            negcount+=1\n",
    "            f.write(sent +\"\\t*negative*\\n\")\n",
    "        #just in case\n",
    "        else:\n",
    "            print('error')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbdb9644",
   "metadata": {},
   "outputs": [],
   "source": [
    "#get a total count so that we can convert our counts into percentages later\n",
    "totalcount = poscount + neucount + negcount\n",
    "\n",
    "#printing count results\n",
    "print('# positive sentences: ',poscount)\n",
    "print('# neutral sentences: ',neucount)\n",
    "print('# negative sentences: ',negcount)\n",
    "\n",
    "#finding percentages too\n",
    "print('% positive sentences: {:.0%}'.format(poscount/totalcount))\n",
    "print('% neutral sentences: {:.0%}'.format(neucount/totalcount))\n",
    "print('% negative sentences: {:.0%}'.format(negcount/totalcount))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}