Module_1_Data_Collection/02_Web_Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another alternative to data collection besides using APIs is through web scraping. In this notebook we will learn how to scrape a web page using `beautifulsoup` and `requests` library. We will be scaping http://books.toscrape.com/. A website that was specifically created in order for developers to learn how to scrape.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script below retrieves the html page of a book entitled \"A Light in the Attic\". The html page was retrieved using the `requests` library. After retrieving, the html is passed on to the `beautifulsoup` parser. By using `beautifulsoup`, we are able to get specific information (ex. title, price, rating) of the book.\n",
    "\n",
    "Take a look at the html document printed by the script. What can you observe? Do you see any specific patterns that are repeating? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<!--[if lt IE 7]>      <html lang=\"en-us\" class=\"no-js lt-ie9 lt-ie8 lt-ie7\"> <![endif]-->\n",
      "<!--[if IE 7]>         <html lang=\"en-us\" class=\"no-js lt-ie9 lt-ie8\"> <![endif]-->\n",
      "<!--[if IE 8]>         <html lang=\"en-us\" class=\"no-js lt-ie9\"> <![endif]-->\n",
      "<!--[if gt IE 8]><!-->\n",
      "<html class=\"no-js\" lang=\"en-us\">\n",
      " <!--<![endif]-->\n",
      " <head>\n",
      "  <title>\n",
      "   A Light in the Attic | Books to Scrape - Sandbox\n",
      "  </title>\n",
      "  <meta content=\"text/html; charset=utf-8\" http-equiv=\"content-type\"/>\n",
      "  <meta content=\"24th Jun 2016 09:29\" name=\"created\"/>\n",
      "  <meta content=\"\n",
      "    It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more\n",
      "\" name=\"description\"/>\n",
      "  <meta content=\"width=device-width\" name=\"viewport\"/>\n",
      "  <meta content=\"NOARCHIVE,NOCACHE\" name=\"robots\"/>\n",
      "  <!-- Le HTML5 shim, for IE6-8 support of HTML elements -->\n",
      "  <!--[if lt IE 9]>\n",
      "        <script src=\"//html5shim.googlecode.com/svn/trunk/html5.js\"></script>\n",
      "        <![endif]-->\n",
      "  <link href=\"../../static/oscar/favicon.ico\" rel=\"shortcut icon\"/>\n",
      "  <link href=\"../../static/oscar/css/styles.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "  <link href=\"../../static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"../../static/oscar/css/datetimepicker.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      " </head>\n",
      " <body class=\"default\" id=\"default\">\n",
      "  <header class=\"header container-fluid\">\n",
      "   <div class=\"page_inner\">\n",
      "    <div class=\"row\">\n",
      "     <div class=\"col-sm-8 h1\">\n",
      "      <a href=\"../../index.html\">\n",
      "       Books to Scrape\n",
      "      </a>\n",
      "      <small>\n",
      "       We love being scraped!\n",
      "      </small>\n",
      "     </div>\n",
      "    </div>\n",
      "   </div>\n",
      "  </header>\n",
      "  <div class=\"container-fluid page\">\n",
      "   <div class=\"page_inner\">\n",
      "    <ul class=\"breadcrumb\">\n",
      "     <li>\n",
      "      <a href=\"../../index.html\">\n",
      "       Home\n",
      "      </a>\n",
      "     </li>\n",
      "     <li>\n",
      "      <a href=\"../category/books_1/index.html\">\n",
      "       Books\n",
      "      </a>\n",
      "     </li>\n",
      "     <li>\n",
      "      <a href=\"../category/books/poetry_23/index.html\">\n",
      "       Poetry\n",
      "      </a>\n",
      "     </li>\n",
      "     <li class=\"active\">\n",
      "      A Light in the Attic\n",
      "     </li>\n",
      "    </ul>\n",
      "    <div id=\"messages\">\n",
      "    </div>\n",
      "    <div class=\"content\">\n",
      "     <div id=\"promotions\">\n",
      "     </div>\n",
      "     <div id=\"content_inner\">\n",
      "      <article class=\"product_page\">\n",
      "       <!-- Start of product page -->\n",
      "       <div class=\"row\">\n",
      "        <div class=\"col-sm-6\">\n",
      "         <div class=\"carousel\" id=\"product_gallery\">\n",
      "          <div class=\"thumbnail\">\n",
      "           <div class=\"carousel-inner\">\n",
      "            <div class=\"item active\">\n",
      "             <img alt=\"A Light in the Attic\" src=\"../../media/cache/fe/72/fe72f0532301ec28892ae79a629a293c.jpg\"/>\n",
      "            </div>\n",
      "           </div>\n",
      "          </div>\n",
      "         </div>\n",
      "        </div>\n",
      "        <div class=\"col-sm-6 product_main\">\n",
      "         <h1>\n",
      "          A Light in the Attic\n",
      "         </h1>\n",
      "         <p class=\"price_color\">\n",
      "          Â£51.77\n",
      "         </p>\n",
      "         <p class=\"instock availability\">\n",
      "          <i class=\"icon-ok\">\n",
      "          </i>\n",
      "          In stock (22 available)\n",
      "         </p>\n",
      "         <p class=\"star-rating Three\">\n",
      "          <i class=\"icon-star\">\n",
      "          </i>\n",
      "          <i class=\"icon-star\">\n",
      "          </i>\n",
      "          <i class=\"icon-star\">\n",
      "          </i>\n",
      "          <i class=\"icon-star\">\n",
      "          </i>\n",
      "          <i class=\"icon-star\">\n",
      "          </i>\n",
      "          <!-- <small><a href=\"/catalogue/a-light-in-the-attic_1000/reviews/\">\n",
      "        \n",
      "                \n",
      "                    0 customer reviews\n",
      "                \n",
      "        </a></small>\n",
      "         -->\n",
      "          <!-- \n",
      "    <a id=\"write_review\" href=\"/catalogue/a-light-in-the-attic_1000/reviews/add/#addreview\" class=\"btn btn-success btn-sm\">\n",
      "        Write a review\n",
      "    </a>\n",
      "\n",
      " -->\n",
      "         </p>\n",
      "         <hr/>\n",
      "         <div class=\"alert alert-warning\" role=\"alert\">\n",
      "          <strong>\n",
      "           Warning!\n",
      "          </strong>\n",
      "          This is a demo website for web scraping purposes. Prices and ratings here were randomly assigned and have no real meaning.\n",
      "         </div>\n",
      "        </div>\n",
      "        <!-- /col-sm-6 -->\n",
      "       </div>\n",
      "       <!-- /row -->\n",
      "       <div class=\"sub-header\" id=\"product_description\">\n",
      "        <h2>\n",
      "         Product Description\n",
      "        </h2>\n",
      "       </div>\n",
      "       <p>\n",
      "        It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more\n",
      "       </p>\n",
      "       <div class=\"sub-header\">\n",
      "        <h2>\n",
      "         Product Information\n",
      "        </h2>\n",
      "       </div>\n",
      "       <table class=\"table table-striped\">\n",
      "        <tr>\n",
      "         <th>\n",
      "          UPC\n",
      "         </th>\n",
      "         <td>\n",
      "          a897fe39b1053632\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Product Type\n",
      "         </th>\n",
      "         <td>\n",
      "          Books\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Price (excl. tax)\n",
      "         </th>\n",
      "         <td>\n",
      "          Â£51.77\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Price (incl. tax)\n",
      "         </th>\n",
      "         <td>\n",
      "          Â£51.77\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Tax\n",
      "         </th>\n",
      "         <td>\n",
      "          Â£0.00\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Availability\n",
      "         </th>\n",
      "         <td>\n",
      "          In stock (22 available)\n",
      "         </td>\n",
      "        </tr>\n",
      "        <tr>\n",
      "         <th>\n",
      "          Number of reviews\n",
      "         </th>\n",
      "         <td>\n",
      "          0\n",
      "         </td>\n",
      "        </tr>\n",
      "       </table>\n",
      "       <section>\n",
      "        <div class=\"sub-header\" id=\"reviews\">\n",
      "        </div>\n",
      "       </section>\n",
      "      </article>\n",
      "      <!-- End of product page -->\n",
      "     </div>\n",
      "    </div>\n",
      "   </div>\n",
      "  </div>\n",
      "  <footer class=\"footer container-fluid\">\n",
      "  </footer>\n",
      "  <!-- jQuery -->\n",
      "  <script src=\"http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js\">\n",
      "  </script>\n",
      "  <script>\n",
      "   window.jQuery || document.write('<script src=\"../../static/oscar/js/jquery/jquery-1.9.1.min.js\"><\\/script>')\n",
      "  </script>\n",
      "  <!-- Twitter Bootstrap -->\n",
      "  <script src=\"../../static/oscar/js/bootstrap3/bootstrap.min.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <!-- Oscar -->\n",
      "  <script charset=\"utf-8\" src=\"../../static/oscar/js/oscar/ui.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <script charset=\"utf-8\" src=\"../../static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <script charset=\"utf-8\" src=\"../../static/oscar/js/bootstrap-datetimepicker/locales/bootstrap-datetimepicker.all.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <script type=\"text/javascript\">\n",
      "   $(function() {\n",
      "                \n",
      "    \n",
      "    oscar.init();\n",
      "\n",
      "            });\n",
      "  </script>\n",
      "  <!-- Version: N/A -->\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#get the html from one of the books in the website\n",
    "page = requests.get('http://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html')\n",
    "\n",
    "#feed it into beautiful soup for parsing\n",
    "soup = BeautifulSoup(page.text, 'html.parser')\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Browsing though the html document can be overwhelming. Don't fret, as you gain more experience in scraping websites this will become more intuitive to you. Lets familiarize ourselves first with the typical structure of an HTML document. \n",
    "\n",
    "```\n",
    "<html>\n",
    "    <head>\n",
    "        <!--this is how comments are written in html-->\n",
    "        <!--we usually place css files under the head tag --> \n",
    "        <link href=\"../../static/oscar/css/styles.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
    "        <link href=\"../../static/oscar/js/bootstrap-datetimepicker/bootstrap-datetimepicker.css\" rel=\"stylesheet\"/>\n",
    "        <link href=\"../../static/oscar/css/datetimepicker.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
    "    </head>\n",
    "    <body>\n",
    "        <!-- this where the content of the web page is located -->\n",
    "        <!-- which means the information that you want to scrape will be located here -->\n",
    "    </body>\n",
    "</html>\n",
    "```\n",
    "\n",
    "The html document is made up of tags each with an opening and closing tag. Typically it is made up of 3 main tags `html`, `head`, `body`. The html tag is a standard being followed to indicate that the document being created is html. The head tag would usually contain libraries or files that need to be imported into the document for example `CSS`(Cascading Style Sheets) files. `CSS` can be imagined as the libraries/files responsible for making your website pretty so things like color, shading, font settings can be found here. Lastly, the body tag is where the content of the page can be found. This is usually where we scrape the information from. \n",
    "\n",
    "Inside the body tag you can observe several types of tags. Some common tags can be found below: \n",
    "\n",
    "| HTML Tag | Description |\n",
    "| --- | --- |\n",
    "| div | The div tag is used to group together html elements that make up a component | \n",
    "| h1,h2,h3,h4,h5 | These are header tags, the smalelr the number the bigger the text that is shown on screen | \n",
    "| p | P stands for paragraph, this is where text content is usually placed | \n",
    "| a | a stands for anchor, this is where hyperlinks are placed | \n",
    "| img | imag stands for Image, this is where images are placed | \n",
    "| ul, ol, li | These are list tags, ul stands for unordered list, ol for ordered list ,li for list | \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another useful tool to aid you in web scraping is the inspector tool that is available in browsers. This can be accessed by pressing `f12`. After pressing a toolbar should pop up on your browser. With the inspector tool you can hover your mosue around the web page and it will automatically show you which part of the html document you are looking at. This makes scraping easier since you don't have to read through the entire html document.\n",
    "\n",
    "Take for example the image below, I hovered my mouse on the container of the book title and on the right side you can see the different tags that make up the book title component. So if we were interested in getting the title, and price we know that the title is placed inside a `h1` tag while the price is placed inside a `p` tag. You can observe also that these tags have attributes called `class`. The class attribute is connected to the css file that was imported in the head tag which allows the browser to know how to render the tag. The class attribute is also useful for us when scraping since it allows us to narrow down the tag that we want to get.  \n",
    "\n",
    "<img src=\"chrome_dev_tools.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a general undesrtanding of the html document let us use beautifulsoup to get information from the website. The most common function that we will use from this library is `find`. This function will return the first tag that matches the criteria given to it. Let's use it to get the title of the book and the price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h1>A Light in the Attic</h1>\n",
      "A Light in the Attic\n"
     ]
    }
   ],
   "source": [
    "# #the find function returns the tag of the element if we want to remove the tags we call the .text attribute \n",
    "print(soup.find('h1'))\n",
    "print(soup.find('h1').text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The find function also accepts attributes to look for inside the tag. For the script below we indicate that we are looking for the paragraph tag who has a class called price_color. This allows our search to be more targeted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p class=\"price_color\">Â£51.77</p>\n",
      "Â£51.77\n"
     ]
    }
   ],
   "source": [
    "print(soup.find('p', attrs={'class':'price_color'}))\n",
    "print(soup.find('p', attrs={'class':'price_color'}).text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another function that is available for us is the find_all function. This returns a list of all elements that match the tag placed inside the find_all function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"price_color\">Â£51.77</p>,\n",
       " <p class=\"instock availability\">\n",
       " <i class=\"icon-ok\"></i>\n",
       "     \n",
       "         In stock (22 available)\n",
       "     \n",
       " </p>,\n",
       " <p class=\"star-rating Three\">\n",
       " <i class=\"icon-star\"></i>\n",
       " <i class=\"icon-star\"></i>\n",
       " <i class=\"icon-star\"></i>\n",
       " <i class=\"icon-star\"></i>\n",
       " <i class=\"icon-star\"></i>\n",
       " <!-- <small><a href=\"/catalogue/a-light-in-the-attic_1000/reviews/\">\n",
       "         \n",
       "                 \n",
       "                     0 customer reviews\n",
       "                 \n",
       "         </a></small>\n",
       "          --> \n",
       " \n",
       " \n",
       " <!-- \n",
       "     <a id=\"write_review\" href=\"/catalogue/a-light-in-the-attic_1000/reviews/add/#addreview\" class=\"btn btn-success btn-sm\">\n",
       "         Write a review\n",
       "     </a>\n",
       " \n",
       "  --></p>,\n",
       " <p>It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more</p>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_results = soup.find_all('p')\n",
    "p_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that it is a list we can use an index to retrieve the information that we need. Take for example if we are only interested in getting the product description we can just get the last element from the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love th It's hard to imagine a world without A Light in the Attic. This now-classic collection of poetry and drawings from Shel Silverstein celebrates its 20th anniversary with this special edition. Silverstein's humorous and creative verse can amuse the dowdiest of readers. Lemon-faced adults and fidgety kids sit still and read these rhythmic words and laugh and smile and love that Silverstein. Need proof of his genius? RockabyeRockabye baby, in the treetopDon't you know a treetopIs no safe place to rock?And who put you up there,And your cradle, too?Baby, I think someone down here'sGot it in for you. Shel, you never sounded so good. ...more\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_results[-1].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get information relative to a tag using the next_sibling function. For example we want to get how many are in stock. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p class=\"instock availability\">\n",
      "<i class=\"icon-ok\"></i>\n",
      "    \n",
      "        In stock (22 available)\n",
      "    \n",
      "</p>\n"
     ]
    }
   ],
   "source": [
    "print(soup.find('p', attrs={'class':'price_color'}).next_sibling.next_sibling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, tables are a common structure found in websites which contain the information that we need. In order to retreive the data we have to iterate over the different rows within the table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<table class=\"table table-striped\">\n",
      " <tr>\n",
      "  <th>\n",
      "   UPC\n",
      "  </th>\n",
      "  <td>\n",
      "   a897fe39b1053632\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Product Type\n",
      "  </th>\n",
      "  <td>\n",
      "   Books\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Price (excl. tax)\n",
      "  </th>\n",
      "  <td>\n",
      "   Â£51.77\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Price (incl. tax)\n",
      "  </th>\n",
      "  <td>\n",
      "   Â£51.77\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Tax\n",
      "  </th>\n",
      "  <td>\n",
      "   Â£0.00\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Availability\n",
      "  </th>\n",
      "  <td>\n",
      "   In stock (22 available)\n",
      "  </td>\n",
      " </tr>\n",
      " <tr>\n",
      "  <th>\n",
      "   Number of reviews\n",
      "  </th>\n",
      "  <td>\n",
      "   0\n",
      "  </td>\n",
      " </tr>\n",
      "</table>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "table_res = soup.find('table', class_=\"table-striped\")\n",
    "print(table_res.prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UPC=a897fe39b1053632\n",
      "Product Type=Books\n",
      "Price (excl. tax)=Â£51.77\n",
      "Price (incl. tax)=Â£51.77\n",
      "Tax=Â£0.00\n",
      "Availability=In stock (22 available)\n",
      "Number of reviews=0\n"
     ]
    }
   ],
   "source": [
    "for row in table_res.find_all('tr'):\n",
    "    header = row.find('th').text\n",
    "    data = row.find('td').text\n",
    "    print(f\"{header}={data}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}