scrape_gamespot.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:8adb8d92d4701ac5ae1601434ff4c33b4ad71a17d484b8995f84121fe6898888"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# GameSpot scraping example"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import requests\n",
      "from pattern import web"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# example of URL at which many review summaries and scores can be found\n",
      "url = 'http://www.gamespot.com/reviews/?page=2'\n",
      "html = requests.get(url).text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# document object model\n",
      "dom = web.Element(html)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# get a list of the separate \"articles\" within the HTML page\n",
      "articles = dom.by_tag(\"article\")\n",
      "# look at the first article\n",
      "article = articles[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# look at a single article's HTML snippet\n",
      "print article.content"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "<a class=\"js-event-tracking\" href=\"/reviews/a-city-sleeps-review/1900-6415920/\">\n",
        "<div class=\"media-well l-item--push hide-mobile \"><div class=\"well--review-gs\"><span class=\"site-logo--mono\">GameSpot Score</span><strong>5</strong><span class=\"score-word\">Mediocre</span></div></div>\n",
        "<figure class=\"media-figure l-item--pull\">\n",
        "<div class=\"media-img imgflare--boxart media-figure--medium\">\n",
        "<img src=\"http://static4.gamespot.com/uploads/scale_tiny/1197/11970954/2639497-screen+shot+2014-08-28+at+1.37.25+pm.jpg\" alt=\"A City Sleeps Review\" />\n",
        "</div>\n",
        "</figure>\n",
        "<div class=\"media-body\">\n",
        "<span class=\"media-superscript\">A City Sleeps</span>\n",
        "<h3 class=\"media-title\">A City Sleeps Review</h3>\n",
        "<div class=\"media-byline\">\n",
        "<ul class=\"system-list\">\n",
        "<li class=\"system label\">Reviewed on:</li>\n",
        "<li class=\"system system--pill system--pill system--pc \">PC</li>\n",
        "</ul>\n",
        "</div>\n",
        "<p class=\"media-deck hide-mobile\"> A City Sleeps is a great looking shooter with compelling mechanics, but it suffers under the weight of its ambitions and prohibitive difficulty.</p>\n",
        "<footer class=\"media-meta\">\n",
        "<time class=\"media-date\" datetime=\"2014-10-21 12:00:00\">Oct 21, 12:00pm</time>\n",
        "</footer>\n",
        "</div>\n",
        "</a>\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for art in articles:\n",
      "    if art.attr['class'] == 'media media-review':\n",
      "        review_summary = art.by_tag('p')\n",
      "        review_score = art.by_tag('strong')\n",
      "        review_time = art.by_tag('time')\n",
      "        print review_summary[0].content + \" SCORE = \" + review_score[0].content\n",
      "        print review_time[0].attr['datetime']\n",
      "        print '*'*80"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " A City Sleeps is a great looking shooter with compelling mechanics, but it suffers under the weight of its ambitions and prohibitive difficulty. SCORE = 5\n",
        "2014-10-21 12:00:00\n",
        "********************************************************************************\n",
        " This top-down arena shooter runs afoul of fun. SCORE = 3\n",
        "2014-10-21 09:00:00\n",
        "********************************************************************************\n",
        " The Legend of Korra doesn&#039;t even come close to capturing the spark of the universe it depicts. SCORE = 3\n",
        "2014-10-21 07:00:00\n",
        "********************************************************************************\n",
        " Fantasia is an excellent and joyful game that turns rhythm gameplay into performance art. SCORE = 8\n",
        "2014-10-21 00:01:00\n",
        "********************************************************************************\n",
        " Falling Skies is a functional, unattractive, and uninspired XCOM clone. SCORE = 4\n",
        "2014-10-16 17:25:00\n",
        "********************************************************************************\n",
        " TRI: Of Friendship and Madness redefines the first-person puzzler through creative simplicity. SCORE = 8\n",
        "2014-10-16 17:17:00\n",
        "********************************************************************************\n",
        " One of the best games of 2012 gets a mild makeover. SCORE = 8\n",
        "2014-10-15 16:50:00\n",
        "********************************************************************************\n",
        " Short Peace&#039;s four animated shorts are wonderful examples of modern animation, but the game portion, Ranko Tsukigime&#039;s Longest Day, is a letdown on almost every front. SCORE = 6\n",
        "2014-10-14 10:54:00\n",
        "********************************************************************************\n",
        " Hack &#039;n&#039; Slash&#039;s puzzles esoteric puzzles are alternately delightful and discouraging. SCORE = 6\n",
        "2014-10-14 09:59:00\n",
        "********************************************************************************\n",
        " Costume Quest 2 is a charming sequel that fixes most of its predecessor&#039;s flaws, though it remains a very short, overly simple experience. SCORE = 7\n",
        "2014-10-13 14:54:00\n",
        "********************************************************************************\n",
        " Fantasy Life is an ambitious but repetitive RPG that makes fails to make the mundane magical. SCORE = 6\n",
        "2014-10-13 08:30:00\n",
        "********************************************************************************\n",
        " Borderlands: The Pre-Sequel brings you more of the loot and lighthearted humor you love, though it never quite overcomes the gravity of its predecessors. SCORE = 7\n",
        "2014-10-13 08:00:00\n",
        "********************************************************************************\n",
        " Bayonetta 2 is a classic in the making, and one of the finest, most wonderfully insane action games ever made. SCORE = 10\n",
        "2014-10-13 00:01:00\n",
        "********************************************************************************\n",
        " Endless Legend adds fantastic twists to familiar strategy. SCORE = 8\n",
        "2014-10-10 18:08:00\n",
        "********************************************************************************\n",
        " Ryse: Son of Rome is a repetitive and tedious homage to the gruesome battles showcased in the film 300. SCORE = 5\n",
        "2014-10-10 17:37:00\n",
        "********************************************************************************\n",
        " Dark Souls II, in all its epic designs, fittingly concludes in the harshness of winter. SCORE = 9\n",
        "2014-10-08 17:49:00\n",
        "********************************************************************************\n",
        " All-new game mechanics, first-rate graphics, and involved sleuthing make Sherlock Holmes: Crimes &amp; Punishments a great adventure game. SCORE = 8\n",
        "2014-10-07 17:08:00\n",
        "********************************************************************************\n",
        " NBA 2K15\u2019s overhauled movement system and improved shooting mechanics make it the best-playing basketball game in existence. SCORE = 9\n",
        "2014-10-07 15:15:00\n",
        "********************************************************************************\n",
        " Clunky combat, bad AI, and so-so stealth leave Styx lingering in the shadows of mediocrity. SCORE = 5\n",
        "2014-10-07 09:00:00\n",
        "********************************************************************************\n",
        " Driveclub is a solid, basic racing game with lots of tracks, lots of cars, and very little in the way of personality. SCORE = 5\n",
        "2014-10-07 06:00:00\n",
        "********************************************************************************\n",
        " Persona 4 Ultimax is a worthy upgrade to a solid fighter, with plenty of new content for competitive players and Persona fans alike. SCORE = 8\n",
        "2014-10-06 13:46:00\n",
        "********************************************************************************\n"
       ]
      }
     ],
     "prompt_number": 6
    }
   ],
   "metadata": {}
  }
 ]
}