Guided_Project_Hacker_News_AMS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## An Analysis of _Hacker News_ Posts\n",
    "\n",
    "This project will examine a sample of posts from the _Hacker News_ website. \n",
    "\n",
    "The data set comprises a sample of c. 20,000 user-submitted stories, and it includes only posts that have been commented on by other users.\n",
    "\n",
    "The analysis will focus on \"Ask HN\" posts and \"Show HN\" posts.\n",
    "* \"Ask HN\" are posts where users ask the Hacker News community a question. \n",
    "* \"Show HN\" are posts where users show the community something of interest. \n",
    "\n",
    "The two main questions this project seeks to answer are:\n",
    "\n",
    "* Does Ask HN or Show HN receive more comments on average?\n",
    "* Do posts created at a certain time receive more comments on average?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction \n",
    "\n",
    "First, we'll open the file and take a look at the first five rows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20']]\n"
     ]
    }
   ],
   "source": [
    "## Opening Hacker News dataset and printing the first 5 rows  \n",
    "\n",
    "opened_file = open('hacker_news.csv')\n",
    "from csv import reader\n",
    "read_file = reader(opened_file)\n",
    "hn = list(read_file)\n",
    "\n",
    "print(hn[:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Headers from a List of Lists\n",
    "\n",
    "Next, we will:\n",
    "* extract the header row and assign it to a variable called `headers`.\n",
    "* remove this first row from `hn`\n",
    "* we'll display `headers` and the first five rows of `hn`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']\n",
      "[['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]\n"
     ]
    }
   ],
   "source": [
    "opened_file = open('hacker_news.csv')\n",
    "from csv import reader\n",
    "read_file = reader(opened_file)\n",
    "hn = list(read_file)\n",
    "headers = hn[0]\n",
    "hn = hn[1:]\n",
    "\n",
    "print(headers)\n",
    "print(hn[1:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting \"Ask HN\" and \"Show HN\" posts from the dataset\n",
    "\n",
    "In this step, we will filter the data so that we can just look at the \"Ask HN\" and the \"Show HN\" posts, and disregard any other posts. For this, we will create three separate lists. \n",
    "* `ask_posts` = these are all the \"Ask HN\" posts\n",
    "* `show_posts`= these are all the \"Show HN\" posts\n",
    "* `other_posts` = these are all the remaining posts in the dataset \n",
    "\n",
    "We will also count the total number of posts for each list below, and print the result. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Ask HN posts: 1744\n",
      "Number of Show HN posts: 1162\n",
      "Number of other posts: 17194\n"
     ]
    }
   ],
   "source": [
    "ask_posts = [] # start with empty lists \n",
    "show_posts = [] \n",
    "other_posts = [] \n",
    "\n",
    "for row in hn: \n",
    "    title = str.lower(row[1]) # loop over each row to get title\n",
    "    \n",
    "    if title.startswith('ask hn'):\n",
    "        ask_posts.append(row) # add the whole row to the list \n",
    "    elif title.startswith('show hn'):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)\n",
    "\n",
    "print('Number of Ask HN posts:', len(ask_posts)) # print the length of the list\n",
    "print('Number of Show HN posts:', len(show_posts))\n",
    "print('Number of other posts:', len(other_posts))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating the Average Number of Comments for Ask HN and Show HN Posts \n",
    "\n",
    "Since we want to find out which type of post (Ask HN or Show HN) receives the most comments, the next thing we will do is find the average number of comments for each type of post. \n",
    "\n",
    "Here we will start with the average number of comments for each \"Ask HN\" post: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14\n"
     ]
    }
   ],
   "source": [
    "total_ask_comments = 0 # start at a count of zero \n",
    "\n",
    "for row in ask_posts: # loop over each row in ask_posts\n",
    "    num_comments = int(row[4]) # convert number of comments to an integer\n",
    "    total_ask_comments += num_comments; # add number of comments to total number of comment\n",
    "\n",
    "avg_ask_comments = total_ask_comments / len(ask_posts) # calculate the average number of comments per post\n",
    "print(int(avg_ask_comments)) # convert to int to round the value.\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will calculate the average number of comments for each \"Show HN\" post:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "total_show_comments = 0\n",
    "\n",
    "for row in show_posts:\n",
    "    num_comments = int(row[4])\n",
    "    total_show_comments += num_comments;\n",
    "\n",
    "avg_show_comments = total_show_comments / len(show_posts)\n",
    "print(int(avg_show_comments))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results:\n",
    "\n",
    "* Average number of comments for an \"Ask HN\" post: 14\n",
    "* Average number of comments for a \"Show HN\" post: 10\n",
    "\n",
    "\n",
    "We can see that \"Ask HN\" posts get more comments than \"Show HN\" posts on average. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Number of Ask Posts and Comments by Hour Created\n",
    "\n",
    "We're next going to see whether posts created at a certain time of day are likely to receive more comments. For this, we will just use the `ask_posts` dataset, since this had the largest average number of comments per post. \n",
    "\n",
    "We will first calculate how many posts are created in each hour of the day. We will also calculate the number of comments received in each hour. \n",
    "\n",
    "We will then calculate the average number of comments a post receives by hour.  \n",
    "\n",
    "The code below: \n",
    "* imports the datetime module, which we'll use to isolate the hour of the day, e.g. 09 (9am), 14 (2pm), etc. \n",
    "* creates a new list of lists from `ask_posts`, called `result_list`, which just contains `created_at` and `num_comments`.\n",
    "* loops over `result_list` and creates two dictionaries: \n",
    "  - `counts_by_hour` = capture the number of Ask HN posts created in each hour of the day\n",
    "  - `comments_by_hour` = capture the number of Ask HN comments created in each hour of the day\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1], ['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17], ['9/26/2015 23:23', 1], ['4/22/2016 12:24', 4], ['11/16/2015 9:22', 1], ['2/24/2016 17:57', 1], ['6/4/2016 17:17', 2], ['9/19/2015 17:04', 7]]\n"
     ]
    }
   ],
   "source": [
    "import datetime as dt # import datetime module\n",
    "\n",
    "result_list = [] # create empty list\n",
    "\n",
    "for row in ask_posts: \n",
    "    created_at = row[6] # get created_at element\n",
    "    num_comments = int(row[4]) # get num_comments element\n",
    "    result_list.append([created_at, num_comments]); # add elements to the result_list\n",
    "\n",
    "print(result_list[:11])   \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'09': 45, '13': 85, '10': 59, '14': 107, '16': 108, '23': 68, '12': 73, '17': 100, '15': 116, '21': 109, '20': 80, '02': 58, '18': 109, '03': 54, '05': 46, '19': 110, '01': 60, '22': 71, '08': 48, '04': 47, '00': 55, '06': 44, '07': 34, '11': 58}\n",
      "{'09': 251, '13': 1253, '10': 793, '14': 1416, '16': 1814, '23': 543, '12': 687, '17': 1146, '15': 4477, '21': 1745, '20': 1722, '02': 1381, '18': 1439, '03': 421, '05': 464, '19': 1188, '01': 683, '22': 479, '08': 492, '04': 337, '00': 447, '06': 397, '07': 267, '11': 641}\n"
     ]
    }
   ],
   "source": [
    "counts_by_hour = {} # new dictionary to count how many posts in each hour of the day\n",
    "comments_by_hour = {} # new dictionary to count total no. of comments in each hour of the day\n",
    "\n",
    "for row in result_list:\n",
    "    created_at = row[0]\n",
    "    num_comments = int(row[1])\n",
    "    dt_created_at = dt.datetime.strptime(created_at, \"%m/%d/%Y %H:%M\") # convert created_at into a datetime object\n",
    "    hour = dt.datetime.strftime(dt_created_at, \"%H\") # isolate the hour from the datetime object\n",
    "    if hour not in counts_by_hour:  \n",
    "        counts_by_hour[hour] = 1 # if hour isn't counted yet, assign the key a value of 1.\n",
    "        comments_by_hour[hour] = num_comments # if hour isn't counted yet, assign the key a value that equals number of comments \n",
    "    else:\n",
    "        counts_by_hour[hour] += 1 # if hour is counted, increase count by 1\n",
    "        comments_by_hour[hour] += num_comments # if hour is counted, add the number of comments to the existing number of comments (the total)\n",
    "print(counts_by_hour)\n",
    "print(comments_by_hour)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to calculate the average number of comments a post receives in a certain hour of the day. \n",
    "\n",
    "The code below: \n",
    "* creates an empty list, called `avg_by_hour`\n",
    "* loops over the dictionary called `comments_by_hour`\n",
    "* appends to the list two elements:\n",
    "  - the key (`hour`) as the first element in the list\n",
    "  - and then for the second element of the list, we calculate the average number of comments per post in that hour, using the values from the two dictionaries we created earlier (`comments_by_hour`, and `counts_by_hour`).  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['09', 5.5777777777777775]\n",
      "['13', 14.741176470588234]\n",
      "['10', 13.440677966101696]\n",
      "['14', 13.233644859813085]\n",
      "['16', 16.796296296296298]\n",
      "['23', 7.985294117647059]\n",
      "['12', 9.41095890410959]\n",
      "['17', 11.46]\n",
      "['15', 38.5948275862069]\n",
      "['21', 16.009174311926607]\n",
      "['20', 21.525]\n",
      "['02', 23.810344827586206]\n",
      "['18', 13.20183486238532]\n",
      "['03', 7.796296296296297]\n",
      "['05', 10.08695652173913]\n",
      "['19', 10.8]\n",
      "['01', 11.383333333333333]\n",
      "['22', 6.746478873239437]\n",
      "['08', 10.25]\n",
      "['04', 7.170212765957447]\n",
      "['00', 8.127272727272727]\n",
      "['06', 9.022727272727273]\n",
      "['07', 7.852941176470588]\n",
      "['11', 11.051724137931034]\n"
     ]
    }
   ],
   "source": [
    "avg_by_hour = []\n",
    "\n",
    "for hour in comments_by_hour:\n",
    "    avg_by_hour.append([hour, comments_by_hour[hour]/counts_by_hour[hour]])\n",
    "\n",
    "\n",
    "for row in avg_by_hour:\n",
    "    print(row)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sorting and Printing Values from a List of Lists\n",
    "\n",
    "We'll now sort and print the values so that we can see which hour of day is most likely that you will receive comments to an Ask HN post.\n",
    "\n",
    "To do this, we will first sorts the data in order (from highest average number of comments, to the lowest. \n",
    "\n",
    "The code below:\n",
    "* creates an empty list, which will store the swapped values\n",
    "* loops over `avg_by_hour` list\n",
    "* adds `avg` as the first element of the list \n",
    "* adds `hour` as the second element of the list\n",
    "* uses the sorted() function to sort the values from highest to lowest, using `reverse=True`\n",
    "* assigns the sorted values to `sorted_swap` and prints the results. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[38.5948275862069, '15'], [23.810344827586206, '02'], [21.525, '20'], [16.796296296296298, '16'], [16.009174311926607, '21'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [13.20183486238532, '18'], [11.46, '17'], [11.383333333333333, '01'], [11.051724137931034, '11'], [10.8, '19'], [10.25, '08'], [10.08695652173913, '05'], [9.41095890410959, '12'], [9.022727272727273, '06'], [8.127272727272727, '00'], [7.985294117647059, '23'], [7.852941176470588, '07'], [7.796296296296297, '03'], [7.170212765957447, '04'], [6.746478873239437, '22'], [5.5777777777777775, '09']]\n"
     ]
    }
   ],
   "source": [
    "swap_avg_by_hour = [] # create empty list\n",
    "\n",
    "for row in avg_by_hour: \n",
    "    hour = row[0]\n",
    "    avg = row[1]\n",
    "    swap_avg_by_hour.append([avg, hour]) # append to list, swapping the current order\n",
    "\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse=True) # sorting\n",
    "print(sorted_swap)\n",
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will print our formatted results.  \n",
    "\n",
    "The code below: \n",
    "* prints the first statement\n",
    "* loops through our sorted list\n",
    "* converts the average amount into 2 decimal places\n",
    "* converts the hour into a datetime object and then formats it to our desired format\n",
    "* formats the final output into a statement using `str.format`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 Hours for Ask Posts Comments\n",
      "15:00: 38.59 average comments per post\n",
      "02:00: 23.81 average comments per post\n",
      "20:00: 21.52 average comments per post\n",
      "16:00: 16.80 average comments per post\n",
      "21:00: 16.01 average comments per post\n"
     ]
    }
   ],
   "source": [
    "print(\"Top 5 Hours for Ask Posts Comments\")\n",
    "\n",
    "for row in sorted_swap[:5]:\n",
    "    average = \"{:.2f}\".format(row[0]) # gets average and formats to 2 dp\n",
    "    hour = row[1] \n",
    "    hour_dt = dt.datetime.strptime(hour, \"%H\") # converts to datetime\n",
    "    hour_dt_object = dt.datetime.strftime(hour_dt, \"%H:%M\") # converts to our desired format of datetime\n",
    "    output = \"{}: {} average comments per post\".format(hour_dt_object, average) # formats the output into a sentence\n",
    "    print(output) # prints the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "The results show that if you want the highest chance of receiving comments, you should create as Ask HM post at 15:00, 02:00, 20:00, 16:00, or 21:00. The documentation states that the timezone for the times in the dataset are in Eastern US time.  \n",
    "\n",
    "So if I wanted to create a post with a higher chance of receiving comments, I should convert these times into my timezone (GMT). This means that I should create my posts at 19:00, 06:00, 00:00, 20:00, or 01:00. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}