Lab_1_answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1 Answers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were required to create a new folder in HDFS called textData. The command for this is given below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hadoop fs -mkdir textData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then had to put the file textData.txt into this new folder on HDFS. This is done with the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hadoop fs -put textData.txt textData/textData.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To confirm that the upload has been successful we can list the contents of the textData directory with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hadoop fs -ls textData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are asked to list the contents of the w1e3-output folder on HDFS. This can be done with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hadoop fs -ls w1e3-output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon running this command you should see something that looks like:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Found 2 items\n",
    "-rwxr-xr-x   3 user341 student          0 2020-02-24 18:17 w1e3-output/_SUCCESS\n",
    "-rwxr-xr-x   3 user341 student     717768 2020-02-24 18:17 w1e3-output/part-00000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see from the output that only 1 part file has been produced here. This means that only one reducer process was executed here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By viewing the contents of the results we can see that our map reduce job is sensitive to the precise punctuation following the words. E.g. it counts \"Amen\" and \"Amen,\" separately. The output could have been improved by removing any punctuation marks from the keyspace."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include below the completed mapper.py file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "import sys\n",
    "\n",
    "# input comes from STDIN (standard input)\n",
    "for line in sys.stdin:\n",
    "        # remove leading and trailing whitespace\n",
    "        line = line.strip()\n",
    "\n",
    "        # TODO: Students to output <lineLength, 1>\n",
    "        # compute length of line using len function\n",
    "        lineLength = len(line)\n",
    "\n",
    "        # write the results to STDOUT (standard output);\n",
    "        # what we output here will be the input for the \n",
    "        # Reduce step, i.e. the input for reducer.py\n",
    "        #\n",
    "        # tab-delimited; the trivial lineLength count is 1\n",
    "        print '%s\\t%s' % (lineLength, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now include the completed reducer.py file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "from operator import itemgetter\n",
    "import sys\n",
    "\n",
    "current_len = None\n",
    "current_count = 0\n",
    "vlen = None\n",
    "\n",
    "# input comes from STDIN\n",
    "for line in sys.stdin:\n",
    "        # remove leading and trailing whitespace\n",
    "        line = line.strip()\n",
    "\n",
    "        # parse the input we got from mapper.py\n",
    "        vlen, count = line.split('\\t', 1)\n",
    "\n",
    "        # convert count (currently a string) to int\n",
    "        try:\n",
    "                count = int(count)\n",
    "        except ValueError:\n",
    "                # count was not a number, so silently\n",
    "                # ignore/discard this line\n",
    "                continue\n",
    "\n",
    "        # this IF-switch only works because Hadoop sorts map output\n",
    "        # by key (here: vlen) before it is passed to the reducer\n",
    "        if current_len == vlen:\n",
    "                current_count += count\n",
    "        else:\n",
    "                if current_len:\n",
    "                        # write result to STDOUT\n",
    "                        # TODO: Students to produce <lineLength, count> output\n",
    "                        print '%s\\t%s' % (current_len, current_count)\n",
    "                current_count = count\n",
    "                current_len = vlen\n",
    "\n",
    "# do not forget to output the last word if needed!\n",
    "if current_len == vlen:\n",
    "        # TODO: Students to produce <lineLength, count> output\n",
    "        print '%s\\t%s' % (current_len, current_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Command to execute the Hadoop job (note: USER must be replaced with your username):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hadoop jar $HADOOP_STR/hadoop-streaming-2.7.0-mapr-1808.jar \\\n",
    "-libjars $HADOOP_STR/hadoop-streaming-2.7.0-mapr-1808.jar \\\n",
    "-input textData/* \\\n",
    "-output w1e4-output \\\n",
    "-mapper \"python mapper.py\" \\\n",
    "-file /home/USER/bd-sp-2017/w1e4/mapper.py \\\n",
    "-reducer \"python reducer.py\" \\\n",
    "-file /home/USER/bd-sp-2017/w1e4/reducer.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the results we notice that asides from a line length of 0 the most frequent line length is 42."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}