PS1_Shakespeare.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PS1-Shakespeare.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mutherr/CS6120-PS1/blob/master/PS1_Shakespeare.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghmXbcymaHxC",
        "colab_type": "text"
      },
      "source": [
        "In this experiment, you will train models to distringuish examples of two different genres of Shakespeare's plays: comedies and tragedies. (We'll ignore the histories, sonnets, etc.) Since he died four hundred years ago, Shakespeare has not written any more plays—although scraps of various other works have come to light. We are not, therefore, interested in building models simply to help categorize an unbounded stream of future documents, as we might be in other applications of text classification; rather, we are interested in what a classifier might have to tell us about what we mean by the terms “comedy” and “tragedy”.\n",
        "\n",
        "You will start by copying and running your `createBasicFeatures` function from the experiment with movie reviews. Do the features the classifier focuses on tell you much about comedy and tragedy in general?\n",
        "\n",
        "You will then implement another featurization function `createInterestingFeatures`, which will focus on only those features you think are informative for distinguishing between comedy and tragedy. Accuracy on leave-one-out cross-validation may go up, but it more important to look at the features given the highest weight by the classifier. Interpretability in machine learning, of course, may be harder to define than accuracy—although accuracy at some tasks such as summarization is hard enoough."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AdVS67_HNRmW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import json\n",
        "import requests\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import cross_validate,LeaveOneOut\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PzjMY8fYQbB6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#read in the shakespeare corpus\n",
        "def readShakespeare():\n",
        "  raw = requests.get(\"https://raw.githubusercontent.com/mutherr/CS6120-PS1-data/master/shakespeare_plays.json\").text.strip()\n",
        "  corpus = [json.loads(line) for line in raw.split(\"\\n\")]\n",
        "\n",
        "  #remove histories from the data, as we're only working with tragedies and comedies\n",
        "  corpus = [entry for entry in corpus if entry[\"genre\"] != \"history\"]\n",
        "  return corpus"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v0r3oL3TdHIU",
        "colab_type": "text"
      },
      "source": [
        "This is where you will implement two functions to featurize the data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "039fPQcF7OkN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#NB: The current contents are for testing only\n",
        "#This function should return: \n",
        "#  -a sparse numpy matrix of document features\n",
        "#  -a list of the correct genre for each document\n",
        "#  -a list of the vocabulary used by the features, such that the ith term of the\n",
        "#    list is the word whose counts appear in the ith column of the matrix. \n",
        "\n",
        "# This function should create a feature representation using all tokens that\n",
        "# contain an alphabetic character.\n",
        "def createBasicFeatures(corpus):\n",
        "  #Your code here\n",
        "  \n",
        "  return texts,genres,vocab\n",
        "\n",
        "# This function can add other features you want that help classification\n",
        "# accuracy, such as bigrams, word prefixes and suffixes, etc.\n",
        "def createInterestingFeatures(corpus):\n",
        "  #Your code here\n",
        "\n",
        "  return texts,genres,vocab"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KfTBqBltXe7Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#given a numpy matrix representation of the features for the training set, the \n",
        "# vector of true classes for each example, and the vocabulary as described \n",
        "# above, this computes the accuracy of the model using leave one out cross \n",
        "# validation and reports the most indicative features for each class\n",
        "def evaluateModel(X,y,vocab,penalty=\"l1\"):\n",
        "  #create and fit the model\n",
        "  model = LogisticRegression(penalty=penalty,solver=\"liblinear\")\n",
        "  results = cross_validate(model,X,y,cv=LeaveOneOut())\n",
        "  \n",
        "  #determine the average accuracy\n",
        "  scores = results[\"test_score\"]\n",
        "  avg_score = sum(scores)/len(scores)\n",
        "  \n",
        "  #determine the most informative features\n",
        "  # this requires us to fit the model to everything, because we need a\n",
        "  # single model to draw coefficients from, rather than 26\n",
        "  model.fit(X,y)\n",
        "  neg_class_prob_sorted = model.coef_[0, :].argsort()\n",
        "  pos_class_prob_sorted = (-model.coef_[0, :]).argsort()\n",
        "\n",
        "  termsToTake = 20\n",
        "  pos_indicators = [vocab[i] for i in neg_class_prob_sorted[:termsToTake]]\n",
        "  neg_indicators = [vocab[i] for i in pos_class_prob_sorted[:termsToTake]]\n",
        "\n",
        "  return avg_score,pos_indicators,neg_indicators\n",
        "\n",
        "def runEvaluation(X,y,vocab):\n",
        "  print(\"----------L1 Norm-----------\")\n",
        "  avg_score,pos_indicators,neg_indicators = evaluateModel(X,y,vocab,\"l1\")\n",
        "  print(\"The model's average accuracy is %f\"%avg_score)\n",
        "  print(\"The most informative terms for pos are: %s\"%pos_indicators)\n",
        "  print(\"The most informative terms for neg are: %s\"%neg_indicators)\n",
        "  #this call will fit a model with L2 normalization\n",
        "  print(\"----------L2 Norm-----------\")\n",
        "  avg_score,pos_indicators,neg_indicators = evaluateModel(X,y,vocab,\"l2\")\n",
        "  print(\"The model's average accuracy is %f\"%avg_score)\n",
        "  print(\"The most informative terms for pos are: %s\"%pos_indicators)\n",
        "  print(\"The most informative terms for neg are: %s\"%neg_indicators)\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mtlbg_aqdcW_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "corpus = readShakespeare()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vpCqIhiqdhen",
        "colab_type": "text"
      },
      "source": [
        "Run the following to train and evaluate two models with basic features:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4IpJ7PKjvc8I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X,y,vocab = createBasicFeatures(corpus)\n",
        "runEvaluation(X, y, vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FmW9F6ZdoyM",
        "colab_type": "text"
      },
      "source": [
        "Run the following to train and evaluate two models with features that are interesting for distinguishing comedy and tragedy:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BW0Ajs2MdtIC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X,y,vocab = createInterestingFeatures(corpus)\n",
        "runEvaluation(X, y, vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1iHudrPb5NPY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}