2. Collection Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collection Analysis with Open Collections API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Import and setup all the things"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import json, requests, math, re, string, nltk\n",
    "\n",
    "# allow matplotlib to run in-line\n",
    "% matplotlib inline \n",
    "\n",
    "ocUrl = 'https://open.library.ubc.ca/'\n",
    "ocApiUrl = 'https://oc-index.library.ubc.ca' # APPY URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set our API Key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get your own API key at https://open.library.ubc.ca/research"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "apiKey = 'ac40e6c2cb345593ed1691e0a8b601bba398e42d85f81f893c5ab709cec63c6c'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose a collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "collection = 'darwin'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get collection info using the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Query the API for the collection information\n",
    "collectionUrl = ocApiUrl+'/collections/'+collection+'?api_key='+apiKey\n",
    "apiResponse = requests.get(collectionUrl).json()\n",
    "\n",
    "# Get count of items in collection\n",
    "itemCount = apiResponse['data']['items']\n",
    "\n",
    "# Get collection name\n",
    "collectionTitle = apiResponse['data']['title']\n",
    "'Collection: \"' + collectionTitle + '\" has ' + str(itemCount) + ' items'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GET and store the collections items using the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "perPage = 25\n",
    "offset = 0\n",
    "pages = math.ceil(itemCount / 25)\n",
    "\n",
    "# Loop through collection item pages to get all items\n",
    "itemIds = []\n",
    "for x in range(0, pages):\n",
    "    collectionItemsUrl = ocApiUrl+'/collections/'+collection\n",
    "    collectionItemsUrl += '/items?limit='+str(perPage)+'&offset='+str(offset)+'&api_key='+apiKey\n",
    "    offset += 25 \n",
    "    # Get list of 25 items\n",
    "    apiResponse = requests.get(collectionItemsUrl).json()\n",
    "    collectionItems = apiResponse['data']\n",
    "    # Add each item id to the itemIds list\n",
    "    for collectionItem in collectionItems:\n",
    "        itemIds.append(collectionItem['_id'])\n",
    "print(itemIds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GET and store items and full text using the API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "items = []\n",
    "fullTexts = []\n",
    "for itemId in itemIds : \n",
    "    itemUrl = ocApiUrl+'/collections/'+collection+'/items/'+itemId\n",
    "    apiResponse = requests.get(itemUrl).json()\n",
    "    item = apiResponse['data']\n",
    "    easyItem = dict()\n",
    "    easyItem['id'] = itemId\n",
    "    easyItem['title'] = item['Title'][0]['value']\n",
    "    if 'FullText' in item:\n",
    "        fullText = item['FullText'][0]['value']\n",
    "        # Lower case full text\n",
    "        cleanFullText = fullText.lower()\n",
    "        # Remove everything but words\n",
    "        pattern = re.compile('[\\W_]+')\n",
    "        cleanFullText = pattern.sub(' ', cleanFullText)\n",
    "        # Add to the full texts list\n",
    "        fullTexts.append(cleanFullText)\n",
    "        easyItem['fullText'] = item['FullText'][0]['value']\n",
    "    else:\n",
    "        easyItem['fullText'] = ''\n",
    "        fullTexts.append('')\n",
    "        \n",
    "    items.append(easyItem)\n",
    "        \n",
    "print(fullTexts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Analysis with NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we have the item's full text we are going to use the Natural Language Toolkit to perform some analysis on it using NLTK.\n",
    "\n",
    "NLTK is a Python Library for working with written language data. It is free and very well documented. Many areas we'll be covering are treated in more detail in the NLTK Book, available for free online from [here](http://www.nltk.org/book/).\n",
    "\n",
    "> Note: NLTK provides tools for tasks ranging from very simple (counting words in a text) to very complex (writing and training parsers, etc.). Many advanced tasks are beyond the scope of this talk, but by the time we're done, you should understand Python and NLTK well enough to perform these tasks on your own!\n",
    "\n",
    "Firstly, we will need to import NLTK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk # imports all the nltk basics\n",
    "nltk.download(\"punkt\") # Word tokenizer\n",
    "nltk.download(\"stopwords\") # Stop words\n",
    "from nltk import word_tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Total character count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "characterLength = 0\n",
    "for fullText in fullTexts:\n",
    "    characterLength += len(fullText)\n",
    "print(characterLength)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Total word count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*For our analysis, we want to break up the full text into words, this step is called tokenization*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "totalTokens = []\n",
    "\n",
    "c = 0\n",
    "for fullText in fullTexts:\n",
    "    tokenisedText = word_tokenize(fullText)\n",
    "    totalTokens += tokenisedText\n",
    "    items[c]['words'] = tokenisedText\n",
    "    c = c + 1\n",
    "#print(tokens)\n",
    "len(totalTokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Item with most words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "highestWordCount = 0;\n",
    "highestId= 0;\n",
    "\n",
    "for key, item in enumerate(items):\n",
    "    if(len(item['words']) > highestWordCount):\n",
    "        highestWordCount = len(item['words'])\n",
    "        highestId = key\n",
    "\n",
    "print(ocUrl+collection+'/items/'+items[highestId]['id'] + ' has the most words with ' + str(highestWordCount) + ' words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Total unique word count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(set(totalTokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Item with most unique words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "uniqueWordCount = 0;\n",
    "winner = 0;\n",
    "\n",
    "for key, item in enumerate(items):\n",
    "    if(len(set(item['words'])) > uniqueWordCount):\n",
    "        uniqueWordCount = len(set(item['words']))\n",
    "        winner = key\n",
    "\n",
    "print(ocUrl+collection+'/items/'+items[highestId]['id'] + ' has the most words with ' + str(uniqueWordCount) + ' unique words')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Longest words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "v = set(totalTokens)\n",
    "long_words = [word for word in v if len(word) > 13]\n",
    "sorted(long_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collocations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get sequence of words or terms that co-occur more often than would be expected by chance. \n",
    "text = nltk.Text(totalTokens)\n",
    "text.collocations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk import FreqDist\n",
    "fdist = FreqDist(text)\n",
    "fdist.most_common(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now we can plot the frequency distributions\n",
    "fdist.plot(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hapaxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fdist.hapaxes() # words that occur only once"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}