03_Fetching_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03 Fetching data.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "WsSJbpIYHZyn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Accessing and downloading Data\n",
        "\n",
        "\n",
        "\n",
        "*   Finding the route to get hold of the data\n",
        "*   Converting spreadsheets to CSV\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Po6b5MT5oidh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import requests\n",
        "import json\n",
        "import string\n",
        "\n",
        "import io\n",
        "\n",
        "# Importing the libraries we need to format the data in a more readable way. \n",
        "import pandas as pd\n",
        "from pandas.io.json import json_normalize\n",
        "\n",
        "from IPython.display import HTML"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xtQxEzY_oo2i",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "base_url = 'https://fairdomhub.org'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dWmo665aosLY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def json_for_resource(type, id):    \n",
        "\n",
        "  headers = {\"Accept\": \"application/vnd.api+json\",\n",
        "           \"Accept-Charset\": \"ISO-8859-1\"}\n",
        "  r = requests.get(base_url + \"/\" + type + \"/\" + str(id), headers=headers)\n",
        "  r.raise_for_status()\n",
        "  return r.json()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4rSNZISLH1Y5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Fetch the JSON for the Data file resource at https://fairdomhub.org/data_files/1049\n",
        "\n",
        "We print out title to be sure we have the correct item."
      ]
    },
    {
      "metadata": {
        "id": "ZF69h5IGotLJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data_file_id = 1049\n",
        "\n",
        "result = json_for_resource('data_files',data_file_id)\n",
        "\n",
        "title = result['data']['attributes']['title']\n",
        "\n",
        "title"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mhW8e3_RIKJP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The attributes contain a 'content_blobs' block. Content Blob is the name we use in SEEK for the entity that corresponds to a file or URL.\n",
        "\n",
        "Note that content_blobs is always an array. Models can currently contain multiple content blobs (multiple files), and we plan to provide the same support to Data files and other assets in the future.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "5fzhQETpA3sp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "result['data']['attributes']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iFgiWDYXJZOP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Here we focus on the details about a single content blob:\n",
        "\n",
        "* **content_type** - this is the mime type of the file, or the whatever the URL points to\n",
        "*  **link** - this is the link that describes the content blob route\n",
        "*  **md5sum** - an MD5 checksum of the contents\n",
        "* **sha1sum** - a SHA1 based checksum of the contents. These checksums are useful for checking the file downloaded is correct, and there hasn't been an error or has been modified since being registered with SEEK.\n",
        "* **original_filename** - the filename if the file, as it was when registered with SEEK\n",
        "* **size** - the size of the file in bytes\n",
        "* **url** - url to an external resource, if the item was registered with SEEK using a URL rather than a direct upload\n",
        "\n",
        "In this case, this is an *Excel XLSX* file, called *1205 amino acid omission pyogenes.xlsx*, and is about 59k"
      ]
    },
    {
      "metadata": {
        "id": "9CSi_gawp9O4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "blob = result['data']['attributes']['content_blobs'][0]\n",
        "\n",
        "blob"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TLLf_c8cKMq4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The route to directly download a file is the content blob route, with the */download* action appended. This is always the case for anything downloadable in SEEK.\n",
        "\n",
        "In this example we display the URL to download the content blob for generete a HTML hyperlink for it.\n",
        "\n",
        "Although in this case we download the content blob itself directly, it is also possible to download with https://fairdomhub.org/data_files/1049/download . Other than Models, this currently results in downloading a single file. For models, a ZIP file is generated that contains all files. To be future proof, we recommend downloading individual files through the content-blob route."
      ]
    },
    {
      "metadata": {
        "id": "oHdmzBq0qNa2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "link = blob['link']\n",
        "filename = blob['original_filename']\n",
        "\n",
        "download_link = link+\"/download\"\n",
        "\n",
        "print(\"Download link is: \" + download_link + \"\\n\")\n",
        "\n",
        "HTML(\"<a href='\"+ download_link + \"'>Download + \" + filename + \"</a>\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Y4xbfOhPLP0B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we saw earlier. this Data file is an Excel spreadsheet. Where data is an Excel spreadsheet, it can be converted to a Comma Seperated File (CSV), by requesting this format through content negotiation. \n",
        "\n",
        "In this case, we request a GET to https://fairdomhub.org/data_files/1049/content_blobs/1518, but instead of requesting JSON we use an Accept: header of 'text/csv'. A parameter 'sheet' can be included to access different sheets, which if missed always defaults to the first sheet.\n",
        "\n",
        "Here we request CSV and display the first sheet in a table using the Pandas module. (NaN is just a blank cell in the spreadsheet)."
      ]
    },
    {
      "metadata": {
        "id": "K9TCGAkvraru",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "headers = { \"Accept\": \"text/csv\" }\n",
        "r = requests.get(link, headers=headers, params={'sheet':'1'})\n",
        "r.raise_for_status()\n",
        "\n",
        "csv = pd.read_csv(io.StringIO(r.content.decode('utf-8')))\n",
        "\n",
        "csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1sLkGNptMdpD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Exercise 3\n",
        "\n",
        "\n",
        "\n",
        "*   Update the notebook to display the content blob details and CSV for sheet 2 of https://fairdomhub.org/data_files/2222\n",
        "*   Update and look at the multiple model file content blobs for https://fairdomhub.org/models/308 (don't worry if the csv step fails)\n",
        "\n"
      ]
    }
  ]
}