Sector and news factor.ipynb

{
  "cells": [
    {
      "metadata": {
        "_uuid": "225708f447eee93041881f9d6c3a3e890cb16718"
      },
      "cell_type": "markdown",
      "source": "## In-depth Introduction\nFirst let's import the module and create an environment."
    },
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "_kg_hide-output": false,
        "scrolled": true
      },
      "cell_type": "code",
      "source": "from kaggle.competitions import twosigmanews\n# You can only call make_env() once, so don't lose it!\nenv = twosigmanews.make_env()",
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "error",
          "ename": "Exception",
          "evalue": "You can only make one environment for this competition.",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-ecdb46c2c700>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mkaggle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompetitions\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtwosigmanews\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# You can only call make_env() once, so don't lose it!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0menv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtwosigmanews\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_env\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/kaggle/lib/kaggle/competitions/twosigmanews/__init__.py\u001b[0m in \u001b[0;36mmake_env\u001b[0;34m()\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_env\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0;34m\"\"\"Returns a new environment supporting the Two Sigma News competition.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTwoSigmaNewsEnv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0m__all__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'make_env'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/kaggle/lib/kaggle/competitions/twosigmanews/env.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    151\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mTwoSigmaNewsEnv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_state\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0m_var00\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_var05\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    152\u001b[0m             raise Exception(\n\u001b[0;32m--> 153\u001b[0;31m                     'You can only make one environment for this competition.')\n\u001b[0m\u001b[1;32m    154\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    155\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mon_load_failure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mException\u001b[0m: You can only make one environment for this competition."
          ]
        }
      ]
    },
    {
      "metadata": {
        "_uuid": "6034b46fce8c9d55d403de32e7cebe8cb9fef96d"
      },
      "cell_type": "markdown",
      "source": "## **`get_training_data`** function\n\nReturns the training data DataFrames as a tuple of:\n* `market_train_df`: DataFrame with market training data\n* `news_train_df`: DataFrame with news training data\n\nThese DataFrames contain all market and news data from February 2007 to December 2016.  See the [competition's Data tab](https://www.kaggle.com/c/two-sigma-financial-news/data) for more information on what columns are included in each DataFrame."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c20fa6deeac9d374c98774abd90bdc76b023ee63"
      },
      "cell_type": "code",
      "source": "(market_train_df, news_train_df) = env.get_training_data()",
      "execution_count": 4,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6811a1a76f08b2a029543cf73bcdf4dfca7dc362"
      },
      "cell_type": "code",
      "source": "market_train_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "84b5a58f67ebded82e6aabc66ca36411e6db35a9"
      },
      "cell_type": "code",
      "source": "market_train_df.tail()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "25115010e14ef3497932902db0cef68501ddca11"
      },
      "cell_type": "code",
      "source": "news_train_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "cdc6b9842073bcb7d63cddc30e5bd7826ccfdfa1"
      },
      "cell_type": "code",
      "source": "news_train_df.tail()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0c1438bcaff717c8fa04085862ff8784871d7c11"
      },
      "cell_type": "code",
      "source": "news_train_df.iloc[-10:]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5044c55a8952d387f4dec4dba555654aef072219"
      },
      "cell_type": "code",
      "source": "#divide into 3 classes\nfrom scipy.stats import describe\nimport numpy as np\nimport matplotlib.pyplot as plt\nreturns = market_train_df['returnsOpenNextMktres10']\nvalid_idx = np.abs(returns) < 0.4\nvalid_returns = returns[valid_idx]\nmarket_train_df = market_train_df[valid_idx]\nprint(describe(returns))\ncutoff_1 = valid_returns.quantile(0.33)\ncutoff_2 = valid_returns.quantile(0.66)\nreturn_class = np.zeros(market_train_df.shape[0])\nreturn_class[valid_returns < cutoff_1] = -1\nreturn_class[valid_returns > cutoff_2] = 1\nmarket_train_df['class'] = return_class\nmarket_train_df.drop(columns = ['returnsClosePrevMktres10', 'returnsOpenPrevMktres10'])\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "6ce5aff371858acab26655629781df7618e46650"
      },
      "cell_type": "code",
      "source": "print(len(valid_returns))\nlen(returns)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "83dbca14efc952a6951a64726264990eb2225082"
      },
      "cell_type": "code",
      "source": "import pandas as pd\n\n#First, assign label to each piece of news\n#The label is the accumulated return over the next 10 business days after the news is released\ntext = news_train_df['headline']\nnum_rows = news_train_df.shape[0]\nlabels = np.ones(num_rows)\nfor i in range(num_rows):\n    x = news_train_df.iloc[i]\n    tmp = market_train_df[(market_train_df['time'] > x['time']) & (market_train_df['assetCode'].isin(list(eval(x['assetCodes']))))]\n    tmp = tmp[np.asarray([diff.days for diff in tmp.time - x.time]) <= 10]\n    labels[i] = np.prod([(1+num) for num in tmp.returnsOpenNextMktres10]) \nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "e4b656414cee1bef4e7d93c54493adf0e247278f"
      },
      "cell_type": "code",
      "source": "#Encode the news data\nimport chakin\n# chakin.search(lang=\"English\")\nchakin.download(number=21)\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "857174261b1b1ef8ef85e5386f3aaeda1c09b822"
      },
      "cell_type": "code",
      "source": "import pandas as pd\nbasic = pd.read_csv('../input/sector/basic industries.csv')['Symbol']\ncapital = pd.read_csv('../input/sector/capital_goods.csv')['Symbol']\nprod = pd.read_csv('../input/sector/consumer products.csv')['Symbol']\nservice = pd.read_csv('../input/sector/consumer service.csv')['Symbol']\nenergy = pd.read_csv('../input/sector/energy.csv')['Symbol']\nfinance = pd.read_csv('../input/sector/finance.csv')['Symbol']\nhealthcare = pd.read_csv('../input/sector/healthcare.csv')['Symbol']\nmisc = pd.read_csv('../input/sector/miscellaneous.csv')['Symbol']\ntech = pd.read_csv('../input/sector/technology.csv')['Symbol']\ntransport = pd.read_csv('../input/sector/transportation.csv')['Symbol']\nutility = pd.read_csv('../input/sector/utilities.csv')['Symbol']\nsectors = [basic, capital, prod, service, energy, finance, healthcare, misc,\n          tech, transport, utility]\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "713190d017a9713d327b7831078eb68379bc62a9"
      },
      "cell_type": "code",
      "source": "#sector name cleaning\nfor i, sec in enumerate(sectors):\n    for j, name in enumerate(sec):\n        sectors[i][j] = sectors[i][j].strip()\n    sectors[i] = set(sectors[i])\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5754f88753e2872dd68e5122083e1dffc39b9971"
      },
      "cell_type": "code",
      "source": "import numpy as np\nsectorMapping = dict()\nnum_rows = market_train_df.shape[0]\nindustry_class = np.zeros(num_rows)\nfor i in range(num_rows):\n    if i % 500000 == 0:\n        print ('step ' + str(i//500000) + ' is done')\n    name = market_train_df.loc[i, 'assetCode']\n    name = name[:name.find('.')] if name.find('.') != -1 else name\n    existed = False\n    for j, sec in enumerate(sectors):\n        if name in sec:\n            industry_class[i] = j + 1\n            sectorMapping[name] = j + 1\n            existed = True\n            break\n    if existed == False:\n        sectorMapping[name] = 0\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f2cdedf78e0da785b9ba8ccd0f157c96fb02504b"
      },
      "cell_type": "code",
      "source": "import pickle\nwith open('sectorMapping.pkl','wb') as f:\n    pickle.dump(sectorMapping, f, pickle.HIGHEST_PROTOCOL)\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b2498e543599a808dca3dc164834e35c8b8ffc6c"
      },
      "cell_type": "code",
      "source": "with open('sectorMapping.pkl', 'rb') as f:\n    x = pickle.load(f)\nprint ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3696e3d37075c1f4559909c4a58c4742533de954"
      },
      "cell_type": "code",
      "source": "import matplotlib.pyplot as plt\nplt.hist(industry_class)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a319deb5e7933d94e9107cb4f2ab8ca9296b0aa0"
      },
      "cell_type": "code",
      "source": "#industry classification for news data\nimport numpy as np\ntopic_codes = news_train_df['subjects']\nindustry_codes = set([\"ADV\",\"APL\",\"BIOT\",\"BUS\",\"DPR\",\"ELI\",\"GDM\",\"IND\",\n                      \"MAC\",\"MUL\",\"REAM\",\"RRL\",\"STL\",\"TEX\",\"WWW\",\n                      \"AER\",\"AUT\",\"BLD\",\"CHE\",\"DRU\",\"FIN\",\"GSFT\",\n                      \"INS\",\"MET\",\"PUB\",\"REC\",\"SHP\",\"TEL\",\"TIM\",\n                      \"AIR\",\"BEV\",\"BNK\",\"CON\",\"ELC\",\"FOD\",\"HDWR\",\n                      \"LEI\",\"MIS\",\"REA\",\"RET\",\"SFWR\",\"TBCS\",\"WHO\"])\n\nsectors = [basic, capital, prod, service, energy, finance, healthcare, misc,\n          tech, transport, utility]\n\nGICS = ['Communication services','Consumer discretionary','Consumer staples','Energy','Financials',\n        'Health care','Industrials','Information technology','Materials','Real estate','Utilities']\n\ncom = ['ELC','TEL']\ndisc = ['ADV','APL','AUT','LEI','PUB','REC','RET','TBCS','TEX']\nstaples = ['BEV','FOD',] \nenergy = ['ENR','ENQ',]\nfin = ['BNK','FIN','INS']\nhealth = ['BIOT','DRU',]\nind = ['AER','AIR','BLD','BUS','CON','IND','RRL','SHP','WHO']\ninfo = ['DPR','ELI','GSFT','HDWR','SFWR','WWW']\nmaterial = ['CHE','GDM','MAC','MET','MIS','STL','TIM',]\nestate = ['REA','REAM']\nutil = ['ELG']\ngeneral = ['MUL']\nsector_list = [com,disc,staples,energy,fin,health,ind,info,material,\n              estate,util]\nindustry_class = np.zeros(len(topic_codes))\nfor i,topic in enumerate(topic_codes):\n    for j,sector_codes in enumerate(sector_list):\n        if topic in sector_codes:\n            industry_class[i] = j + 1\nprint ('Done')\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "eed91eaea71acabc9543249bba29e3caa2cfea72"
      },
      "cell_type": "code",
      "source": "news_train_df.head()",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": "                       time      ...       volumeCounts7D\n0 2007-01-01 04:29:32+00:00      ...                    7\n1 2007-01-01 07:03:35+00:00      ...                    3\n2 2007-01-01 11:29:56+00:00      ...                   17\n3 2007-01-01 12:08:37+00:00      ...                   15\n4 2007-01-01 12:08:37+00:00      ...                    0\n\n[5 rows x 35 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>time</th>\n      <th>sourceTimestamp</th>\n      <th>firstCreated</th>\n      <th>sourceId</th>\n      <th>headline</th>\n      <th>urgency</th>\n      <th>takeSequence</th>\n      <th>provider</th>\n      <th>subjects</th>\n      <th>audiences</th>\n      <th>bodySize</th>\n      <th>companyCount</th>\n      <th>headlineTag</th>\n      <th>marketCommentary</th>\n      <th>sentenceCount</th>\n      <th>wordCount</th>\n      <th>assetCodes</th>\n      <th>assetName</th>\n      <th>firstMentionSentence</th>\n      <th>relevance</th>\n      <th>sentimentClass</th>\n      <th>sentimentNegative</th>\n      <th>sentimentNeutral</th>\n      <th>sentimentPositive</th>\n      <th>sentimentWordCount</th>\n      <th>noveltyCount12H</th>\n      <th>noveltyCount24H</th>\n      <th>noveltyCount3D</th>\n      <th>noveltyCount5D</th>\n      <th>noveltyCount7D</th>\n      <th>volumeCounts12H</th>\n      <th>volumeCounts24H</th>\n      <th>volumeCounts3D</th>\n      <th>volumeCounts5D</th>\n      <th>volumeCounts7D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2007-01-01 04:29:32+00:00</td>\n      <td>2007-01-01 04:29:32+00:00</td>\n      <td>2007-01-01 04:29:32+00:00</td>\n      <td>e58c6279551b85cf</td>\n      <td>China's Daqing pumps 43.41 mln tonnes of oil i...</td>\n      <td>3</td>\n      <td>1</td>\n      <td>RTRS</td>\n      <td>{'ENR', 'ASIA', 'CN', 'NGS', 'EMRG', 'RTRS', '...</td>\n      <td>{'Z', 'O', 'OIL'}</td>\n      <td>1438</td>\n      <td>1</td>\n      <td></td>\n      <td>False</td>\n      <td>11</td>\n      <td>275</td>\n      <td>{'0857.HK', '0857.F', '0857.DE', 'PTR.N'}</td>\n      <td>PetroChina Co Ltd</td>\n      <td>6</td>\n      <td>0.235702</td>\n      <td>-1</td>\n      <td>0.500739</td>\n      <td>0.419327</td>\n      <td>0.079934</td>\n      <td>73</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>6</td>\n      <td>7</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2007-01-01 07:03:35+00:00</td>\n      <td>2007-01-01 07:03:34+00:00</td>\n      <td>2007-01-01 07:03:34+00:00</td>\n      <td>5a31c4327427f63f</td>\n      <td>FEATURE-In kidnapping, finesse works best</td>\n      <td>3</td>\n      <td>1</td>\n      <td>RTRS</td>\n      <td>{'FEA', 'CA', 'LATAM', 'MX', 'INS', 'ASIA', 'I...</td>\n      <td>{'PGE', 'PCO', 'G', 'ESN', 'MD', 'PCU', 'DNP',...</td>\n      <td>4413</td>\n      <td>1</td>\n      <td>FEATURE</td>\n      <td>False</td>\n      <td>55</td>\n      <td>907</td>\n      <td>{'STA.N'}</td>\n      <td>Travelers Companies Inc</td>\n      <td>8</td>\n      <td>0.447214</td>\n      <td>-1</td>\n      <td>0.600082</td>\n      <td>0.345853</td>\n      <td>0.054064</td>\n      <td>62</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>3</td>\n      <td>3</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2007-01-01 11:29:56+00:00</td>\n      <td>2007-01-01 11:29:56+00:00</td>\n      <td>2007-01-01 11:29:56+00:00</td>\n      <td>1cefd27a40fabdfe</td>\n      <td>PRESS DIGEST - Wall Street Journal - Jan 1</td>\n      <td>3</td>\n      <td>1</td>\n      <td>RTRS</td>\n      <td>{'RET', 'ENR', 'ID', 'BG', 'US', 'PRESS', 'IQ'...</td>\n      <td>{'T', 'DNP', 'PSC', 'U', 'D', 'M', 'RNP', 'PTD...</td>\n      <td>2108</td>\n      <td>2</td>\n      <td>PRESS DIGEST</td>\n      <td>False</td>\n      <td>15</td>\n      <td>388</td>\n      <td>{'WMT.DE', 'WMT.N'}</td>\n      <td>Wal-Mart Stores Inc</td>\n      <td>14</td>\n      <td>0.377964</td>\n      <td>-1</td>\n      <td>0.450049</td>\n      <td>0.295671</td>\n      <td>0.254280</td>\n      <td>67</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>11</td>\n      <td>17</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>23768af19dc69992</td>\n      <td>PRESS DIGEST - New York Times - Jan 1</td>\n      <td>3</td>\n      <td>1</td>\n      <td>RTRS</td>\n      <td>{'FUND', 'FIN', 'CA', 'SFWR', 'INS', 'PUB', 'B...</td>\n      <td>{'T', 'DNP', 'PSC', 'U', 'D', 'M', 'RNP', 'PTD...</td>\n      <td>1776</td>\n      <td>6</td>\n      <td>PRESS DIGEST</td>\n      <td>False</td>\n      <td>14</td>\n      <td>325</td>\n      <td>{'GOOG.O', 'GOOG.OQ', 'GOOGa.DE'}</td>\n      <td>Google Inc</td>\n      <td>13</td>\n      <td>0.149071</td>\n      <td>-1</td>\n      <td>0.752917</td>\n      <td>0.162715</td>\n      <td>0.084368</td>\n      <td>83</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>5</td>\n      <td>13</td>\n      <td>15</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>2007-01-01 12:08:37+00:00</td>\n      <td>23768af19dc69992</td>\n      <td>PRESS DIGEST - New York Times - Jan 1</td>\n      <td>3</td>\n      <td>1</td>\n      <td>RTRS</td>\n      <td>{'FUND', 'FIN', 'CA', 'SFWR', 'INS', 'PUB', 'B...</td>\n      <td>{'T', 'DNP', 'PSC', 'U', 'D', 'M', 'RNP', 'PTD...</td>\n      <td>1776</td>\n      <td>6</td>\n      <td>PRESS DIGEST</td>\n      <td>False</td>\n      <td>14</td>\n      <td>325</td>\n      <td>{'XMSR.O'}</td>\n      <td>XM Satellite Radio Holdings Inc</td>\n      <td>11</td>\n      <td>0.149071</td>\n      <td>-1</td>\n      <td>0.699274</td>\n      <td>0.209360</td>\n      <td>0.091366</td>\n      <td>102</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "87898ef0148e2900398bf60af938859525eb54de"
      },
      "cell_type": "code",
      "source": "#Construct novelty related factors\nimport numpy as np\nfrom scipy.stats import describe\nnovelty0 = news_train_df['noveltyCount12H']\nnovelty1 = news_train_df['noveltyCount24H'] - news_train_df['noveltyCount12H']\nnovelty2 = news_train_df['noveltyCount3D'] - news_train_df['noveltyCount24H']\nnovelty3 = news_train_df['noveltyCount5D'] - news_train_df['noveltyCount3D']\nnovelty4 = news_train_df['noveltyCount7D'] - news_train_df['noveltyCount5D']\nnovelty = np.column_stack((novelty0,novelty1,novelty2,novelty3,novelty4))\nt = np.array([0.25, 0.75, 2, 4, 6])\ndecay_novelty = - t * novelty\ndecay_novelty = np.sum(np.exp(decay_novelty), axis = 1)\n\nvolumn0 = news_train_df['volumeCounts12H']\nvolumn1 = news_train_df['volumeCounts24H'] - news_train_df['volumeCounts12H']\nvolumn2 = news_train_df['volumeCounts3D'] - news_train_df['volumeCounts24H']\nvolumn3 = news_train_df['volumeCounts5D'] - news_train_df['volumeCounts3D']\nvolumn4 = news_train_df['volumeCounts7D'] - news_train_df['volumeCounts5D']\nvolumn = np.column_stack((volumn0,volumn1,volumn2,volumn3,volumn4))\ndecay_volumn = - t * volumn\ndecay_volumn = np.sum(np.exp(decay_volumn), axis = 1)\n\ninfo_novelty = describe(decay_novelty)\ninfo_volumn = describe(decay_volumn)\n\nprint (describe(decay_novelty))\nprint (describe(decay_volumn))\n\nprint ('Done')",
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": "DescribeResult(nobs=9328750, minmax=(9.995705704761785e-09, 5.0), mean=4.60058717598803, variance=0.5314065756566537, skewness=-3.3361984203958133, kurtosis=13.687558154422085)\nDescribeResult(nobs=9328750, minmax=(3.3427955219162848e-80, 5.0), mean=2.555924856047671, variance=2.5442995571699183, skewness=-0.11380412585219192, kurtosis=-1.2377537087664192)\nDone\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9b5aa2ad284daa3c5d2fc6e9096029b201486917"
      },
      "cell_type": "code",
      "source": "#Construct novelty-adjusted sentiment factor\ncols = ['sentimentClass','sentimentNegative','sentimentNeutral','sentimentPositive']\ndf = news_train_df[cols]\ndf['decay_novelty'] = decay_novelty\ndf['sentimentClass'] += 1\nprint ('Done')",
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Done\n",
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:4: SettingWithCopyWarning: \nA value is trying to be set on a copy of a slice from a DataFrame.\nTry using .loc[row_indexer,col_indexer] = value instead\n\nSee the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n  after removing the cwd from sys.path.\n/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \nA value is trying to be set on a copy of a slice from a DataFrame.\nTry using .loc[row_indexer,col_indexer] = value instead\n\nSee the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n  \"\"\"\n",
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3f39158e938f6096e62a99d26de44345bffdfdf7"
      },
      "cell_type": "code",
      "source": "import matplotlib.pyplot as plt\nplt.hist(news_train_df['sentimentClass'])",
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 52,
          "data": {
            "text/plain": "(array([2813011.,       0.,       0.,       0.,       0., 2899088.,\n              0.,       0.,       0., 3616651.]),\n array([-1. , -0.8, -0.6, -0.4, -0.2,  0. ,  0.2,  0.4,  0.6,  0.8,  1. ]),\n <a list of 10 Patch objects>)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 432x288 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAD8CAYAAACyyUlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGFlJREFUeJzt3X+wX3Wd3/Hny0SQ1ioBUsoSxuCa1kY7BkwxrZ2uwi4EdsbgFG2Y2SVrU6MrdHan245h/QNXZQqd2WWGqbJllyzBbkWK65CuoWkEHGdnlh9xjUBgkSvokDSSLOHHOo4o+O4f3891D9fv/XnuvV8Cz8fMme/5vs/nfD6fe+7Nfd3vOef7TaoKSZL6eM2oJyBJOvoZJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0tHfUEFstJJ51UK1euHPU0JOmo8o1vfONvqmr5dO1eNWGycuVK9uzZM+ppSNJRJcn3ZtLO01ySpN4ME0lSb4aJJKk3w0SS1JthIknqzTCRJPVmmEiSejNMJEm9GSaSpN5eNe+Al6RRWrn1KyMb+7tX/eqCj+ErE0lSb4aJJKm3acMkyeuS3JvkW0n2Jfm9Vr8xyeNJ9rZlTasnybVJxpLcn+TMTl+bkjzalk2d+juTPND2uTZJWv2EJLtb+91Jlk03hiRp8c3klcnzwNlV9Q5gDbA+ybq27T9X1Zq27G2184FVbdkCXAeDYACuAN4FnAVcMR4Orc2HO/utb/WtwB1VtQq4oz2fdAxJ0mhMGyY18IP29LVtqSl22QDc1Pa7Gzg+ySnAecDuqjpSVU8DuxkE0ynAG6rq7qoq4Cbgwk5f29v69gn1YWNIkkZgRtdMkixJshc4xCAQ7mmbrmynma5JcmyrnQo80dl9f6tNVd8/pA5wclUdbOvfB06eZoyJ896SZE+SPYcPH57JlypJmoMZhUlVvVhVa4AVwFlJ3g5cDrwV+OfACcDHF2yWgzkUU78iGrbP9VW1tqrWLl8+7X8UJkmao1ndzVVVzwB3Aeur6mA7zfQ88CcMroMAHABO6+y2otWmqq8YUgd4cvz0VXs8NM0YkqQRmMndXMuTHN/WjwN+Bfjrzi/5MLiW8WDbZQdwSbvjah3wbDtVtQs4N8myduH9XGBX2/ZcknWtr0uA2zp9jd/1tWlCfdgYkqQRmMk74E8BtidZwiB8bqmqP09yZ5LlQIC9wEdb+53ABcAY8EPgQwBVdSTJp4H7WrtPVdWRtv4x4EbgOOD2tgBcBdySZDPwPeCDU40hSRqNacOkqu4HzhhSP3uS9gVcOsm2bcC2IfU9wNuH1J8CzpnNGJKkxec74CVJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm/ThkmS1yW5N8m3kuxL8nutfnqSe5KMJflikmNa/dj2fKxtX9np6/JWfyTJeZ36+lYbS7K1U5/1GJKkxTeTVybPA2dX1TuANcD6JOuAq4FrquotwNPA5tZ+M/B0q1/T2pFkNbAReBuwHvhckiVJlgCfBc4HVgMXt7bMdgxJ0mhMGyY18IP29LVtKeBs4NZW3w5c2NY3tOe07eckSavfXFXPV9XjwBhwVlvGquqxqvoxcDOwoe0z2zEkSSMwo2sm7RXEXuAQsBv4DvBMVb3QmuwHTm3rpwJPALTtzwIndusT9pmsfuIcxpAkjcCMwqSqXqyqNcAKBq8k3rqgs5onSbYk2ZNkz+HDh0c9HUl6xZrV3VxV9QxwF/AvgOOTLG2bVgAH2voB4DSAtv2NwFPd+oR9Jqs/NYcxJs73+qpaW1Vrly9fPpsvVZI0CzO5m2t5kuPb+nHArwAPMwiVi1qzTcBtbX1He07bfmdVVatvbHdinQ6sAu4F7gNWtTu3jmFwkX5H22e2Y0iSRmDp9E04Bdje7rp6DXBLVf15koeAm5N8BvgmcENrfwPw+SRjwBEG4UBV7UtyC/AQ8AJwaVW9CJDkMmAXsATYVlX7Wl8fn80YkqTRmDZMqup+4Iwh9ccYXD+ZWP8R8IFJ+roSuHJIfSewcz7GkCQtPt8BL0nqbSanuSQtsJVbvzKScb971a+OZFy98vjKRJLUm2EiSerNMJEk9WaYSJJ68wL8DIzq4ih4gVTS0cFXJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLUm2EiSept2jBJclqSu5I8lGRfkt9q9U8mOZBkb1su6OxzeZKxJI8kOa9TX99qY0m2duqnJ7mn1b+Y5JhWP7Y9H2vbV043hiRp8c3klckLwO9U1WpgHXBpktVt2zVVtaYtOwHato3A24D1wOeSLEmyBPgscD6wGri408/Vra+3AE8Dm1t9M/B0q1/T2k06xpyPgiSpl2nDpKoOVtVftfW/BR4GTp1ilw3AzVX1fFU9DowBZ7VlrKoeq6ofAzcDG5IEOBu4te2/Hbiw09f2tn4rcE5rP9kYkqQRmNU1k3aa6Qzgnla6LMn9SbYlWdZqpwJPdHbb32qT1U8EnqmqFybUX9JX2/5saz9ZX5KkEZhxmCR5PfAl4Ler6jngOuAXgTXAQeD3F2SGPSTZkmRPkj2HDx8e9XQk6RVrRmGS5LUMguRPq+rPAKrqyap6sap+CvwRf3ea6QBwWmf3Fa02Wf0p4PgkSyfUX9JX2/7G1n6yvl6iqq6vqrVVtXb58uUz+VIlSXMwk7u5AtwAPFxVf9Cpn9Jp9n7gwba+A9jY7sQ6HVgF3AvcB6xqd24dw+AC+o6qKuAu4KK2/ybgtk5fm9r6RcCdrf1kY0iSRmDp9E14N/DrwANJ9rba7zK4G2sNUMB3gY8AVNW+JLcADzG4E+zSqnoRIMllwC5gCbCtqva1/j4O3JzkM8A3GYQX7fHzScaAIwwCaMoxJEmLb9owqaq/ADJk084p9rkSuHJIfeew/arqMYbcjVVVPwI+MJsxJEmLz3fAS5J6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0ZJpKk3gwTSVJv04ZJktOS3JXkoST7kvxWq5+QZHeSR9vjslZPkmuTjCW5P8mZnb42tfaPJtnUqb8zyQNtn2uTZK5jSJIW30xembwA/E5VrQbWAZcmWQ1sBe6oqlXAHe05wPnAqrZsAa6DQTAAVwDvAs4CrhgPh9bmw5391rf6rMaQJI3GtGFSVQer6q/a+t8CDwOnAhuA7a3ZduDCtr4BuKkG7gaOT3IKcB6wu6qOVNXTwG5gfdv2hqq6u6oKuGlCX7MZQ5I0ArO6ZpJkJXAGcA9wclUdbJu+D5zc1k8Fnujstr/VpqrvH1JnDmNMnO+WJHuS7Dl8+PDMvkhJ0qzNOEySvB74EvDbVfVcd1t7RVHzPLeXmMsYVXV9Va2tqrXLly9foJlJkmYUJkleyyBI/rSq/qyVnxw/tdQeD7X6AeC0zu4rWm2q+ooh9bmMIUkagZnczRXgBuDhqvqDzqYdwPgdWZuA2zr1S9odV+uAZ9upql3AuUmWtQvv5wK72rbnkqxrY10yoa/ZjCFJGoGlM2jzbuDXgQeS7G213wWuAm5Jshn4HvDBtm0ncAEwBvwQ+BBAVR1J8mngvtbuU1V1pK1/DLgROA64vS3MdgxJ0mhMGyZV9RdAJtl8zpD2BVw6SV/bgG1D6nuAtw+pPzXbMSRJi893wEuSejNMJEm9GSaSpN4ME0lSb4aJJKk3w0SS1JthIknqzTCRJPVmmEiSejNMJEm9GSaSpN4ME0lSb4aJJKk3w0SS1JthIknqzTCRJPVmmEiSejNMJEm9GSaSpN6mDZMk25IcSvJgp/bJJAeS7G3LBZ1tlycZS/JIkvM69fWtNpZka6d+epJ7Wv2LSY5p9WPb87G2feV0Y0iSRmMmr0xuBNYPqV9TVWvashMgyWpgI/C2ts/nkixJsgT4LHA+sBq4uLUFuLr19RbgaWBzq28Gnm71a1q7SceY3ZctSZpP04ZJVX0dODLD/jYAN1fV81X1ODAGnNWWsap6rKp+DNwMbEgS4Gzg1rb/duDCTl/b2/qtwDmt/WRjSJJGpM81k8uS3N9Ogy1rtVOBJzpt9rfaZPUTgWeq6oUJ9Zf01bY/29pP1pckaUTmGibXAb8IrAEOAr8/bzOaR0m2JNmTZM/hw4dHPR1JesWaU5hU1ZNV9WJV/RT4I/7uNNMB4LRO0xWtNln9KeD4JEsn1F/SV9v+xtZ+sr6GzfP6qlpbVWuXL18+ly9VkjQDcwqTJKd0nr4fGL/Tawewsd2JdTqwCrgXuA9Y1e7cOobBBfQdVVXAXcBFbf9NwG2dvja19YuAO1v7ycaQJI3I0ukaJPkC8B7gpCT7gSuA9yRZAxTwXeAjAFW1L8ktwEPAC8ClVfVi6+cyYBewBNhWVfvaEB8Hbk7yGeCbwA2tfgPw+SRjDG4A2DjdGJKk0Zg2TKrq4iHlG4bUxttfCVw5pL4T2Dmk/hhD7saqqh8BH5jNGJKk0fAd8JKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9WaYSJJ6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLU27RhkmRbkkNJHuzUTkiyO8mj7XFZqyfJtUnGktyf5MzOPpta+0eTbOrU35nkgbbPtUky1zEkSaMxk1cmNwLrJ9S2AndU1SrgjvYc4HxgVVu2ANfBIBiAK4B3AWcBV4yHQ2vz4c5+6+cyhiRpdKYNk6r6OnBkQnkDsL2tbwcu7NRvqoG7geOTnAKcB+yuqiNV9TSwG1jftr2hqu6uqgJumtDXbMaQJI3IXK+ZnFxVB9v694GT2/qpwBOddvtbbar6/iH1uYwhSRqR3hfg2yuKmoe5zPsYSbYk2ZNkz+HDhxdgZpIkmHuYPDl+aqk9Hmr1A8BpnXYrWm2q+ooh9bmM8XOq6vqqWltVa5cvXz6rL1CSNHNzDZMdwPgdWZuA2zr1S9odV+uAZ9upql3AuUmWtQvv5wK72rbnkqxrd3FdMqGv2YwhSRqRpdM1SPIF4D3ASUn2M7gr6yrgliSbge8BH2zNdwIXAGPAD4EPAVTVkSSfBu5r7T5VVeMX9T/G4I6x44Db28Jsx5Akjc60YVJVF0+y6ZwhbQu4dJJ+tgHbhtT3AG8fUn9qtmNIkkbDd8BLknozTCRJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm+9wiTJd5M8kGRvkj2tdkKS3UkebY/LWj1Jrk0yluT+JGd2+tnU2j+aZFOn/s7W/1jbN1ONIUkajfl4ZfLeqlpTVWvb863AHVW1CrijPQc4H1jVli3AdTAIBuAK4F3AWcAVnXC4DvhwZ7/104whSRqBhTjNtQHY3ta3Axd26jfVwN3A8UlOAc4DdlfVkap6GtgNrG/b3lBVd1dVATdN6GvYGJKkEegbJgX83yTfSLKl1U6uqoNt/fvAyW39VOCJzr77W22q+v4h9anGeIkkW5LsSbLn8OHDs/7iJEkzs7Tn/v+qqg4k+YfA7iR/3d1YVZWkeo4xpanGqKrrgesB1q5du6DzkKRXs16vTKrqQHs8BHyZwTWPJ9spKtrjodb8AHBaZ/cVrTZVfcWQOlOMIUkagTmHSZK/n+QfjK8D5wIPAjuA8TuyNgG3tfUdwCXtrq51wLPtVNUu4Nwky9qF93OBXW3bc0nWtbu4LpnQ17AxJEkj0Oc018nAl9vdukuB/1lV/yfJfcAtSTYD3wM+2NrvBC4AxoAfAh8CqKojST4N3NfafaqqjrT1jwE3AscBt7cF4KpJxpAkjcCcw6SqHgPeMaT+FHDOkHoBl07S1zZg25D6HuDtMx1DkjQavgNektSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1ZphIknozTCRJvRkmkqTeDBNJUm+GiSSpN8NEktSbYSJJ6s0wkST1dlSHSZL1SR5JMpZk66jnI0mvVkdtmCRZAnwWOB9YDVycZPVoZyVJr05HbZgAZwFjVfVYVf0YuBnYMOI5SdKr0tEcJqcCT3Se7281SdIiWzrqCSykJFuALe3pD5I8MseuTgL+Zn5mNTu5esrNI5vXDLxc5+a8Oqb5+QKP12y9LOeVq3vN600zaXQ0h8kB4LTO8xWt9jNVdT1wfd+BkuypqrV9+5lvL9d5wct3bs5rdpzX7Lya53U0n+a6D1iV5PQkxwAbgR0jnpMkvSodta9MquqFJJcBu4AlwLaq2jfiaUnSq9JRGyYAVbUT2LkIQ/U+VbZAXq7zgpfv3JzX7Div2XnVzitVtdBjSJJe4Y7mayaSpJcJw6RJ8oEk+5L8NMmkdz1M9hEu7UaAe1r9i+2mgPmY1wlJdid5tD0uG9LmvUn2dpYfJbmwbbsxyeOdbWsWa16t3YudsXd06qM8XmuS/GX7ft+f5N92ts3r8ZruI3+SHNu+/rF2PFZ2tl3e6o8kOa/PPOYwr/+Y5KF2fO5I8qbOtqHf00Wa128kOdwZ/993tm1q3/dHk2xa5Hld05nTt5M809m2kMdrW5JDSR6cZHuSXNvmfX+SMzvb5vd4VZXL4FTfPwX+CfA1YO0kbZYA3wHeDBwDfAtY3bbdAmxs638I/OY8zeu/Alvb+lbg6mnanwAcAf5ee34jcNECHK8ZzQv4wST1kR0v4B8Dq9r6LwAHgePn+3hN9fPSafMx4A/b+kbgi219dWt/LHB662fJIs7rvZ2fod8cn9dU39NFmtdvAP9tyL4nAI+1x2VtfdlizWtC+//A4IagBT1ere9/DZwJPDjJ9guA24EA64B7Fup4+cqkqaqHq2q6NzUO/QiXJAHOBm5t7bYDF87T1Da0/mba70XA7VX1w3kafzKzndfPjPp4VdW3q+rRtv7/gEPA8nkav2smH/nTne+twDnt+GwAbq6q56vqcWCs9bco86qquzo/Q3czeB/XQuvzEUnnAbur6khVPQ3sBtaPaF4XA1+Yp7GnVFVfZ/DH42Q2ADfVwN3A8UlOYQGOl2EyO5N9hMuJwDNV9cKE+nw4uaoOtvXvAydP034jP/+DfGV7iXtNkmMXeV6vS7Inyd3jp954GR2vJGcx+GvzO53yfB2vmXzkz8/atOPxLIPjs5AfFzTbvjcz+Ot23LDv6WLO69+078+tScbfuPyyOF7tdODpwJ2d8kIdr5mYbO7zfryO6luDZyvJV4F/NGTTJ6rqtsWez7ip5tV9UlWVZNLb79pfHP+MwXtvxl3O4JfqMQxuD/w48KlFnNebqupAkjcDdyZ5gMEvzDmb5+P1eWBTVf20led8vF6JkvwasBb4pU75576nVfWd4T3Mu/8NfKGqnk/yEQav6s5epLFnYiNwa1W92KmN8ngtmldVmFTVL/fsYrKPcHmKwcvHpe2vy5/7aJe5zivJk0lOqaqD7ZffoSm6+iDw5ar6Safv8b/Sn0/yJ8B/Wsx5VdWB9vhYkq8BZwBfYsTHK8kbgK8w+EPi7k7fcz5eQ0z7kT+dNvuTLAXeyODnaSb7LuS8SPLLDAL6l6rq+fH6JN/T+fjlOJOPSHqq8/SPGVwjG9/3PRP2/do8zGlG8+rYCFzaLSzg8ZqJyeY+78fL01yzM/QjXGpwResuBtcrADYB8/VKZ0frbyb9/ty52vYLdfw6xYXA0Ls+FmJeSZaNnyZKchLwbuChUR+v9r37MoNzybdO2Dafx2smH/nTne9FwJ3t+OwANmZwt9fpwCrg3h5zmdW8kpwB/HfgfVV1qFMf+j1dxHmd0nn6PuDhtr4LOLfNbxlwLi99hb6g82pzeyuDi9l/2akt5PGaiR3AJe2urnXAs+0Ppvk/XvN9d8HRugDvZ3De8HngSWBXq/8CsLPT7gLg2wz+svhEp/5mBv/Yx4D/BRw7T/M6EbgDeBT4KnBCq68F/rjTbiWDvzZeM2H/O4EHGPxS/B/A6xdrXsC/bGN/qz1ufjkcL+DXgJ8AezvLmoU4XsN+XhicNntfW39d+/rH2vF4c2ffT7T9HgHOn+ef9+nm9dX272D8+OyY7nu6SPP6L8C+Nv5dwFs7+/67dhzHgA8t5rza808CV03Yb6GP1xcY3I34Ewa/vzYDHwU+2raHwX8i+J02/trOvvN6vHwHvCSpN09zSZJ6M0wkSb0ZJpKk3gwTSVJvhokkqTfDRJLUm2EiSerNMJEk9fb/AQOeCj7UW0rWAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "780d434773e6dc6ac6195e7592bf1408b2865a1c"
      },
      "cell_type": "code",
      "source": "#Compute the novelty-adjusted sentiment score, which can be used as a news factor\ndef adjusted_score(info):\n    avg = 4.600587\n    std = 0.7289763889\n    if info['sentimentClass'] == 1:\n        return 0\n    elif info['sentimentClass'] == 0 and info['decay_novelty'] > avg: #momentum \n        return info['sentimentNegative']\n    elif info['sentimentClass'] == 0 and info['decay_novelty'] < avg - std: #reversion\n        return -info['sentimentNegative']\n    elif info['sentimentClass'] == 2 and info['decay_novelty'] > avg: #momentum \n        return info['sentimentPositive']\n    elif info['sentimentClass'] == 2 and info['decay_novelty'] < avg - std: #reversion\n        return -info['sentimentPositive']\n    else: \n        return 0\n    \nadjusted_score = df.apply(adjusted_score, axis = 1)\nprint ('Done')    ",
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Done\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b0d4feb8cf8024e59355005da30a2882ccc6a121"
      },
      "cell_type": "code",
      "source": "describe(adjusted_score)",
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 73,
          "data": {
            "text/plain": "DescribeResult(nobs=9328750, minmax=(-0.856939971446991, 0.856939971446991), mean=0.02010050459192407, variance=0.2619629977419411, skewness=-0.07086393921575833, kurtosis=-0.955395225256523)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "0f48739682dc5668abaf48b1bf00d696937e42a7"
      },
      "cell_type": "code",
      "source": "#check if every news data point has industry code\nnum_missing_ind = 0\nidx = []\nfor i, val in enumerate(topic_codes):\n    if i % 500000 == 0:\n        print (\"step \" + str(i//500000) + \" is done!\")\n    topics = eval(val)\n    flag = False\n    for item in topics:\n        if item in industry_codes:\n            flag = True\n        if flag == True:\n            continue\n    if flag == False:\n        num_missing_ind += 1\n        idx.append(i)\nprint ('Done')\n        ",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c3fd9956492e7022a9a33f8de2d712f58cd34c60"
      },
      "cell_type": "code",
      "source": "len(idx)/len(topic_codes)\nnews_train_df.iloc[0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "840aa03b49d675953f080e4069f79f435282bb43"
      },
      "cell_type": "markdown",
      "source": "i## `get_prediction_days` function\n\nGenerator which loops through each \"prediction day\" (trading day) and provides all market and news observations which occurred since the last data you've received.  Once you call **`predict`** to make your future predictions, you can continue on to the next prediction day.\n\nYields:\n* While there are more prediction day(s) and `predict` was called successfully since the last yield, yields a tuple of:\n    * `market_observations_df`: DataFrame with market observations for the next prediction day.\n    * `news_observations_df`: DataFrame with news observations for the next prediction day.\n    * `predictions_template_df`: DataFrame with `assetCode` and `confidenceValue` columns, prefilled with `confidenceValue = 0`, to be filled in and passed back to the `predict` function.\n* If `predict` has not been called since the last yield, yields `None`."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "724c38149860c8e9058474ac9045c2301e8a20da"
      },
      "cell_type": "code",
      "source": "# You can only iterate through a result from `get_prediction_days()` once\n# so be careful not to lose it once you start iterating.\ndays = env.get_prediction_days()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "8d5781f889893e3e34b054687fd538c1a76bfdcc"
      },
      "cell_type": "code",
      "source": "(market_obs_df, news_obs_df, predictions_template_df) = next(days)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "fe1cb6accc3536258e7687c64ab3e5e5caa6334a"
      },
      "cell_type": "code",
      "source": "market_obs_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "11c5b072aefdd54e6fbe9ae71dc3ea41909ade19"
      },
      "cell_type": "code",
      "source": "news_obs_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f99f6364881767a4071da9ea19e18163a7ce066b"
      },
      "cell_type": "code",
      "source": "predictions_template_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "11e95f2e3d493ee6e1023c7a4191310adde5d2bf"
      },
      "cell_type": "markdown",
      "source": "Note that we'll get an error if we try to continue on to the next prediction day without making our predictions for the current day."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b8ac953a3afbfca2fb200bbf8f0f7339bec7e6f1"
      },
      "cell_type": "code",
      "source": "next(days)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ba72731adf652d6011652e906d8b340d6572904e"
      },
      "cell_type": "markdown",
      "source": "### **`predict`** function\nStores your predictions for the current prediction day.  Expects the same format as you saw in `predictions_template_df` returned from `get_prediction_days`.\n\nArgs:\n* `predictions_df`: DataFrame which must have the following columns:\n    * `assetCode`: The market asset.\n    * `confidenceValue`: Your confidence whether the asset will increase or decrease in 10 trading days.  All values must be in the range `[-1.0, 1.0]`.\n\nThe `predictions_df` you send **must** contain the exact set of rows which were given to you in the `predictions_template_df` returned from `get_prediction_days`.  The `predict` function does not validate this, but if you are missing any `assetCode`s or add any extraneous `assetCode`s, then your submission will fail."
    },
    {
      "metadata": {
        "_uuid": "9cd8317a5e52180b592ee2abc1d2177214642a3c"
      },
      "cell_type": "markdown",
      "source": "Let's make random predictions for the first day:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a3f2197ed790f1aff1356a6954575fde976a4935"
      },
      "cell_type": "code",
      "source": "import numpy as np\ndef make_random_predictions(predictions_df):\n    predictions_df.confidenceValue = 2.0 * np.random.rand(len(predictions_df)) - 1.0",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ca72b7003f24f4aa0c4afe25b600aae31abd64d5"
      },
      "cell_type": "code",
      "source": "make_random_predictions(predictions_template_df)\nenv.predict(predictions_template_df)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ff62c167b459c5895383fb05fd9260c14be8c1b8"
      },
      "cell_type": "markdown",
      "source": "Now we can continue on to the next prediction day and make another round of random predictions for it:"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4e2293d44aad86d09d25326c4ede6f566ab69721"
      },
      "cell_type": "code",
      "source": "(market_obs_df, news_obs_df, predictions_template_df) = next(days)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "140aee54dc838549f87f041a97c7a809ee4e0f6f"
      },
      "cell_type": "code",
      "source": "market_obs_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b02d43dec4b881564cd43ff6239c7aa97d94a7af"
      },
      "cell_type": "code",
      "source": "news_obs_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "5233f1b22f5ddac08adb50bbaa6444a0da4a24bc"
      },
      "cell_type": "code",
      "source": "predictions_template_df.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a19142739096135d237e2837d8e10c992e53a6e5"
      },
      "cell_type": "code",
      "source": "make_random_predictions(predictions_template_df)\nenv.predict(predictions_template_df)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8056b881707072c379ad2e89b9c59c3c041a2ab7"
      },
      "cell_type": "markdown",
      "source": "## Main Loop\nLet's loop through all the days and make our random predictions.  The `days` generator (returned from `get_prediction_days`) will simply stop returning values once you've reached the end."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ef60bc52a8a228e5a2ce18e4bd416f1f1f25aeae"
      },
      "cell_type": "code",
      "source": "for (market_obs_df, news_obs_df, predictions_template_df) in days:\n    make_random_predictions(predictions_template_df)\n    env.predict(predictions_template_df)\nprint('Done!')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7c8fbcca87c7f6abc53e86408417bf12ce21bb7f"
      },
      "cell_type": "markdown",
      "source": "## **`write_submission_file`** function\n\nWrites your predictions to a CSV file (`submission.csv`) in the current working directory."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "2c8ed34ffb2c47c6e124530ec798c0b4eb01ddd5"
      },
      "cell_type": "code",
      "source": "env.write_submission_file()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d38aa8a67cad3f0c105db7e764ec9b805db39ceb"
      },
      "cell_type": "code",
      "source": "# We've got a submission file!\nimport os\nprint([filename for filename in os.listdir('.') if '.csv' in filename])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "13f2ecc6357eec9ffbcf7f596c332e65b9db7a02"
      },
      "cell_type": "code",
      "source": "# #Merge data set\n# import pandas as pd\n# import numpy as np\n# from datetime import datetime\n# import sys\n# from functools import reduce\n\n\n# # when debug mode is on, we only take a sub-sample of total data\n# debug_mode = True\n# # when we first load this in notebook, turn reload on. afterwards, turn it off no need to reload data everytime\n# reload = False\n\n# # if reload:\n# #     news_train_dir = \"./new_train_df.csv\"\n# #     news_train_df = pd.read_csv(news_train_dir)\n\n# #     market_train_dir = \"./market_train_df.csv\"\n# #     market_train_df = pd.read_csv(market_train_dir)\n\n# # globals\n# news_col_extractor = [\"time\", \"assetCodes\", \"headline\", \"urgency\", \"takeSequence\", \n#                     \"subjects\", \"audiences\", \"relevance\", \n#                     'sentimentClass','sentimentNegative', 'sentimentNeutral', 'sentimentPositive',\n#                     'noveltyCount12H', 'noveltyCount24H', 'noveltyCount3D', 'noveltyCount5D', 'noveltyCount7D', \n#                     'volumeCounts12H','volumeCounts24H', 'volumeCounts3D', 'volumeCounts5D','volumeCounts7D'\n#                    ]\n\n# market_col_extractor = [\"time\", \"assetCode\", \"volume\", \"close\", \"open\", \n#                         \"returnsClosePrevRaw1\", \"returnsOpenPrevRaw1\", \"returnsClosePrevRaw10\", \"returnsOpenPrevRaw10\",\n#                         \"returnsOpenNextMktres10\", \"universe\"]\n\n# identity = lambda series: reduce(lambda x, y: x, series)\n# coalesce = lambda x: list(x)\n\n# if debug_mode:\n#     news = news_train_df[60000:70000]\n#     market = market_train_df[:5000]\n# else:\n#     news = news_train_df\n#     market = market_train_df\n\n# assetCode_set = set(market_train_df[\"assetCode\"].unique())\n\n# # extract relevant columns based on their descriptions\n# def extract_df(news_train_df, market_train_df):\n#     news_df = news_train_df[news_col_extractor]\n#     market_df = market_train_df[market_col_extractor]\n#     return news_df, market_df\n\n# # given a dataframe with field time convert into datetime, month and week\n# def extract_time_dependent_features(df, obj = None):\n#     # 1. get date\n#     df[\"datetime\"] = df[\"time\"].apply(lambda ts: ts[:10])\n#     # 2. get month\n#     if obj == \"news\":\n#         return df\n#     df[\"month\"] = df[\"datetime\"].apply(lambda ts: ts[5:7])\n#     # 3. get week\n#     df[\"week\"] = df[\"datetime\"].apply(lambda ts: datetime.strptime(ts, '%Y-%m-%d').strftime('%a'))\n#     return df\n\n# # apply helper aggregator to reduce assetCode in preparation for joining\n# def assetCodeMapper(assetCodeSet):\n#     assets = list(eval(assetCodeSet).intersection(assetCode_set))\n#     if assets == []:\n#         return None\n#     else:\n#         return assets[0]\n    \n# # join if assetCode in assetCodes and time days are the same\n# def mergeDframes(news_df, market_df):\n#     anchor = [\"datetime\", \"assetCode\"]\n#     mergedDF = market_df.merge(news_df, on=[\"datetime\", \"assetCode\"], how=\"left\").dropna()\n#     return mergedDF\n\n# # squash columns so that (datetime, assetCodes) are unique\n# def squash(res):\n#     df = res.groupby(\"datetime\")\n#     df = res.groupby([\"datetime\", \"assetCode\"]).agg({'volume': identity,\n#                                                     'open': identity,\n#                                                     'close': identity,\n#                                                     'returnsClosePrevRaw1': identity, \n#                                                     'returnsOpenPrevRaw1': identity, \n#                                                     'returnsClosePrevRaw10': identity,\n#                                                     'universe': identity,\n#                                                     'month': identity,\n#                                                     'week': identity,\n#                                                     'headline': coalesce,\n#                                                     'urgency': coalesce,\n#                                                     'takeSequence': coalesce,\n#                                                     'subjects': coalesce,\n#                                                     'audiences': coalesce,\n#                                                     'relevance': coalesce,\n#                                                     'sentimentClass': coalesce, \n#                                                     'sentimentNegative': coalesce, \n#                                                     'sentimentNeutral': coalesce,\n#                                                     'sentimentPositive': coalesce, \n#                                                     'noveltyCount12H': coalesce, \n#                                                     'noveltyCount24H': coalesce,\n#                                                     'noveltyCount3D': coalesce, \n#                                                     'noveltyCount5D': coalesce, \n#                                                     'noveltyCount7D': coalesce, \n#                                                     'volumeCounts12H': coalesce,\n#                                                     'volumeCounts24H': coalesce, \n#                                                     'volumeCounts3D': coalesce, \n#                                                     'volumeCounts5D': coalesce, \n#                                                     'volumeCounts7D': coalesce,\n#                                                     'returnsOpenNextMktres10': identity\n#                                                    })\n#     return df\n\n# # helper functiuons for urgency related partition calculation\n# def urgency_helper(x, column, urgency_type):\n#     relevance = [0 if i == urgency_type else i for i in x.relevance]\n#     return np.multiply(relevance, column).sum() / sum(relevance)\n\n# def urgency_dist_helper(x):\n#     d = Counter(x)\n#     if 1 not in d:\n#         d[1] = 0\n#     if 3 not in d:\n#         d[3] = 0\n#     return d[1], d[3]\n\n# def urgency_time_helper(x, column, urgency_type):\n#     # as indicator function\n#     relevance = [0 if i == urgency_type else 1 for i in x.relevance]\n#     return np.multiply(relevance, column).sum()\n\n# # generate relevance weighted features\n# def generate_relevance_weighted_sentiment(squashedDf):\n#     # we are removing urgency = 2 type because there are too few of them for learning\n#     squashedDf = squashedDf[squashedDf[\"urgency\"] != 2]\n    \n#     # for article and alert, let's compute different values\n#     urgency_ls = [1, 3]\n#     urgency_name = [\"alert\", \"article\"]\n#     time_ls = [\"12H\", \"24H\", \"3D\", \"5D\", \"7D\"]\n#     for i in range(len(urgency_ls)):\n#         name = urgency_name[i]\n#         u = urgency_ls[i]\n#         squashedDf[name+\"_relevance_weighted_sentiment\"] = squashedDf.apply(lambda x: urgency_helper(x, x.sentimentClass, u), axis=1)\n#         squashedDf[name+\"_relevance_weighted_negative_sentiment\"] = squashedDf.apply(lambda x: urgency_helper(x, x.sentimentNegative, u), axis=1)\n#         squashedDf[name+\"_relevance_weighted_positive_sentiment\"] = squashedDf.apply(lambda x: urgency_helper(x, x.sentimentPositive, u), axis=1)\n#         squashedDf[name+\"_relevance_weighted_neutral_sentiment\"] = squashedDf.apply(lambda x: urgency_helper(x, x.sentimentNeutral, u), axis=1)\n#         for time in time_ls:\n#             squashedDf[name+\"_news_volume_sum_\"+time] = squashedDf.apply(lambda x: urgency_time_helper(x, x[\"volumeCounts\"+time], u), axis=1)\n#             squashedDf[name+\"_news_novelty_sum_\"+time] = squashedDf.apply(lambda x: urgency_time_helper(x, x[\"noveltyCount\"+time], u), axis=1)\n#     squashedDf[\"relevance_weighted_sentiment\"] = squashedDf.apply(lambda x: np.multiply(x.relevance, x.sentimentClass).sum() / sum(x.relevance), axis=1)\n#     squashedDf[\"relevance_weighted_negative_sentiment\"] = squashedDf.apply(lambda x: np.multiply(x.relevance, x.sentimentNegative).sum() / sum(x.relevance), axis=1)\n#     squashedDf[\"relevance_weighted_positive_sentiment\"] = squashedDf.apply(lambda x: np.multiply(x.relevance, x.sentimentPositive).sum() / sum(x.relevance), axis=1)\n#     squashedDf[\"relevance_weighted_neutral_sentiment\"] = squashedDf.apply(lambda x: np.multiply(x.relevance, x.sentimentNeutral).sum(), axis=1)\n#     for time in time_ls:\n#         squashedDf[\"news_volume_sum_\"+time] = squashedDf.apply(lambda x: sum(x[\"volumeCounts\"+time]), axis=1)\n#         squashedDf[\"news_novelty_sum_\"+time] = squashedDf.apply(lambda x: sum(x[\"noveltyCount\"+time]), axis=1)\n#     squashedDf[\"alert\"] = squashedDf.urgency.apply(lambda x: urgency_dist_helper(x)[0])\n#     squashedDf[\"article\"] = squashedDf.urgency.apply(lambda x: urgency_dist_helper(x)[1])\n#     return squashedDf\n\n# def extract_features(df):\n#     extract_ls = ['month','week', 'alert', 'article',\n              \n#                 'alert_relevance_weighted_sentiment',\n#                 'alert_relevance_weighted_negative_sentiment',\n#                 'alert_relevance_weighted_positive_sentiment',\n#                 'alert_relevance_weighted_neutral_sentiment',\n#                 'alert_news_volume_sum_12H', 'alert_news_novelty_sum_12H',\n#                 'alert_news_volume_sum_24H', 'alert_news_novelty_sum_24H',\n#                 'alert_news_volume_sum_3D', 'alert_news_novelty_sum_3D',\n#                 'alert_news_volume_sum_5D', 'alert_news_novelty_sum_5D',\n#                 'alert_news_volume_sum_7D', 'alert_news_novelty_sum_7D',\n              \n#                 'article_relevance_weighted_sentiment',\n#                 'article_relevance_weighted_negative_sentiment',\n#                 'article_relevance_weighted_positive_sentiment',\n#                 'article_relevance_weighted_neutral_sentiment',\n#                 'article_news_volume_sum_12H', 'article_news_novelty_sum_12H',\n#                 'article_news_volume_sum_24H', 'article_news_novelty_sum_24H',\n#                 'article_news_volume_sum_3D', 'article_news_novelty_sum_3D',\n#                 'article_news_volume_sum_5D', 'article_news_novelty_sum_5D',\n#                 'article_news_volume_sum_7D', 'article_news_novelty_sum_7D',\n              \n#                 'relevance_weighted_sentiment', 'relevance_weighted_negative_sentiment',\n#                 'relevance_weighted_positive_sentiment',\n#                 'relevance_weighted_neutral_sentiment', 'news_volume_sum_12H',\n#                 'news_novelty_sum_12H', 'news_volume_sum_24H', 'news_novelty_sum_24H',\n#                 'news_volume_sum_3D', 'news_novelty_sum_3D', 'news_volume_sum_5D',\n#                 'news_novelty_sum_5D', 'news_volume_sum_7D', 'news_novelty_sum_7D',\n              \n#               'returnsOpenNextMktres10']\n#     df = df[df.universe == 1.0][extract_ls].dropna()\n#     return df\n    \n# # orchestration\n\n# news_df, market_df = extract_df(news_train_df, market_train_df)\n# # market_df['time'] = market_df['time'].apply(lambda ts: ts.strftime(format='%Y-%m-%d'))\n# news_df['time'] = news_df['time'].apply(lambda ts: ts.strftime(format='%Y-%m-%d'))\n# market_df = extract_time_dependent_features(market_df)\n# news_df = extract_time_dependent_features(news_df, \"news\")\n# news_df[\"assetCode\"] = news_df[\"assetCodes\"].apply(assetCodeMapper)\n# mergedDF = mergeDframes(news_df, market_df)\n# squashedDf = squash(mergedDF)\n# featureDf = generate_relevance_weighted_sentiment(squashedDf)\n# df = extract_features(featureDf)\n# print ('Done')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f464f37885ffa763a2592e2867d74685f75be506"
      },
      "cell_type": "markdown",
      "source": "As indicated by the helper message, calling `write_submission_file` on its own does **not** make a submission to the competition.  It merely tells the module to write the `submission.csv` file as part of the Kernel's output.  To make a submission to the competition, you'll have to **Commit** your Kernel and find the generated `submission.csv` file in that Kernel Version's Output tab (note this is _outside_ of the Kernel Editor), then click \"Submit to Competition\".  When we re-run your Kernel during Stage Two, we will run the Kernel Version (generated when you hit \"Commit\") linked to your chosen Submission."
    },
    {
      "metadata": {
        "_uuid": "2e3a267ea3149403c49ff59515a1a669ca2d1f9f"
      },
      "cell_type": "markdown",
      "source": "## Restart the Kernel to run your code again\nIn order to combat cheating, you are only allowed to call `make_env` or iterate through `get_prediction_days` once per Kernel run.  However, while you're iterating on your model it's reasonable to try something out, change the model a bit, and try it again.  Unfortunately, if you try to simply re-run the code, or even refresh the browser page, you'll still be running on the same Kernel execution session you had been running before, and the `twosigmanews` module will still throw errors.  To get around this, you need to explicitly restart your Kernel execution session, which you can do by pressing the Restart button in the Kernel Editor's bottom Console tab:\n![Restart button](https://i.imgur.com/hudu8jF.png)"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}