0_Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BEFORE WE START**: Make sure that any open notebooks that are no longer needed are shut down to free up compute resources (note that closing the tab is not enough!). This will reduce the risk of running into memory and/or performance issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity recognition on the Capture-24 dataset\n",
    "\n",
    "<img src=\"wrist_accelerometer.jpg\" width=\"300\"/>\n",
    "\n",
    "The Capture-24 dataset contains wrist-worn accelerometer data\n",
    "collected from 151 participants. To obtain ground truth annotations, the\n",
    "participants also wore a body camera during daytime, and used sleep diaries to\n",
    "register their sleep times. Each participant was recorded for roughly 24 hours.\n",
    "The accelerometer was an Axivity AX3 wrist watch (image above) that mearures\n",
    "acceleration in all three axes (x, y, z) at a sampling rate of 100Hz.\n",
    "The body camera was a Vicon Autographer with a sampling rate of 1 picture every 20 seconds.\n",
    "Note that the camera images are not part of the data release &mdash; only the\n",
    "raw acceleration trace with text annotations are provided.\n",
    "\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from glob import glob\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from imblearn.ensemble import BalancedRandomForestClassifier\n",
    "from sklearn import decomposition\n",
    "from sklearn import preprocessing\n",
    "from sklearn import manifold\n",
    "from sklearn import metrics\n",
    "from tqdm.auto import tqdm\n",
    "from joblib import Parallel, delayed\n",
    "import urllib.request\n",
    "import shutil\n",
    "import zipfile\n",
    "\n",
    "import utils\n",
    "\n",
    "pd.options.display.max_rows = 999\n",
    "pd.options.display.max_colwidth = None\n",
    "\n",
    "# For reproducibility\n",
    "np.random.seed(42)\n",
    "N_JOBS = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download dataset\n",
    "Download the required dataset from https://ora.ox.ac.uk/objects/uuid:99d7c092-d865-4a19-b096-cc16440cd001 then unzip and copy the folder into the current directory.\n",
    "\n",
    "Alternatively, run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: The required dataset is already available at /srv/capture24.\n",
    "# The following creates a symbolic link (shortcut in Windows) to it in the current directory.\n",
    "if not os.path.exists(\"capture24\"):\n",
    "    os.symlink(\"/srv/capture24\", \"capture24\")\n",
    "\n",
    "# If you really want to download the dataset, uncomment and run the following code (note that this will take a while):\n",
    "# print(f\"Downloading Capture-24...\")\n",
    "# url = \"https://ora.ox.ac.uk/objects/uuid:99d7c092-d865-4a19-b096-cc16440cd001/download_file?file_format=&safe_filename=capture24.zip&type_of_work=Dataset\"\n",
    "# with urllib.request.urlopen(url) as f_src, open(\"capture24.zip\", \"wb\") as f_dst:\n",
    "#     shutil.copyfileobj(f_src, f_dst)\n",
    "# print(\"Unzipping...\")\n",
    "# with zipfile.ZipFile(\"capture24.zip\", \"r\") as f:\n",
    "#     f.extractall(\".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this data is downwloaded, we can print out its content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Content of capture24/')\n",
    "print(os.listdir(\"capture24/\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1**: How many participants are in this dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "First, let's load the accelerometry data for a single particpant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = utils.load_data('capture24/P001.csv.gz')\n",
    "print('\\nParticipant P001:')\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2**: What do columns `x`, `y`, `z` mean? Can you guess the units? Hint: sqrt(x^2 + y^2 + z^2) = ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3**: How many hours of useable data did participant `P001` provide?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the list of different activities performed by P001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nList of unique activities by participant P001\")\n",
    "print(data['annotation'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4**: How many unique activities are in `P001`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The annotations are based on the [Compendium of Physical\n",
    "Activity](https://sites.google.com/site/compendiumofphysicalactivities/home) (CPA).\n",
    "The leading numbers in the annotations are CPA codes used to uniquely identify different human\n",
    "activities, e.g. \"7030 sleeping\", \"11580 office work such as writing and\n",
    "typing\". The trailing numbers indicate metabolic equivalent of task (MET).\n",
    "These numbers will not be important for now.\n",
    "\n",
    "In the Capture-24 dataset, more than 200 distinct annotations were collected\n",
    "across all participants.\n",
    "\n",
    "To develop a model for activity recognition, let's segment the data into windows of\n",
    "30 seconds. \n",
    "\n",
    "The activity recognition model will then be trained to classify the\n",
    "individual windows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, Y, T = utils.make_windows(data, winsec=30)\n",
    "print(\"X shape:\", X.shape)\n",
    "print(\"Y shape:\", Y.shape)\n",
    "print(\"T shape:\", T.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5**: Why is the shape of array `X` (N, 3000, 3)? What does each number mean?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned, there can be hundreds of distinct annotations, many of which are\n",
    "physically very similar (e.g. \"sitting, child care\", \"sitting, pet care\").\n",
    "For our purposes, it is enough to map these annotations onto a simpler\n",
    "set of labels. The provided file *annotation-label-dictionary.csv*\n",
    "contains different annotation-to-label mappings that can be used.\n",
    "We will summarise the different annotations into 6 activity classes:\n",
    "\"sleep\", \"sit-stand\", \"vehicle\", \"walking\", \"bicycling\", and \"mixed\".\n",
    "\n",
    "**Exercise 6**: Open the *annotation-label-dictionary.csv* file in Excel or any\n",
    "other spreadsheet app. What other mappings (e.g. \"label:Walmsley2020\") are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "anno_label_dict = pd.read_csv(\n",
    "    \"capture24/annotation-label-dictionary.csv\", \n",
    "    index_col='annotation', \n",
    "    dtype='string'\n",
    ")\n",
    "\n",
    "# Map annotations using Willetts' labels  (see paper reference at the bottom)\n",
    "Y = anno_label_dict.loc[Y, 'label:Willetts2018'].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's count the number of 30-sec windows for each activity class.\n",
    "For this participant, we only observe 5 classes (no \"bicycling\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('\\nLabel distribution (# windows)')\n",
    "print(pd.Series(Y).value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 7**: How many hours of `sit-stand` do we have for `P001`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Visualization\n",
    "Visualization helps us get some insight and anticipate the difficulties that\n",
    "may arise during the modelling.\n",
    "Let's visualize some examples for each activity label.\n",
    "\n",
    "\n",
    "Run the below script multiple times to draw new samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NPLOTS = 5\n",
    "unqY = np.unique(Y)\n",
    "fig, axs = plt.subplots(len(unqY), NPLOTS, sharex=True, sharey=True, figsize=(8,6))\n",
    "for y, row in zip(unqY, axs):\n",
    "    idxs = np.random.choice(np.where(Y==y)[0], size=NPLOTS)\n",
    "    row[0].set_ylabel(y)\n",
    "    for x, ax in zip(X[idxs], row):\n",
    "        ax.plot(x)\n",
    "        ax.set_ylim(-5,5)\n",
    "fig.tight_layout()\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plots, we can already tell it should be easier to classify \"sleep\"\n",
    "and maybe \"sit-stand\", with the signal variance being a good discriminative\n",
    "feature for this.\n",
    "Next, let's try to visualize the data in a scatter-plot.\n",
    "The most standard approach to visualize high-dimensional points is to\n",
    "scatter-plot the first two principal components of the data.\n",
    "\n",
    "## PCA visualization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def scatter_plot(X, Y):\n",
    "    unqY = np.unique(Y)\n",
    "    fig, ax = plt.subplots()\n",
    "    for y in unqY:\n",
    "        X_y = X[Y==y]\n",
    "        ax.scatter(X_y[:,0], X_y[:,1], label=y, alpha=.5, s=10)\n",
    "    fig.legend()\n",
    "    fig.show()\n",
    "\n",
    "print(\"Plotting first two PCA components...\")\n",
    "scaler = preprocessing.StandardScaler()  # PCA requires normalized data\n",
    "X_scaled = scaler.fit_transform(X.reshape(X.shape[0],-1))\n",
    "pca = decomposition.PCA(n_components=2)  # two components\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "scatter_plot(X_pca, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## t-SNE visualization\n",
    "PCA's main limitation is in dealing with data that is not linearly separable.\n",
    "Another popular high-dimensional data visualization tool is _t-distributed\n",
    "stochastic neighbor embedding_ (t-SNE).  Let's first use it on top of PCA to\n",
    "visualize 50 principal components.\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Plotting t-SNE on 50 PCA components...\")\n",
    "pca = decomposition.PCA(n_components=50)\n",
    "X_pca = pca.fit_transform(X_scaled)\n",
    "tsne = manifold.TSNE(n_components=2,  # project down to 2 components\n",
    "    init='random', random_state=42, perplexity=100, learning_rate='auto')\n",
    "X_tsne_pca = tsne.fit_transform(X_pca)\n",
    "scatter_plot(X_tsne_pca, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature extraction\n",
    "Let's extract a few signal features for each window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_features(xyz):\n",
    "    ''' Extract features. xyz is an array of shape (N,3) '''\n",
    "\n",
    "    feats = {}\n",
    "    feats['xMean'], feats['yMean'], feats['zMean'] = np.mean(xyz, axis=0)\n",
    "    feats['xStd'], feats['yStd'], feats['zStd'] = np.std(xyz, axis=0)\n",
    "    v = np.linalg.norm(xyz, axis=1)  # magnitude stream\n",
    "    feats['mean'], feats['std'] = np.mean(v), np.std(v)\n",
    "\n",
    "    return feats\n",
    "\n",
    "X_feats = pd.DataFrame([extract_features(x) for x in tqdm(X)])\n",
    "print(X_feats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 8**: Edit the function above to implement your own features.\n",
    "What other features do you think would be useful to extract to distinguish activities?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the data again using t-SNE, but this time using the extracted\n",
    "features rather than the principal components.\n",
    "\n",
    "*Note: this may take a while*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Plotting t-SNE on extracted features...\")\n",
    "tsne = manifold.TSNE(n_components=2,\n",
    "    init='random', random_state=42, perplexity=100, learning_rate='auto')\n",
    "X_tsne_feats = tsne.fit_transform(X_feats)\n",
    "scatter_plot(X_tsne_feats, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity classification\n",
    "Now the fun part. Let's train a balanced random forest on the extracted features to\n",
    "perform activity classification. We use the implementation from\n",
    "[`imbalanced-learn`](https://imbalanced-learn.org/stable/) package, which has\n",
    "better support for imbalanced datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = BalancedRandomForestClassifier(\n",
    "    n_estimators=100,\n",
    "    replacement=True,\n",
    "    sampling_strategy='not minority',\n",
    "    n_jobs=N_JOBS,\n",
    "    random_state=42,\n",
    ")\n",
    "clf.fit(X_feats, Y)\n",
    "Y_pred = clf.predict(X_feats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9**: Tweak the parameters of the classifier. See the [API reference](https://imbalanced-learn.org/stable/references/generated/imblearn.ensemble.BalancedRandomForestClassifier.html#imblearn.ensemble.BalancedRandomForestClassifier) for parameter descriptions. Here are some ideas:\n",
    "\n",
    "- Increase the number of `n_estimators`\n",
    "- Set `max_features` (default is `sqrt` i.e. square root of the total number of features)\n",
    "- Set `max_depth` (default is `None`, i.e. as deep as needed)\n",
    "\n",
    "**Exercise 10**: What is the parameter in random forest that will *always* improve performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot predicted vs. true activity profiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('\\nClassifier performance in training set')\n",
    "print(metrics.classification_report(Y, Y_pred, zero_division=0))\n",
    "\n",
    "fig, axs = utils.plot_compare(T, Y, Y_pred, trace=X_feats['std'])\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classification performance is very good, but this is in-sample! \n",
    "\n",
    "**Exercise 11**: What is the difference between macro and weighted avg?\n",
    "\n",
    "Let's load another participant and test again to see the true (out-of-sample) performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load another participant\n",
    "data2 = utils.load_data(\"capture24/P002.csv.gz\")\n",
    "X2, Y2, T2 = utils.make_windows(data2, winsec=30)\n",
    "Y2 = anno_label_dict.loc[Y2, 'label:Willetts2018'].to_numpy()\n",
    "X2_feats = pd.DataFrame([extract_features(x) for x in X2])\n",
    "Y2_pred = clf.predict(X2_feats)\n",
    "\n",
    "print('\\nClassifier performance on held-out subject')\n",
    "print(metrics.classification_report(Y2, Y2_pred, zero_division=0))\n",
    "\n",
    "fig, axs = utils.plot_compare(T2, Y2, Y2_pred, trace=X2_feats['std'])\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the classification performance is much worse out of sample, with\n",
    "the macro-averaged F1-score dropping from .90 to .37.\n",
    "On the other hand, the scores for the easy classes \"sleep\" and \"sit-stand\" remained good.\n",
    "Finally, note that participant P001 didn't have the \"bicycling\" class while\n",
    "participant P002 didn't have the \"vehicle\" class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "**Feature extraction**\n",
    "\n",
    "- [On the role of features in human activity recognition](https://dl.acm.org/doi/10.1145/3341163.3347727)\n",
    "- [A Comprehensive Study of Activity Recognition Using Accelerometers](https://www.mdpi.com/2227-9709/5/2/27)\n",
    "\n",
    "**Papers using the Capture-24 dataset**\n",
    "\n",
    "- [Reallocating time from machine-learned sleep, sedentary behaviour or\n",
    "light physical activity to moderate-to-vigorous physical activity is\n",
    "associated with lower cardiovascular disease\n",
    "risk](https://www.medrxiv.org/content/10.1101/2020.11.10.20227769v2.full?versioned=true)\n",
    "(Walmsley2020 labels)\n",
    "- [GWAS identifies 14 loci for device-measured\n",
    "physical activity and sleep\n",
    "duration](https://www.nature.com/articles/s41467-018-07743-4)\n",
    "(Doherty2018 labels)\n",
    "- [Statistical machine learning of sleep and physical activity phenotypes\n",
    "from sensor data in 96,220 UK Biobank\n",
    "participants](https://www.nature.com/articles/s41598-018-26174-1)\n",
    "(Willetts2018 labels)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}