Intro_to_software_engineering_for_scientists.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vipof2Q5x0Wz"
   },
   "source": [
    "# An introduction to software design (for scientists)\n",
    "\n",
    "Scientists write terrible code. Arguably, most people write terrible code, but our job is to explore and try new thing, so it makes sense that this involves a lot of nasty code. Think \"prototype\" algorithms, messy data munging and loooots of absolute file paths - **hero code** written during late-night, caffeine fuelled binges, and requiring a superhero (you) to safely use, repurpose, or even run on a different computer. \n",
    "\n",
    "This is a problem. While hero code is part of the creative process - it's inevitable when we want to test a new idea or see if something works - writing ONLY hero code wastes time and causes excessive hair loss by:\n",
    "\n",
    "1. Being generally hard to debug (as complexity increases)\n",
    "2. Requiring extensive rewriting every time it is used (in different situations)\n",
    "3. Including it's author as a dependency - if nobody can use your code without your help then either nobody will use it or you'll be bombarded by pleas for help\n",
    "\n",
    "## Aims\n",
    "\n",
    "This tutorial aims to introduce three things:\n",
    "\n",
    "1. some of the principles of software design and why they matter, with a worked example.\n",
    "\n",
    "2. basic GIT usage.\n",
    "\n",
    "3. the python package system.\n",
    "\n",
    "We assume basic knowledge of python and numpy, although this is not entirely needed to grasp the main ideas being presented.\n",
    "\n",
    "## A short note on GIT\n",
    "\n",
    "This does not aim to be a comprehensive introduction to GIT - there are plenty of these [available already](https://medium.com/swlh/an-introduction-to-git-and-github-22ecb4cb1256) - but we do intentionally use git in a basic way so that you can start becoming familiar with it. \n",
    "\n",
    "In short, git is essentially a version control system that tracks how code changes over time, and allows changes to be removed (*reverted*), or multiple versions (*forks*) to be created (e.g. for editing by different people) and subsequently merged (*pulled*).\n",
    "\n",
    "Combined with services like [GitHub](https://github.com/), this provides access to millions of open-source code projects and facilitates collaboration between developers.\n",
    "\n",
    "----\n",
    "<center>\n",
    "<img src=\"https://hikaruzone.files.wordpress.com/2015/10/in-case-of-fire-1-git-commit-2-git-push-3-leave-building2.png\" width=\"500px\">\n",
    "</center>\n",
    "\n",
    "---\n",
    "\n",
    "We won't go into the details of using git (please refer to the internets for that), but we'll use it to copy code and (later) look at some of the funky things it can do in the software engineering context.\n",
    "\n",
    "To start todays tutorial, we will *clone* (read copy) the tutorial files from GitHub. Start by making your own *fork* of the base repository by clicking the fork button on this page: https://gitlab.hzdr.de/thiele67/intro_to_software_design.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Y9-o2pGzxvMS",
    "outputId": "473e0aac-71ea-4195-d08a-cf5978d24a73"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'intro_to_software_design'...\n",
      "remote: Enumerating objects: 11, done.\u001b[K\n",
      "remote: Total 11 (delta 0), reused 0 (delta 0), pack-reused 11\u001b[K\n",
      "Unpacking objects: 100% (11/11), done.\n",
      "/content/intro_to_software_design\n"
     ]
    }
   ],
   "source": [
    "!git clone https://gitlab.hzdr.de/thiele67/intro_to_software_design.git\n",
    "%cd intro_to_software_design/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ccdi-6m03a0z"
   },
   "source": [
    "Looking in the Files browser on the left, you should now see a folder called intro_to_sofware_design. This is the git repository that we have cloned - enter it and you'll see the tutorial files, including the jupyter notebook that contains the main part of this tutorial.\n",
    "\n",
    "But ... before launching this we need to quicly install *hylite* in this Colab instance. This can take a while, so please give it some time. \n",
    "\n",
    "Once it is done, please continue to the next part of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "5bkdQiiQ3YeR"
   },
   "outputs": [],
   "source": [
    "!pip install hylite\n",
    "\n",
    "# clear output (it isn't easy being clean!)\n",
    "from IPython.display import clear_output\n",
    "clear_output()\n",
    "\n",
    "import hylite # check in case there are errors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4DNyZsg7ADDm"
   },
   "source": [
    "## Part One: Hero Code\n",
    "\n",
    "-------\n",
    "\n",
    "As already mentioned, hero code is part of the creative process - it's inevitable when we want to test a new idea or see if something works. But writing ONLY hero code wastes time and causes excessive hair loss by:\n",
    "\n",
    "1. Being generally hard to debug (as complexity increases)\n",
    "2. Requiring extensive rewriting every time it is used (in different situations)\n",
    "3. Including it's author as a dependency - if nobody can use your code without your help then either nobody will use it or you'll be bombarded by pleas for help\n",
    "\n",
    "This last point is important: we all want our work to have impact. Code that runs once can do little more than create a pretty figure; code that can be deployed and run billions of times by many different users can change the world. And if you don't write code that is deployable, then nobody else will - they have better things to do.\n",
    "\n",
    "----\n",
    "\n",
    "<center><img src=\"https://miro.medium.com/max/480/1*VNh2ihTMppouMBHH5zZlzw.jpeg\" width=\"500px\"></center>\n",
    "\n",
    "---\n",
    "\n",
    "To start this mini-tutorial, I've included a typical example of hero-code. It loads a hyperspectral image dataset, applies a principal component analysis (PCA) to reduce it's dimensionality, and then classifies the result using a k-means classifer. While this could be easily done with existing libraries, I've implemented everything from scratch to serve as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "omRwuqn0NKLp"
   },
   "outputs": [],
   "source": [
    "import hylite\n",
    "from hylite import io\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209
    },
    "id": "mQgyWS2z_fpg",
    "outputId": "4e9704dc-366e-4119-f431-81e5d1c49426"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkcAAADACAYAAAAOace+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhlV1nv8d97xpp77qQ7Q2ciBGJA8CIOIFFAJIwXFRSEwKNXcQgqCHrjFLhcr6hoNCD6ODCKIOITFOUGUEJQJkEwXkIISbo7SSed7urumoczrfvH2pVeXWets/eprq7q7vp+nqeep87ea689nH32WWfv9b7LnHMCAACAV1rvDQAAADid0DgCAAAI0DgCAAAI0DgCAAAI0DgCAAAI0DgCAAAI0DjCujGzfWb2jPXeDuBMY2Y7zOxOMxtc721ZYmY3mNn7TlHdbzWzn16lut5lZm9ejbrOdGZWN7M7zGzXem/LEjN7pZn96ymq+zoze0uRsjSOcEbKLnANM5sxs6Nm9gkzuyKYf7mZfcjMxs1s0sxuN7PXmlk5KDOSLf+x9dmLM4OZOTO7bL23ox9mdquZ/UTw+mozO2ZmP7Ke27WKfkXSu5xz8+u9Iaspe9+OmVl92azfk3S9mdXWeHtOaEiZ2ZVm9pCZ/dJabscp9JOSbnPOPbTeG7IasvNnIbuuT5rZbWZ2VVDkzyS9zMx25tVF4whnst9xzo1IOl/SIUnvkiQzu1TSFyTdL+kq59wmST8s6b9JGg2W/0FJi5KeaWbnruF2Yw2Z2fdLulnSq5xzH1jv7TlZWcPhWkmn5C7NejGziyQ9VZKT9PxwXvblfefy6WvJzJ4g6VOS3uyc+7312o5V9mpJ713vjVhlP5d9L2yVdKuC/XPOLUj6mKRX5FVC4wjr7UnZbd1jZvZOMxvotwLn3Jyk90v6lmzSGyV91jn32qVfRM65bzjnXuqcmwgWvVbSn0i6XdKP9VpH9ovxE9ldqofN7Ppset3MbjSzB7O/G5d+9WZ3Kx4wszeY2aHsF+cLzewaM7srq+v6YB03mNnfmtkHzWzazP7DzB4fzH9M9stowsy+ZmbPD+a9y8zebmb/mC37hayRuDT/imD7v2FmLy6yrJndlhX7z+zX2EvMbLuZfTTbjqNm9hkzOyXXEjO7wMz+zswOm9kRM3tbn8s/V9LfSHqpc+7mRJl9Zvb67O7irJn9hZmdY2Yfy47HJ81sS1D+O8zss9n+/6eZXR3Me5WZfT1b7l4z+6lg3tL58LrgfHhVMP+a7LMwbWYHetydeLKkCefcA8Gyr8zWN21me83sZdn0S83sX7JjN25mf2Vmm1ey72Z2kfm7iD+Znes976D0Ok4Jr5D0efkfOddG5t8q6Tk5dcS24wnZZ2nazD4oqe9rjJl9u6RPSLreOff2RJlbzezN2T7PmNk/mNm27JhPmdm/m28ALpXv9Zl8jpl9JVvufjO7IZi39D5ca2b3Ze/rr4bbamZfypZ92Mx+P7G9F0q6RP6H5NK06DloZluyz/xh89fqj5rZ+Sex787MXpOds+Nm9ruWuIb0Ok69OOfakj4g6bHLZt2qIueRc44//tblT9I+Sf9P0gXyrfx/k/9VVmTZdy2VlTQi3zj6TPb6oPxdgl7L75HUyT44r5N0e4+yo5IeysoNZK+fnM17k/wFfaekHZI+K+l/ZfOultSS9BuSqpL+h6TD2baOSrpS0ryki7PyN0hqSvqhrPwvSdqb/V+VdLek6yXVJH2fpGlJjw6OxxFJ3y6pIumvJH0gmzcsfxftVdm8J0gal/TYvGWz+U7SZcHr/yPfqFzarqdKslNwfpQl/aekP8j2YUDSUwoue6ukj0g6JukZBc7Dz0s6R9J58nch/yM7TgOS/kXSb2Zlz8uO1TXyPy6fmb3ekc1/jqRLJZmkp0mak/TEZefDm7Ljdk02f0s2/yFJT83+37K0XGR7f1bSPwavhyVNBefCLklXZv9flm1jXf78vE3SjSvc94uyc+Gvs3VeJX8+PyM4f99X5Dgl9utuST8j6dvkPwfnLJv/Ikn/0ec5VJO0X9IvZsf8h7K6+7nOfFzSUUkvL3DO3Z29/5sk3SHpLknPkP9cvUfSOwt+Jq/Ojm9J0uMkPSzphcvehz+TNCjp8fJ3wB+Tzf/c0rbKXxu/I7G9z5H0tWXTouegpG3yd9qH5K9dH5J080r2PbimfEr+un9hVvYnsnmvlPSvRY5T4j1Yqqcm6X/LPzYMyzxR0tHc9/5kLl788Xcyf/IX5lcHr6+RdE/BZd8laUHShHxj6O8lXZrNa0r6gZzlf03SV7P/z5PUlvSERNkflfSVxLx7JF0TvH6WpH3Z/1fLN37K2evR7KLw5KD8l4OL3g2SPh/MKy1drLK/g5JKwfy/lnRDcDz+fNmxvDP7/yXKGo7B/D/V8S+95LLZ6+WNozfJNzwuix2TVTw/vlP+y7eygmVvlW8wfFHSYIHz8GXB6w9Lekfw+jplXwSSflnSe5ctf4ukaxN13yzp55edD5Vg/iFlX16S7pP0U5LGcrb3V3Vi43U4+xz8YIF9fWF4Lve57xdl58IVwfzfkfQXwfm71Djq9zg9Rf5zuz17faekX1xW5pmS7u3zPPgeSQ8qaLzL/4Dpp3E0Jf8jZXuBc+5Xg9dvlfSx4PXzdPya0/MzGan7Rkl/sOx9OD+Y/0VJP5L9f5v83fO87X2ZgutNn+fgt0o6tpJ9z147Bddo+UbxP2f/v1LHG0f9Hqdb5X9wTMg3GCclPX1ZmUdJaue99zxW64OZXZjdMiznl0ZB9wf/75e0u49lf885t9k5d65z7vnOuXuy6Ufkfz338gr5OyRyzh2Q9GnFb+VL/s7WPYl5u7PtXrJ8H444f3tX8l+Mkv8VqGDaSPD6kePhnOtIeiCrb7ek+7Np4brOC14fDP6fC+rdI+nJ2eONCTObkL8wnltg2Zjflf+V+PHstviv9Ch7Mi6QtN8511rh8r8uf4G82Y4/6vxY9hmeWXr0lFn+nqTeoz2SfnjZsXyKsvPNzJ5tZp/PHgFMyDc0twd1HVm2P+Gx/sGs/H4z+7SZfWdiv44p6DvnnJuV/xJ5taSHzD8evSLbnnPM7APZI5Ip+X5K25fVV3TflxT5zPY8ThHXSvq4c248e/1+dX8eR+W/9PqxW9IBl30rBtvcj7dL+pKkTwSPGP8kOI+uD8r2cx4lP5Nm9mQz+1T2GGtS/r1d/r6lPrM/LulySXdmj7Oem9ivE86jTPQcNLMhM/tTM9ufnUe3Sdq87LvwVJ1Hedeu5V7jnNssf1ftuZL+1sweF8wflW809XTGNo7sxF7p4+b7JexaVuZR2YXhcPbc85tmdtPSs1LzfQA6wUl+wMzemFqnc+4+59xI8GWHk3dB8P+F8r/yTtYn5T/kUWb2XfK/Hv6nmR00s4Py/TheamaVyCL3yz+bj3lQ/gO85GT34ZHjkT2DPz+r70FJFyx7Ln+hpAMF6rxf0qezhuTS34hzbkWh0c65aefc65xzl8h3kH2tmT19JXXluF/ShYn3pIhZ+Qv9JkkfMrOqc+7Z2b6POOf+aoXb9N5lx3LYOffbWQPsw/KRVedkF+h/kn/Elss59+/OuRfIP6K9Wb6vVMzt8l9+4bK3OOeeKd/4uFP+kYsk/Zb8r/SrnHNj8n3rCm1PD0U+s8njtLyg+XQEL5b0tODz+IuSHm9BnztJj5F/zNqPhySdZ2bhPl/YZx1tSS+Vv6tyi5mNOedeHZxHv9VnfVL+Z/L98nfDL3A+oORPVPw8+qZz7kflz6O3yDcOhiNFb5d0cfj56nEOvk7So+Xveo/J35FT0W1KKHoereja5ZzrOOc+I/9D7vuDWYXOozO2cZRZ6pV+mXyr9JEIAvOhx1+QP+BPyN7Q75a/A/CUoI4Hl07ybPqPm9kL12oHoJ81s/PNbKv844IPrkKdvynpu7JOfku/xC4zs/eZ74x6rXznysfK3x7+VvnO3IOSnh2p76OSdpnZL5jvgD1qZk/O5v21pF8zn3dmu3z/opOJIvo2M3tRdsH6Bfk7H5+XP5fnJL3BzKrmO7c+T77DYZ6PSrrczF6eLVs1syeZ2WMKbtPDChqHZvbc7Hia/C+wtnz/rdX2Rfkvt982s2EzGzCz7+6nAufctKQfkL/D9n47+bu+75P0PDN7lpmVs226OvvBVZPv23NYUsvMnq0TL8pJZlYzs5eZ2SbnXFP+UU7qmH5R/lf7edmy55jZC7IvwEVJM8Gyo9nryaz861e01yf69exOwpXyfUFin9lex2m5F8qfQ+Hn8TGSPqMTo4qeJh9p1I/Pyffzek123r9Ivm9dX7L35Ifl+7v8U6Kx0Y+8z+SofL+YBfOdwV9atGIz+zEz25HdZV6609Z1Ljnfof9uZccj5xwclb/7M5Fdq3+z/13u8nrzHb0vkPTzip9HJ3Xtyu58PVbS14LJhc6jdW0cmdmvmNk95nvG32Fm/30l9TgfgXSz/IdqyQ2S/s35iKUHsnKHnHM3ukQ4r3Nur/zz6OW925e2dylKoJK97reH/h+ajzyYMrMvm9lTg3mDZvZu85EAXzcf4RRGo+w2sw+bvwu218xes5JjdRp6v3yHx3vlG65vlk54hNnvrzxlj9e+U/7Z/NfM35b+sPyt8ab8r9SbnHMHg7+98iGfXY/Wsi/YZ8o3Rg5K+qak781mvzmr93ZJ/yXfmfVkEsx9RP4RyTFJL5f0Iudc0znXyNb/bPkL9B9LeoVz7s68CrPt/35JPyL/Y+Gg/C/K5blkUm6Q9G7zt7VfLH/X7ZPyX7qfk/THzrlPFd7DgrI7tM+T//Fzn/wjxpdIkpk91cxmCtYzIf/+XS7pPXYSkXXOufslvUC+Y/xh+V+2r5fvCzYt6TXyv7aPyX+h/X0f1b9c0j7zjy1eLf/4ILYNDfm+MEsRliVJr5V/b4/KX/yXflm/Ub4D6qSkf5T0d31sT8qn5b9U/1n+0fbHI9uYPE6R+q6V76x7X/iZlPQ2+Zw0FfNPBR4rf50vLDtWL5Lvx3JU/vx55Bj0c50J6lqQ9A92Egk4C3wmf0bSm8xsWv4HV+ouYswPyF/3ZiT9oXxfpFQ+rD+VP++WpM7BG+V/PI7L/1j7v31sT8pH5PtcflX+3PyL5QVWeO16W/aezshf03/NOfcxSTIfDX2NpHfnbl1ep6RT+SffEt8t/4F5ifxt8F0Fl71Vx3ulb5O/WH8kmH9Q0itz6rha0gPB60fJP6b4vkT5i+RvUVeCbeinh/6PZdtakb9NeVDSQDbvt+UvOlvkH6XcvrRt2fH5svyHpCb/K/5eSc9az/ePv9X9U9ChlT/+ev3JR57dqZwO2Ku8zhOuf2u8v2+V9DPrfdzPtj/5RsYdRb93V3G9JwR5rOF6r5PPj5dbdl3vHDnnPuSce9D5Z4MflP9F3s8tzz/K7gqMy3dWuy6Yt11BhzUz+7nsl++Mmf1ZUG53Nn1KvmHzBUn9pC5/p3PuHufcpPytunucc590vtPlh+RDD5f2933OuSPOuZZz7q3yJ+ajs9kvlvRbzrljzt/p+qNgHU+SD4F9k3Ou4Zy7V75PwdmS7RdAH5xzh51zV7izLEN2ivN93P54vbfjbOOcW3TOPdadJRmy8zjnbnLOvaFI2fV+rPYKM/uqHe+F/i3q7pHfy2uc76z2OB2/47LkhIgl59zbnO8geaN8voslDzrfyWtM0mb556r5t9yOK9xD38x+KXtkNpnt7yYd39/dOrH3fvj/Hh1vxC0dq+vlc5MAAIBVtNIokJNmZnvk7348XdLnnHNtM/uqVtD73Tn3X+bHv3m7mT3R+ftn/yz/fPidfdQzaWbv1+p0Cj5B1r/oDfL7+zXnXMfMjun4/j4k37i7I3sd9uS/X9Je59yjVnu7cPpwzt2w3tsApDjn9unkI90AOedO+/NoPe8cDcs/dzws+bT7Oj78w0q8W/5OytKQCjdIeqqZ/X4Q1bFdPgoiysxG5B9VfS1V5iSMykdNHJZUMbPfkDQWzP8b+dDyLdn2/lww74uSps3sl7OO22Uz+xYze9Ip2E4AADa0dWscOefukO9k9zn5R1FXyQ8fIam/aJSsvoZ8z/xfz17fJZ+75nz5caGms/ofXCqT2R30bN8vn848GiVykm6R7+F/V7aeBZ346OxN8tE4e+U7l/+tfFiunI/aea58NN5e+T5Wfy7/WA4AAKwiy3pw4zRjZj8tH4L5tPXeltVUKldcudw7grxcKpZjs1IgcfJAuVasrk6BsSgbBSJ3C94tLheIojflp+RxlcVC61t0jdwyzU6xyP6Wy2+Tz+n2cefcjkIVrsBwvea2Dg+dquq7HKqfsl056+xcPLwq9RydndPsYuOUPX4pD21ylU07V6Wu1drnotb6fOyMVfMLrYPSVDO3TOPg3Su6Fq1bnyOcKMvjcYn8nbRHyYf69zUC+ZmgXK5r27nRNFKPGBvKzewuSdrRGM8tc/lYsTRJ22bzc4qVD1yZW8bNF2uMjeni3DK1Un4jZGHr3YXWt699X26ZB6cuK1TXePN5uWW+rHP7HaKhL1uHh/QLz3xKfsFVctPFK0omviFdt/cdq1LPjZ/oJ2i4f5VNO7Xr2htXpa7V2uei1vp8XHjWefmF1sHALfkDBOx/y3NXdC2icXT6qMkn5LpYPqvpB+QT/QEAgDVE4+g04Zzbr5PrkA4AAFbBmT62GgAAwKqicQQAABDo67GaH3Q1Nr3nMonp6XZZaplSuccypfi89PrTG52K4Fv3yL4VrL7fRZqNptqt1mmfoAsAgFOlz8aRVIssUY1NzNTr8RDAcikd1VOrx8Oq62PpUOr6UHxebSAenlwup8OkW+14eGC7mQ4x77SKhZ8v6dXQUye1jnRTp5OYFW3NHp/bNWXfXXt7lAcA4OxHh2ysqZI6GnBzPcu0FouFw99f3pVbpt0eLVTXLpvNLbN98M7cMo3BYts+tTiRW2ZzdXNumbGdC4XWt71IHhaXaJUvM3D01twyXy5U05mjaKg2If+rdwwO1b+xKvWcaQjTL67Qtr9lZXXT5wgAACBA4wgAACDQ12O1gYEBXfbo7iy6YyMjPZaJ9/mpV9LpyAeH4n2ORrenMwaPbN8WX/9wvC9SuZJuFy42432OWj36HLUb8XnNRuKxR4++QO1m/PHG4kJ6qIhWJ15fp92jS3ZkkUMHDqXLAwCwAXDnCAAAIEDjCAAAIEDjCAAAIEDjCAAAIEDjCAAAIEDjCAAAINBXKH99cEiPuvJxXdN3bBlLLjMyGA/zr/cYPmNoMB7mv+Wc7en17NoZnT4wOhydXu6x563U2Gqd9Da32/Hw+8bifHz6QiNZ12Ji3sJ8OpS/6eLb1mq2kssosp//9olPpcuvgnLJadNQ76FWOuV4KofljlXzs1GPl4qd4ubi71OovSM/G/XsaPqzECov5P8u2dSZyS2zZ7TH+xsYLt+bW2ZX865idc3dk1+oWOLus06RTNpFMiAXzchdBFm74zpj1dwMywO3HChU1+l4jM/kzNenA+4cAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABPqKVqvWatq95+Ku6edsGUouM1KLR+9US+l2Wa0anzcymh7gdnBkS3T6wFg8Wq02UE7WVUpE0lVq6Sgql2hnNhbjEWazE1PJuuZm45FTC4vpCLdWOR7h126ml3GRCLtqNT0gMAAAGwF3jgAAAAI0jgAAAAJ9PVYDTlapXNLwWPoxrCSVBoo92lus1nPLzDeKJZQ8nEiieUJdPR7FLpnZvbnQ+ppz+QksR6bzMym2m8cKre+KVn7ixk0jDxSqa+jceJLUE+wrVNWGtJoJHldrfadjEsOzTdH3/Ux9L4omzCzidEhgyZ0jAACAAI0jAACAAI0jAACAQF99jqxUUmUg0s/D0v06Komw/FIl3a+kJYtOn5xOh78vJlIDpMY4HdqaHiDU2nPR6eVKepDPSjVxDCzeT6WaKi9pKDGr1iPM3g3EF2p14gPiSpI63X1HKhW6oQEANjbuHAEAAARoHAEAAARoHAEAAARoHAEAAATofYu1Va5Jmy7oWaQyFh+Pbrl6PT8J5Mzi1kJ1jZe25ZZ5cCw+fl9o4fxNhdbnxpu5ZSoazy3TbKSDFELlwfzEmrvmCiR3lDRUJEfnvkJV4TRxticoXKmiyQiLJEA8k4/daiZ4XM31ncpkkf01jjpttRemuyaXhuODu0pSxeIX3HKPgWfbiWi15mL6C8VNTcanW7yuViRS65F5i937KEnlUjpDcq02GJ0+MDIanV6vp79hqgPxulwnHS2nejxarddXcMd1H4NSj/cFAICNgG9CAACAAI0jAACAAI0jAACAAI0jAACAAI0jAACAQF/Rap12S4uT3eHFnU09Qpzb8VWUK73ChuMRZo0eSzQW4uOhtTrxwdVm52aTdS3OxqPVLBLdtWQgEWG2eceO6PTqzp3JuuoDtej0cjl+XCTJJcZws15ReZH9Sa8BAICNgTtHAAAAARpHAAAAATJkY00NjI3q8qd/T88yo4PxR5HLjbv0o8kl+w8XSecsTSqerDO0ODaWW2Z6y0Kh9U2XH85fn92fW+a+drEHoZ2x/N9BD8ydW6iuTeMH8wt95d5Cda3UofqONc04XDSD9NluLY/DjYuH12xdZ6O1zmq9Hk7lPnLnCAAAIEDjCAAAIEDjCAAAINBfKH+rqdnxh7qmL25P91Vo1OJ9PkrlHiOqJ7pRdNo9wtKb8UFZG7Pz8emN9CCui/PxMH9Tev2DQ/GBXzuN+PqtnR4StpXo2zIwkD5mVo4nOlhsxFMZSFI7Mshsp9NJlgcAYCPgzhEAAECAxhEAAECAxhEAAECAxhEAAECAJJBYU/Xhki799qGeZc6fPr9QXa6RnyzyoXqxDuYTzR4BApmFgeHcMoc3xce4W+5AJ7+u+8r5SSfvaxRL3Lj/4pHcMocrFxSqa/N09/iKXT54apNAFrHwrPNWra6bblm7hJP9IDnl6aHIubYRkjKeTfpqHDnXUaPRnQG4OXcovdBoPPNwuZK+aWXV+MCrNcUHd5Ukp3hUVsnFo9JKPb4zS4kBXnuMO6uFxXj02ZFD8SyvC7MzybqGE8dsaDT9BVepxaPl5NLHuRIZLNe10tFtAABsBDxWAwAACNA4AgAACNA4AgAACNA4AgAACNA4AgAACNA4AgAACPQVym+lsuoj3WHm9VI6/LvamYtOH6qn22UDWzYnNiAxXVKjEV9PoxkfkLXZY+DZudn4wLNz891pDJbMz8TXPz91LF7X1ESyrsnxePj/YI9Q/mo9nqdnoJZOfzCyaWvXtHYrfVwAANgISAKJNVWpNbX9wgd7lrn0jnieqeUutwdyy8wOFcvbNL2QbkQuOdbZllvmYRVLAvn14XgDPPQlxRv2J2zTbLGbv9MX784tM3V+sSSQcyP526Wf/ctCda1UZ6y6qkke86zluvpRJDkliSJPD0XPoSLJIlfzPb3p4tMzwel647EaAABAgMYRAABAgMYRAABAgMYRAABAoK8O2eVKVSPbdnZNH96cHmF8oBSP8BqupjvKbtkaH3h1dHu6Q6xTfCTZdis+IOz8fLpT6fTcfHT60SPxyDNJGj/4cHR6YyHe8XZ68mh6/Yvx9VeOptuy9YH4wLOjwz1Gdu90j6TbaROtBgDY2LhzBAAAEKBxBAAAEKBxBAAAEKBxBAAAECBDNtaUc3Nqdf6rZ5n2gZsL1XXu3Hh+ocqOQnWpcVFukSMLj84tc9/iuYVW17L8LODHOvFggtABlx7SJnT3pniQQ2j2su7hZGI6p8FlozTVzM0kfLpmtV5NRfaRLNpnnyJZrc/k93RVP7tvWdlifV3lSpWKhnd0X/yHz0+PeVadPBidXnfTyWXGqvEvhfMu2pVcplaLR2s1F+NRaVOTk8m6js7Eo8XKtVpymU4nHn23uJAac+1Isq7FmXhUXHNuMbmMmvHhLyqt9DLVWveNw047/wsZAICzGY/VAAAAAjSOAAAAAjSOAAAAAjSOAAAAAjSOAAAAAjSOAAAAAv0lLCmVVK7VuyaP7Lg4ucjoaDzMv9ojZLwzHQ9zb00fTi5T23JOdHop0fwrlbsHXV1iiWWqtWpymYHhoej0wc2botOHRtMDwrbnpuLTS+mw/EpiXruVHkh2YToWys/AswCAjW39s7lhQym7jsYWZ3qWmdr29UJ1HflGPLdUaFu5UFVSZ09ukfbiSG6Z5rF0o/eE1VmPnFWZwWr+/u0azk/uKEnH0mm1HlFqx3NyLTdQzt+u9M+Y1dEZq26IJI84dYokEj2TFUkUuR7OlM8tj9UAAAACNI4AAAACNI4AAAACNI4AAAACfXXINpnK5e7BV2sDw8ll6onRwN3EoeQyszPxaK0j++9NLtNsx3velhPb1mynO5V2XDxiy1k6wq1UjR/KeiqKbTQexSZJ7fn4oLitufQI7K4RHyy33YwPvCtJC7PdI8OnBtAFAGCj4M4RAABAgMYRAABAgMYRAABAgMYRAABAgAzZWFuNukoHLulZ5MjovkJVfdbuyS3z+CM7CtU1NLMrt8yhhfNzy0yN5dcjSYvVdEf5JaV6fpnNKpYCfPPBTm4ZdygeCLFcbdex3DKnOkM2vNXK8Fw0m/J1e9+xKuvD2edMyXxdVF+NI+ekVqP7ItsrIqq0bXd0eqfHMjOz09HpjXvS0WoLikeF1bfFx1xLBLdJkpqJaLW2ekRyJQZxq9YG4tuVGHNOklrz8f1f7KSH3Vicn41Obyymx7BrNLqj3zrt/C9RAADOZjxWAwAACNA4AgAACNA4AgAACNA4AgAACNA4AgAACNA4AgAACPQVyt9uNXV0vDuDyZGHJ5LLbN0Sz/tSG06HsrvReCj71L5vJJdpHTwYnV538V200bFkXe1OPJy90yOnTLnaPSCvJFUSofy1kfT652dGotNbE/F1SNLsfPcgspK0MBtPSyBJzrrbxp1OenBdAAA2ApJAYm0tDKlz57f1LDJzZbVQVfv33JZbZn7iikJ1bT10aW6ZhYk9uWWmt+8stL750XRDd4kbzq9noNIj91Zg9MB9uWUa33y4UF2tXcXKrbfVSpAonX0J7gD0xmM1AP3NwJ4AAAY5SURBVACAAI0jAACAAI0jAACAAI0jAACAQF8dslutVjRabWLiSHKZhcZidPrQ5vhAsZJU2bI9Or158IH0eqbiI4U3K4kosnK602+pGp9n1XjkmSRVhuKRbOX5eLRYbWQ0WVd1eFN8/bX0Mi0bj05f6DGwe6fdPSgt0WoAgI2OO0cAAAABGkcAAAABGkcAAAABkkBibc0NSl+5qmeRsUvS/dFC939vd/+35e46lp+4UZKG9p6TW6Y0viW3zMLU1kLra22L9ysL2bb8j6c1Fwqtr1aKZ50Pua3prO2hiUvS2e1PJ0UTNxZJFrmaCSUBnP64cwQAABCgcQQAABDo67Gac07NVndo+vz8fHKZ+dmpeF09QvkHN8UfXzR27k4us3AsHsrenIkPiutG0o8G6mPx9adC/CWp7OLtzGo9nkpgcCwdlt+ci69/LhHiL0nl+mB0utNkcplWszvO3zlC+QEAGxt3jgAAAAI0jgAAAAI0jgAAAAI0jgAAAAI0jgAAAAJ9JoE0ybrbU41GenTT+el48rl2Y1tymfLmeDK6oV3phH6NRFRce3EuOt3N9EiKlxgU1lyPwWrNotPriWi1VBSbJFkzPljv4pF0gsHp4ZHo9LlqfEBcSWrNz3RPdJ1keQAANgIyZGNNldrS4LHe6QLObRVLJzA5uiO3zOHLitU1uy+/UVjfn5+5u3NkoND6qu14YzpUS6SHCJWVTgkRGmxuzy0zsPVIobrKD0Ua1QA2tKJZ5Itmrl9vPFYDAAAI0DgCAAAI0DgCAAAI0DgCAAAI9D22WqvZ3XF1fmYhuUxrNh7J1lxoJ5dptxKdVWvpzqeuHu8I25yLR6u1jzycrMtGh+PT2+ltLln8UA4PxrerVKsn6yp34utp7khH+M0ejkeyLU6mO9m2G93HplTK7ygMAMDZjDtHAAAAARpHAAAAARpHAAAAAZJAYk05J7Vavctsm433E1vuksV437DQ3CXxbOPLTVwVz7Aeat3TI6t6pjSenyhSkkrTOQdBUrk+mFumVtpUaH2bp/KTQDZ2Hi5U1+LkQ7ll0j36Tj9FktIVTXB3trvp4p/OLXPd3neswZYApxZ3jgAAAAI0jgAAAAJ9PVbrOKeFZnfYfsuayWValXhoeEvpsayanXh9pVKPUPrhRJj/9GR08sJU+hGCm9wcnd5rrKuBofj6B4bjIfulSnrg2XYj/mhmeNOW5DKjW+Nh/tNHxpPLLEx3j5FlpfRAtQAAbATcOQIAAAjQOAIAAAjQOAIAAAjQOAIAAAjQOAIAAAj0Fa1WKplq9WrX9HLFJZdptGaj02fn0kn3BgbjmzVYT7flqsMj0eml1IC08+lEdjZxNL6OejrpYHkkHq1Wqcaj9TqddBJAc/FIvlK1+9g/sm2jY9HptZF0ksDKwLHudZdObXu5YyXNVnonb6zMpCMZQxdPxfc59NB58WjF5eYen5/g0d21N7dM6Z70ZyHUmc3fdjebX1e9WSwJ5K5yPAIzNHg0f5skyabyE1jeWaimlStNNXMTMxZJ7lhU0bpIFlksUaR0ZiSLXM1zaDWdyedZkW0/HY47d44AAAACNI4AAAACNI4AAAACNI4AAAACNI4AAAACNI4AAAACfYXym5U0MDDYPd2lQ44nj8YHeK2V0wOcdpqN6PQtW9Ih4KlNaCUGi52bm0/W1Tx8KDq9OpwOdW5vis9bnI+vv7G4mKxrdqp7QFhJml+IT5ekVuJ4dmrxgW8lqVONzDPaywCAjY1vQgAAgACNIwAAgEBfj9WAk9WxuubrF/UsM9/eX6iuEbc9v8xodxbwGNuTX+7IE+/KLTP4pfTj4pDbd2l+oYn0I9EltfTT2RPsKJD8etPRYpeDzt6CKwV6yMukfaj+jTXakjNPkQzSZ3sWbenUZtLmzhEAAECAxhEAAECgr8dqk0eOjP/9X76n2DMPnKn2rPcGAACwnvpqHDnndpyqDQEAADgd8FgNAAAgQOMIAAAgQOMIAAAgQOMIAAAgYK7HuGjAajOzw5KIeDz77TmVARycRxsC5xBWw4rOIxpHAAAAAR6rAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABGgcAQAABP4/F3yoGg4ZqrEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x288 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load awesome dataset\n",
    "image = io.load('/Users/thiele67/Documents/notebooks/teaching/Hero_to_Zero/sampleA.hdr')\n",
    "# image = io.load('sampleA.hdr')\n",
    "fig,ax = plt.subplots(1,4,figsize=(10,4))\n",
    "image.quick_plot(hylite.RGB, rot=True, ax=ax[0])\n",
    "\n",
    "# not sure what this does exactly, but it seemed like a good idea at the time\n",
    "X = image.data[3:-3,3:-3,10:-5].reshape((-1,435))\n",
    "X = X / 10000.\n",
    "\n",
    "# performance optimisation hack\n",
    "X = X[::3, : ]\n",
    "\n",
    "# PCA\n",
    "mean = np.mean(X, axis=0)\n",
    "X = X - mean[None, :]\n",
    "cov = np.dot(X.T, X) / (X.shape[0] - 1) \n",
    "eigval, eigvec = np.linalg.eig(cov)\n",
    "idx = np.argsort(np.abs(eigval))[::-1]\n",
    "eigvec = eigvec[:, idx]\n",
    "eigval = np.abs(eigval[idx])\n",
    "data = image.data[3:-3,3:-3,10:-5] - mean\n",
    "out = np.zeros((data.shape[0],data.shape[1],3))\n",
    "for b in range(0, 3):\n",
    "    out[..., b] = np.dot(data, eigvec[:, b])\n",
    "pca = hylite.HyImage(out)\n",
    "pca.quick_plot((0,1,2), rot=True, vmin=2, vmax=98, tscale=True, ax=ax[1])\n",
    "fig.show()\n",
    "\n",
    "# classify\n",
    "np.random.seed(42)\n",
    "xcentroid = np.random.randint(0,pca.xdim(),3)\n",
    "ycentroid = np.random.randint(0,pca.ydim(),3)\n",
    "centroids = np.vstack([xcentroid,ycentroid]).T\n",
    "centroids = pca.data[centroids[:,0], centroids[:,1],:]\n",
    "delta = np.array([np.inf, np.inf, np.inf])\n",
    "while np.max(delta > 0.0001):\n",
    "    clss = np.argmin( np.linalg.norm( (pca.data[None,:,:,:] - centroids[:,None,None,:]), axis=-1 ), axis=0 )\n",
    "    means = np.array([np.mean( pca.data[clss==i, :], axis=0) for i in range(3)])\n",
    "    delta = np.linalg.norm(centroids - means, axis=1 )\n",
    "    centroids = means\n",
    "hylite.HyImage(clss[...,None]).quick_plot(0,cmap='tab10',rot=True, ax=ax[2])\n",
    "\n",
    "# classify another image (using previously \"learned\" centroids)\n",
    "image = io.load('/Users/thiele67/Documents/notebooks/teaching/Hero_to_Zero/sampleB.hdr')\n",
    "# image = io.load('sampleB.hdr')\n",
    "\n",
    "data = image.data[3:-3,3:-3,10:-5] - mean\n",
    "out = np.zeros((data.shape[0],data.shape[1],3))\n",
    "for b in range(0, 3):\n",
    "    out[..., b] = np.dot(data, eigvec[:, b])\n",
    "clss = np.argmin( np.linalg.norm( (out[None,:,:,:] - centroids[:,None,None,:]), axis=-1 ), axis=0 )\n",
    "\n",
    "hylite.HyImage(clss[...,None]).quick_plot(0,cmap='tab10',rot=True, ax=ax[3])\n",
    "\n",
    "ax[0].set_title('a. RGB image')\n",
    "ax[1].set_title('b. PCA components')\n",
    "ax[2].set_title('c. K-means (sample A)')\n",
    "ax[3].set_title('d. K-means (sample B)')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LiR9lJ_JA3CX"
   },
   "source": [
    "#### Exercise\n",
    "\n",
    "Use the above code to create a set of elbow plots for different numbers of PCA components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LLBUARmXA5sK"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bMePUzngA6YE"
   },
   "source": [
    "## Part two: A better way\n",
    "\n",
    "As the above exercise should have demonstrated, simple adjustments such as the number of classes or number of PCA dimensions are not easy (and basically require the code to be entirely re-written). Hence, while the code is not wrong (it works!) it is not reusable and basically useless except for the specific task and dataset it was written for.\n",
    "\n",
    "There are many different (and often conflicting) [design principles](https://www.callicoder.com/software-development-principles/) that can help ensure code is maintainable, durable and reusable. The most important points to keep in mind for this exercise are summarised below.\n",
    "\n",
    "###### KISS: Keep it simple, stupid\n",
    "It's very easy to make things far more complex than they need to be. Try and keep software tools focused and as simple as possible.\n",
    "\n",
    "###### DRY: Don't repeat yourself\n",
    "Break code into well defined, bit-sized functions rather than repeating (copy + paste is evil!) code. This makes it easier to (1) find and correct bugs, (2) make changes or updates and (3) provides lots of useful components for building new software.\n",
    "\n",
    "###### Single responsibility principle\n",
    "Functions should have a single purpose and be organised (into modules) according to their responsibilities\n",
    "\n",
    "###### Continuous testing\n",
    "Test always - every block of code should be tested individually during development (in e.g., a notebook environment) and then test functions written to automatically identify issues caused by subsequent changes. \n",
    "\n",
    "###### Document everything\n",
    "Code without documention is unusable. And there are now so many tools for creating and using documentation that there are no excuses either 😉\n",
    "\n",
    "----------\n",
    "<center>\n",
    "<img src=\"https://preview.redd.it/0jelwo9pgcv21.jpg?width=640&crop=smart&auto=webp&s=afc465891e9cd751e8cb866612cff494c60b7a8a\" width=\"400px\">\n",
    "\n",
    "**Sometimes it is easiest to start from scratch...**\n",
    "</center>\n",
    "\n",
    "---------\n",
    "\n",
    "To demonstrate some of the above principals, lets break our \"hero code\" example into chunks that fit with our design principles. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eVTEQpyuBLDT"
   },
   "source": [
    "#### Exercise\n",
    "\n",
    "Define a set of functions (no need to implement them) that breaks the above hero code into chunks that conform especially with the KISS and DRY principles. Also write docstrings for these functions following the [google docstrings](https://gist.github.com/redlotus/3bc387c2591e3e908c9b63b97b11d24e) style. If done consistently (from the start of a project!) these can then easily be converted to a friendly website, such as for [hylite](https://hifexplo.github.io/hylite/hylite.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Yf7lbImZBBT6"
   },
   "outputs": [],
   "source": [
    "# write functions here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 539
    },
    "id": "uNMphGsGBRkI",
    "outputId": "d5526b67-02f5-46ee-f4b6-6ae37ef19939"
   },
   "outputs": [],
   "source": [
    "# write test code here (more on this later!)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EtQjasGxGN97"
   },
   "source": [
    "## Part three: Testing and modules\n",
    "\n",
    "### Testing\n",
    "While the code above gives us some confidence that our functions are doing what they should, it is best practice to write some python functions that can do this automatically. Then, if you make some cool improvments or changes you can just run these test functions to see if your code still works.\n",
    "\n",
    "Python provides several *testing frameworks* to help with this, but to start with we'll write some test functions using pytest.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_YZUxVGzVPwM"
   },
   "outputs": [],
   "source": [
    "!pip install pytest\n",
    "# clear output (it isn't easy being clean!)\n",
    "from IPython.display import clear_output\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0gsPs31JVX7Z"
   },
   "source": [
    "While we could probably convert our code the test cells above into pytest functions, this is not generally good practice as it relies on external data. This is the biggest challenge when writing good test code - (1) how do I automatically generate a realistic input for my function (think intput data), and (2) how do I know if the results are correct?\n",
    "\n",
    "For the case of K-means it is not too hard however - as we can create a synthetic dataset with three subclusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VdN6Odj9V13x"
   },
   "outputs": [],
   "source": [
    "def generate_data(n=3,ndims=10,points_per_class=10):\n",
    "    \"\"\"\n",
    "    Generate a synthetic dataset with n clusters for testing k-means.\n",
    "    \"\"\"\n",
    "    X = []\n",
    "    for i in range(n):\n",
    "    X.append( i*3 + np.random.rand(points_per_class,ndims))\n",
    "    return np.vstack(X)\n",
    "\n",
    "def test_pca():\n",
    "    \"\"\"\n",
    "    N.B. This test function simply runs code but does not check its output - this\n",
    "       a \"shallow\" test.\n",
    "    \"\"\"\n",
    "    X = generate_data(3,10,10)\n",
    "\n",
    "    loadings, mean = pca_loadings( X, step=1, ndims=3 )\n",
    "    assert loadings.shape == (10,3), \"Error - loadings returned wrong shape (%s).\" % loadings.shape\n",
    "\n",
    "    pca = apply_loadings( X, loadings )\n",
    "    assert pca.shape == (30,3), \"Error - pca returned wrong shape (%s).\" % pca.shape\n",
    "\n",
    "def test_kmeans():\n",
    "    X = generate_data(3,10,50)\n",
    "    X=X.reshape( (15,10,10) ) # change shape of inputs to test shape independence\n",
    "    loadings, mean = pca_loadings( X, step=2, ndims=2 ) # try to ensure test functions test a wide range of parameter configurations\n",
    "    pca = apply_loadings( X, loadings )\n",
    "    centroids, clss =  kmeans(pca, 3 )\n",
    "\n",
    "    # check classification is correct\n",
    "    clss = clss.ravel()\n",
    "    assert len(np.unique(clss)) == 3, \"Error - kmeans gave incorrect number of classes\"\n",
    "    assert (clss[:50] == clss[0]).all(), \"Error - kmeans gave incorrect classification.\"\n",
    "    assert (clss[50:100] == clss[50]).all(), \"Error - kmeans gave incorrect classification.\"\n",
    "    assert (clss[100:-1] == clss[100]).all(), \"Error - kmeans gave incorrect classification.\"\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "x8gb_-r8hStb",
    "outputId": "ef302980-75f6-4637-b912-96e921ebc622"
   },
   "outputs": [],
   "source": [
    "# running tests is as easy as this now!\n",
    "test_pca()\n",
    "test_kmeans()\n",
    "print(\"All tests passed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q-8ZrKmLZFk_"
   },
   "source": [
    "Now that we have these automatic test functions, we can put them in a python file that accompanies our module. But first - we need a module!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BQBT9IRTUfu1"
   },
   "source": [
    "\n",
    "### Modules\n",
    "Modules (or *packages*) are containers of code that can be easily loaded into a python script using the `import` statement. This makes code much much easier to reuse (rather than e.g., copying our functions into every notebook we want to use them in).\n",
    "\n",
    "\n",
    "Unfortunately, to create a module we have to move out of the jupyter notebook environment, but (fortunately) the community of geeks on GitHub can provide a template package we can pull from and then modify.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dFUDLiesECR-",
    "outputId": "feb960be-002e-4bcc-a58e-bb5bfa60c9d4"
   },
   "outputs": [],
   "source": [
    "# download a template python package\n",
    "!git clone https://github.com/hbergen/minimal-python-package.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "40H7vlw2HXZV"
   },
   "source": [
    "The above will have downloaded a template package for us. If we look into this we will see the key ingredients of a python package:\n",
    "\n",
    "1. *winterfell*: This directory actually is our python package. **Rename this to whatever you want your call your toolbox** - I chose *samsatool*\n",
    "\n",
    "2. *winterfell/__init__.py*: This is what makes the directory a package. Code in this file will be called when we import the package. **Copy the functions we declared above into this file, along with the relevant import statements.**\n",
    "\n",
    "\n",
    "3. *setup.py*: this script explains how python should install this package. Details of this are beyond the scope of this tutorial, but open it and have a look. Among other things, it contains the metadata for our python package (e.g. author, licence, etc.). **In the *packages* entry change 'winterfell' to your package name**\n",
    "\n",
    "4. Create a new python file called **tests.py** and copy our test functions into this. \n",
    "\n",
    "These tests can then be run automatically using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yW-Z-Go4aSLC",
    "outputId": "92aa5f07-191d-497c-9465-6b32049558df"
   },
   "outputs": [],
   "source": [
    "!python -m pytest minimal-python-package/tests.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "esy1CEWGaRB4"
   },
   "source": [
    "If our tests all passed, now we can **install** our new python toolbox!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aZCIlpOcHTzA",
    "outputId": "a63661e2-03a3-48cb-956e-b4e4c5616a6b"
   },
   "outputs": [],
   "source": [
    "!pip install ./minimal-python-package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lQoQLb78JaQ1"
   },
   "source": [
    "Assuming it installed properly (fingers crossed!) we can now import our fantastic functions and use them without having to see all that nasty code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dn_JBOpOJZQh"
   },
   "outputs": [],
   "source": [
    "import samsatool\n",
    "#from importlib import reload  # use this to force reload a module after reinstalling\n",
    "#reload(samsatool) # (if you made a mistake)\n",
    "from samsatool import nearest_neighbour, kmeans\n",
    "from samsatool import apply_loadings, pca_loadings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JqMAhmGaJYnX",
    "outputId": "210df0e0-e16c-4c33-c1cb-9602e07fd422"
   },
   "outputs": [],
   "source": [
    "# show the help docstrings for our functions\n",
    "help( pca_loadings ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 252
    },
    "id": "oJ07qD77KWJU",
    "outputId": "2c7e947a-663d-4a4f-9d2d-86decc9b6b9f"
   },
   "outputs": [],
   "source": [
    "import hylite\n",
    "from hylite import io\n",
    "\n",
    "# apply these to a different image to show how they work\n",
    "image = io.load('sampleB.hdr')\n",
    "\n",
    "loadings, mean = pca_loadings(image.data)\n",
    "pca = apply_loadings(image.data, loadings)\n",
    "centroids, clss =  kmeans(pca, 3 )\n",
    "\n",
    "fig,ax = hylite.HyImage(clss[...,None]).quick_plot(0, figsize=(4,4), rot=True)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GJmSW8lCOh6F"
   },
   "source": [
    "### Part four: Where next with GIT?\n",
    "\n",
    "So far we have only used a single git command - *git clone* - for downloading code from e.g., github. While this is by far the most common git operation, it can obviously do far far more.\n",
    "\n",
    "I would suggest looking at a proper [git tutorial](https://medium.com/swlh/an-introduction-to-git-and-github-22ecb4cb1256) to learn about the basics of creating and updating git repositories using *stage*, *commit*, *push* and *pull*. But GoogleColab really isn't the place to do that as setting up git permissions here be rather messy....\n",
    "\n",
    "Instead, lets look at some of the more advanced things that a git repository can do, as inspiration to help justify the (potentially steep) learning curve.\n",
    "\n",
    "- [version control and releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases)\n",
    "- [automatic testing](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python)\n",
    "- [build and host documentation websites](https://docs.github.com/en/pages)\n",
    "\n",
    "The [hylite](https://github.com/hifexplo/hylite) repository uses all of these - have an explore, especially inside the [.github/workflows](https://github.com/hifexplo/hylite/tree/master/.github/workflows) folder. Also see how [tags](https://github.com/hifexplo/hylite/tags) can be used to create and store different releases.\n",
    "\n",
    "\n",
    "\n",
    "-----\n",
    "\n",
    "<center><img src=\"https://miro.medium.com/max/1400/1*B-OPCn67xfe4POnbiJL0Rw.png\" width=\"400px\"></center>\n",
    "\n",
    "----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hwc8wSNIiNPy"
   },
   "source": [
    "## Part Five: Some conclusions\n",
    "\n",
    "1. Software is a compromise between functionality and complexity (which detetermines robustness and maintainability). Complexity is often necessary, but basic principles like DRY can help keep it to a minimum.\n",
    "\n",
    "2. Test, test and then test again. And where possible, write reusable test code - your future self will love you for it (even though it's boring).\n",
    "\n",
    "3. Document completely. Code without documentation is like a bike without pedals - completely useless.\n",
    "\n",
    "4. Everyone starts with hero code - that's normal - but generally it pays off to rewrite the good bits properly 🤓\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"https://pics.me.me/brenan-keller-brenankeller-a-qa-engineer-walks-into-a-bar-66974927.png\" width=\"400px\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4SjXIBEdOPVs"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}