Workshop III - Pandas and Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Workshop III - Pandas and Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Pandas?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From https://towardsdatascience.com/a-quick-introduction-to-the-pandas-python-library-f1b678f34673\n",
    "\n",
    "\n",
    "Pandas is quite a game changer when it comes to analyzing data with Python and it is one of the most preferred and widely used tools in data munging/wrangling if not THE most used one. Pandas is an open source, free to use (under a BSD license) and it was originally written by Wes McKinney (here’s a link to his GitHub page).\n",
    "\n",
    "What’s cool about Pandas is that it takes data (like a CSV or TSV file, or a SQL database) and creates a Python object with rows and columns called data frame that looks very similar to table in a statistical software (think Excel or SPSS for example. People who are familiar with R would see similarities to R too). This is so much easier to work with in comparison to working with lists and/or dictionaries through for loops or list comprehension\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import scipy\n",
    "import pingouin as pg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pd.DataFrame({'Yes': [50, 21], 'No': [131, 2]})\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pd.DataFrame({\n",
    "    'Cat': ['I liked it.', 'It was awful.', 'Crunchy'],\n",
    "    'Dog': ['Pretty good.', 'Bland.', 'Bad.'],\n",
    "    'Bird' : ['Terrible.', 'Even worse.', 'Best thing ever']})\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pd.DataFrame(\n",
    "    {\n",
    "    'Cat': ['I liked it.', 'It was awful.', 'Crunchy'],\n",
    "    'Dog': ['Pretty good.', 'Bland.', 'Bad.'],\n",
    "    'Bird' : ['Terrible.', 'Even worse.', 'Best thing ever']},\n",
    "    index=['Cat food', 'Dog food', 'Bird seed'])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access the bits of data we want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By row or column number/s - iloc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "a.iloc[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By row or column names - loc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.loc[:, 'Cat']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.loc[['Cat food', 'Bird seed'],['Cat', 'Bird']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "titanic = sns.load_dataset('titanic')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.loc[titanic.age > 40, [\"age\",\"fare\"]].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Almost any way of manipulating your data can be done\n",
    "\n",
    "- Load from CSV or Excel\n",
    "- Add new rows or columns\n",
    "- Change values\n",
    "- Merge data frames\n",
    "- Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame(\"https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf\", width=800, height=1500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The reproducibility crisis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame(\"https://www.nature.com/articles/533452a.pdf\", width=800, height=1500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a p-value?\n",
    "\n",
    "## If a p-value is less than a cutoff, what does that mean?\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins = sns.load_dataset(\"penguins\")\n",
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sns.displot(penguins, x=\"flipper_length_mm\", height=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "a = pd.DataFrame(penguins.loc[:, \"flipper_length_mm\"])\n",
    "a.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sns.catplot(data=penguins, x=\"flipper_length_mm\", kind=\"box\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Population = complete set of individuals we want information about\n",
    "\n",
    "### Sample = subset of the population\n",
    "\n",
    "#### - We want a non-biased sample\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Whats the difference between a standard deviation and a standard error of the mean?\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Probabilities describe the process of sampling from a population\n",
    "\n",
    "### The probabilities lie on a distribution\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discrete random variables v.s continuous random variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binomial distribution - discrete random variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from numpy import random\n",
    "\n",
    "sns.displot(random.binomial(n=10, p=0.5, size=1000), kind=\"hist\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See also Poisson distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normal distribution - continuous random variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "x = np.random.standard_normal(100000)\n",
    "sns.displot(x,kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See also t-distribution, f-distribution, chi-squared distribution - test statistics lie on a probability distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Null hypothesis\n",
    "\n",
    "i.e. there is no difference between the mean of two populations\n",
    "\n",
    "### 2. Alternative hypothesis\n",
    "\n",
    "i.e. There is a difference between the mean of two populations\n",
    "\n",
    "### 3. Test statistics\n",
    "\n",
    "### 4. Rejection region\n",
    "\n",
    "### 5. Check assumptions and draw conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nominal variables - Fisher's Exact Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pd.DataFrame(\n",
    "    {\n",
    "    'Cat': [7,2],\n",
    "    'Dog': [1,12],},\n",
    "    index=['Cat food', 'Dog food'])\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Null hypothesis : Whether an animal likes cat food or dog food is independent on whether they are a cat or a dog\n",
    "- Alternative hypothesis : Whether an animal likes cat food or dog food is dependent on whether they are a cat or a dog\n",
    "- Level of significance : 0.05\n",
    "- Assumptions of the test: individual observations are independent; totals are fixed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "oddsratio, pvalue = scipy.stats.fisher_exact(a)  \n",
    "pvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As p<0.05 We reject the null hypothesis; there is an association between being a cat or a dog and liking cat food or dog food\n",
    "- one variable has the ability to predict the other\n",
    "- the test statistic does not indicate anything about the strength of the association"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is a p-value?\n",
    "\n",
    "## The probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "display.Image(\"https://s3.amazonaws.com/libapps/accounts/73970/images/hypothesis_testing.png\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## t - test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The t-test is any statistical hypothesis test in which the test statistic follows a Student's t-distribution under the null hypothesis (wikipedia)\n",
    "- The t-test was developed by a chemist working for the Guinness brewing company as a simple way to measure the consistent quality of stout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_subset = penguins[penguins.species != \"Chinstrap\"]\n",
    "sns.catplot(data=penguins_subset, x= \"species\", y = \"flipper_length_mm\", orient=\"v\", kind=\"box\", height=6, aspect=0.6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Null hypothesis : The flipper length means of the Adelie and Gentoo populations are equal\n",
    "- Alternative hypothesis : The flipper length means of the Adelie and Gentoo populations are different\n",
    "- Level of significance : 0.05\n",
    "- Assumptions of the test: Data in each group must be obtained via a random sample from the population; data in each group are normally distributed; data values are continuous; variances for the two independent groups are equal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get out values from the penguins dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "a_flip = penguins.loc[penguins.species=='Adelie','flipper_length_mm']\n",
    "g_flip = penguins.loc[penguins.species=='Gentoo','flipper_length_mm']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test for normality - Shapiro test\n",
    "- Null hypothesis: flipper length for the population is normally distributed\n",
    "- Alternative hypothesis: flipper length for the population is not normally distributed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(scipy.stats.shapiro(a_flip))\n",
    "print(scipy.stats.shapiro(g_flip))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Alternative hypothesis rejected in each case - both populations are normally distributed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test for equality of variances (homoscedasticity) - Bartlett test\n",
    "- Null hypothesis: flipper length variances for the two populations are equal\n",
    "- Alternative hypothesis: flipper length variances for the two populations are different"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scipy.stats.bartlett(a_flip, g_flip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use the pingouin library to conduct the t - test\n",
    "- this is newish python library (so be aware) that gives lots of additional outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pg.ttest(a_flip, g_flip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As p<0.05 We reject the null hypothesis; the mean flipper length of the Gentoo and Adelie populations is different\n",
    "- the test statistic does not indicate anything about the strength of the association"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<iframe src=\"https://pingouin-stats.org/generated/pingouin.ttest.html#pingouin.ttest\" width=\"1000\" height=\"800\"></iframe>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As p<0.05 We reject the null hypothesis; there is an association between being a cat or a dog and liking cat food or dog food\n",
    "- one variable has the ability to predict the other\n",
    "- the test statistic does not indicate anything about the strength of the association"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ANOVA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Null and alternative hypotheses?\n",
    "#### Assumptions - (approx) normal distributions, independent samples, randomly sampled, equal variances\n",
    "#### What to do if these assumptions aren't met?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(\"https://pingouin-stats.org/_images/flowchart_one_way_ANOVA.svg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot(data=penguins, x= \"species\", y = \"flipper_length_mm\", orient=\"v\", kind=\"box\", height=6, aspect=0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pg.homoscedasticity(data=penguins.dropna(), dv='flipper_length_mm', group='species', method = \"bartlett\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pg.anova(data=penguins, dv='flipper_length_mm', between='species', detailed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Null hypothesis rejected - the means of the populations are not equal\n",
    "#### We need a post-hoc test to see which means are not equal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pg.pairwise_tukey(data=penguins, dv='flipper_length_mm', between='species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The multiple comparisons problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - The look-elsewhere effect is a phenomenon in the statistical analysis of scientific experiments where an apparently statistically significant observation may have actually arisen by chance because of the sheer size of the parameter space to be searched (wikipedia)\n",
    " - The more inferences are made, the more likely erroneous inferences are to occur. \n",
    " - Set a stricter significance threshold for individual comparisons, so as to compensate for the number of inferences being made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pvals = [0.04, 0.001, 0.02, 0.009]\n",
    "pg.multicomp(pvals, alpha=0.05, method='holm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final thoughts :\n",
    "\n",
    "### Think about how the data will be analysed in the experimental design phase, not after the experiment is conducted\n",
    "### Look at the assumptions of the test being used (and how strict they are).  Use a more appropriate one if required\n",
    "### Use multiple correction when doing multiple tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe src=\"https://cdn2.hubspot.net/hubfs/4627953/Essential%20Dos%20and%20Donts%20Ebook/GraphPad%20Ebook%20%7C%20Essential%20Dos%20Don%27ts.pdf?hsCtaTracking=5c7f8486-7fbb-4d16-a068-ad03d1b3af54%7C759129a2-44d6-48d7-b81c-6cb28641839e\" width=\"1000\" height=\"800\"></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<iframe src=\"https://cdn2.hubspot.net/hubfs/4627953/Essential%20Dos%20and%20Donts%20Ebook/GraphPad%20Ebook%20%7C%20Essential%20Dos%20Don%27ts.pdf?hsCtaTracking=5c7f8486-7fbb-4d16-a068-ad03d1b3af54%7C759129a2-44d6-48d7-b81c-6cb28641839e\" width=\"1000\" height=\"800\"></iframe>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}