Assignment_week_04b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignment 04b: Sleeping habits\n",
    "\n",
    "As a data scientist you will work with ordinal or categorical data as well. What are good methods to visualize such data? What are proper statistics to use? What needs to be done in the data preparation steps to conduct certain analysis. During this week we will work with data from a sleeping study. This dataset is not mandatory. You are encouraged to use data from your project when possible. \n",
    "\n",
    "Keywords: statistics, categorical data, ordinal data, survey-based study of the sleeping, exploratory data analysis, normalization, hypothesis testing, p-value\n",
    "\n",
    "\n",
    "More to read:\n",
    "- https://fennaf.gitbook.io/bfvm22prog1/\n",
    "\n",
    "https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/\n",
    "There are a number of cheatsheets and tutorials on the internet. The next overview is a compact overview of tutorials\n",
    "- https://www.kaggle.com/hamelg/python-for-data-21-descriptive-statistics\n",
    "- https://www.kaggle.com/hamelg/python-for-data-22-probability-distributions\n",
    "- https://www.kaggle.com/hamelg/python-for-data-23-confidence-intervals\n",
    "- https://www.kaggle.com/hamelg/python-for-data-24-hypothesis-testing\n",
    "- https://www.kaggle.com/hamelg/python-for-data-25-chi-squared-tests\n",
    "- https://www.kaggle.com/hamelg/python-for-data-26-anova/notebook\n",
    "- https://www.kaggle.com/hamelg/python-for-data-27-linear-regression\n",
    "\n",
    "\n",
    "You will learn about analysing data with pandas and numpy and you will learn to visualize with bokeh. Concretely, you will preprocess the Sleep Study data in an appropiate format in order to conduct statistical and visual analysis. \n",
    "\n",
    "Learning objectives\n",
    "- Conduct exploratory data analysis for categarical data\n",
    "- Provide statistical evidence of factor effect\n",
    "- Maintain development environment \n",
    "- Apply coding standards and FAIR principles\n",
    "\n",
    "Please add topics you want to learn here: https://padlet.com/ffeenstra1/z9duo25d39dcgezz\n",
    "\n",
    "\n",
    "## About the data\n",
    "\n",
    "The data is collected from a survey-based study of the sleeping habits of individuals within the US. \n",
    "\n",
    "Below is a description of each of the variables contained within the dataset.\n",
    "\n",
    "- Enough = Do you think that you get enough sleep?\n",
    "- Hours = On average, how many hours of sleep do you get on a weeknight?\n",
    "- PhoneReach = Do you sleep with your phone within arms reach?\n",
    "- PhoneTime = Do you use your phone within 30 minutes of falling asleep?\n",
    "- Tired = On a scale from 1 to 5, how tired are you throughout the day? (1 being not tired, 5 being very tired)\n",
    "- Breakfast = Do you typically eat breakfast?\n",
    "\n",
    "The two research questions you should answer in this assignment are:\n",
    "1. Is there a differences in Hours sleep caused by having breakfast (yes, no)?\n",
    "2. Is there a differences in Hours sleep caused by having breakfast and the tireness (score)\n",
    "\n",
    "\n",
    "The assignment consists of 6 parts:\n",
    "\n",
    "- [part 1: load the data](#0)\n",
    "- [part 2: data inspection](#1)\n",
    "- [part 3: check assumptions](#2)\n",
    "   - [check normality 3.1](#ex-31)\n",
    "   - [check equal variance 3.2](#ex-32)\n",
    "- [part 4: prepare the data](#3)\n",
    "- [part 5: answer the research question](#4)\n",
    "- [part 6: enhanced plotting](#5)\n",
    "\n",
    "Part 1 till 5 are mandatory, part 6 is optional (bonus)\n",
    "Mind you that you cannot copy code without referencing the code. If you copy code you need to be able to explain your code verbally and you will not get the full score. \n",
    "\n",
    "\n",
    "**NOTE If your project data is suitable you can use that data instead of the given data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ANOVA\n",
    "\n",
    "Analysis of variance (ANOVA) compares the variances between groups versus within groups. It basically determines whether the differences between groups is larger than the differences within a group (the noise). \n",
    "A graph picturing this is as follow: https://link.springer.com/article/10.1007/s00424-019-02300-4/figures/2\n",
    "\n",
    "\n",
    "In ANOVA, the dependent variable must be a continuous (interval or ratio) level of measurement. For instance Glucose level. The independent variables in ANOVA must be categorical (nominal or ordinal) variables. For instance trial category, time of day (AM versus PM) or time of trial (different categories). Like the t-test, ANOVA is also a parametric test and has some assumptions. ANOVA assumes that the data is normally distributed.  The ANOVA also assumes homogeneity of variance, which means that the variance among the groups should be approximately equal. ANOVA also assumes that the observations are independent of each other. \n",
    "\n",
    "A one-way ANOVA has just one independent variable. A two-way ANOVA (are also called factorial ANOVA) refers to an ANOVA using two independent variables. For research question 1 we can use the one-way ANOVA, for research question two we can use two-way ANOVA. But first we need to check the assumptions. \n",
    "\n",
    "If your data is not normally distributed you might want to look for an alternative. See also https://www.statisticshowto.com/probability-and-statistics/statistics-definitions/parametric-and-non-parametric-data/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Part 1: Load the data (10 pt)\n",
    "\n",
    "load the `sleep.csv` data. \n",
    "Preferably we read the data not with a hard coded data path but using a config file. See https://fennaf.gitbook.io/bfvm22prog1/data-processing/configuration-files/yaml. Get yourself familiar with the data. Answer the following questions.\n",
    "\n",
    "1. What is the percentage missing data?\n",
    "2. Considering the research question, what is the dependent variable and what are the indepent variables? Are they of the correct datatype? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code printing percentage missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code printing answer dependent and independent variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code printing answer about datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## Part 2: Inspect the data (30 pt)\n",
    "\n",
    "Inspect the data practically. Get an idea about how well the variable categories are ballanced. Are the values of a variable equally divided? What is the mean value of the dependent variable? Are there correlations amongs the variables?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ul>\n",
    "<li>Create some meaninful overviews such as variable value counts</li>\n",
    "<li>Create a scatter plot ploting the relation between being tired and hours of sleep with different colors for Breakfast</li>\n",
    "    <li>Print some basic statistics about the target (mean, standard deviation)</li>\n",
    "    <li>Create a heatmap to check for correlations among variables. </li>\n",
    "\n",
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "    <ul><li>the gitbook has a bokeh heatmap example</li></ul>\n",
    "</details>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code your answer to the value counts and distribution plots here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code for the scatter plot here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code your answer to the target statistics here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code your answer for the heatmap here and briefly state your finding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## Part 3: Check Assumptions\n",
    "\n",
    "Before we answer the research question with ANOVA we need to check the following assumptions:\n",
    "\n",
    "1. ANOVA assumes that the dependent variable is normaly distributed\n",
    "2. ANOVA also assumes homogeneity of variance\n",
    "3. ANOVA also assumes that the observations are independent of each other. Most of the time we need domain knowledge and experiment setup descriptions to estimate this assumption\n",
    "\n",
    "We are going to do this graphically and statistically. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-31'></a>\n",
    "### Check normality (10 pt)\n",
    "\n",
    "<ul><li>\n",
    "Plot the distribution of the dependent variable. Add a vertical line at the position of the average. Add a vertical line for the robuust estimation. Add the normal distribution line to the plot. Comment on the normallity of the data. Do you want the full points? Plot with bokeh!</li>\n",
    "\n",
    "<li>Use a Shapiro-Wilk Test or an Anderson-Darling test to check statistically</li></ul>\n",
    "\n",
    "\n",
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "    <ul><li>check the code of lesson 1 DS1 bayesian statistics</li>\n",
    "        <li>heart_failure case of gitbook uses bokeh histograms</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code to plot here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# briefly summarize your findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-32'></a>\n",
    "### Check homogeneity of variance (20 pt)\n",
    "\n",
    "<ul><li>\n",
    "Use boxplots for the check of homoegeneity of variance. Do you want the full points? Plot with bokeh!</li>\n",
    "\n",
    "<li>Use a Levene’s & Bartlett’s Test of Equality (Homogeneity) of Variance to test equal variance statistically</li><ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code to plot here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code for the statistical test here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# briefly summarize your findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## Part 4: Prepare your data (10 pt)\n",
    "\n",
    "Create a dataframe with equal samplesize. Make three categories for tireness 1-2 = no, 3 = maybe, 4-5 = yes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4'></a>\n",
    "## Part 5: Answer the research questions (20 pt)\n",
    "\n",
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "    <ul><li>use one-way ANOVA for research question 1</li>\n",
    "    <li>Use two-way ANOVA for research question 2</li>\n",
    "    <li>https://reneshbedre.github.io/blog/anova.html</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='5'></a>\n",
    "## Part 6: Enhanced plotting (20 pt)\n",
    "\n",
    "Create a panel with 1) your dataframe with equal samplesize 2) a picture of a sleeping beauty, 3) the scatter plot of tired / hours of sleep with different colors for Breakfast from part 2 4) the boxplots given the p-value for the anova outcome in the title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your solution here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}