stat functions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_PDF(rv, x=4):\n",
    "    '''Input: a random variable object, standard deviation\n",
    "       output : x and y values for the normal distribution\n",
    "       '''\n",
    "    \n",
    "    # Identify the mean and standard deviation of random variable \n",
    "    mean = rv.mean()\n",
    "    std = rv.std()\n",
    "\n",
    "    # Use numpy to calculate evenly spaced numbers over the specified interval (4 sd) and generate 100 samples.\n",
    "    xs = np.linspace(mean - x*std, mean + x*std, 100)\n",
    "    \n",
    "    # Calculate the peak of normal distribution i.e. probability density. \n",
    "    ys = rv.pdf(xs)\n",
    "\n",
    "    return xs, ys # Return calculated values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cohen's  d\n",
    "Cohen’s D is one of the most common ways to measure effect size. As an effect size, Cohen's d is typically used to represent the magnitude of differences between two (or more) groups on a given variable, with larger values representing a greater differentiation between the two groups on that variable.\n",
    "\n",
    "The basic formula to calculate Cohen’s  dd  is:\n",
    "\n",
    "dd  = effect size (difference of means) / pooled standard deviation\n",
    "\n",
    "The denominator is the standardiser, and it is important to select the most appropriate one for a given dataset. The pooled standard deviation is the average spread of all data points around their group mean (not the overall mean).\n",
    "\n",
    "\n",
    "effect_size = mean_difference / sd\n",
    "\n",
    "\n",
    "\n",
    "# Interpreting  d\n",
    "Most people don't have a good sense of how big  d=2.0d=2.0  is. If you are having trouble visualizing what the result of Cohen’s D means, use these general “rule of thumb” guidelines (which Cohen said should be used cautiously):\n",
    "\n",
    "Small effect = 0.2\n",
    "\n",
    "Medium Effect = 0.5\n",
    "\n",
    "Large Effect = 0.8\n",
    "\n",
    "Here is an excellent online visualization tool developed by Kristoffer Magnusson to help interpret the results of cohen's  dd statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Cohen_d(group1, group2):\n",
    "\n",
    "    # Compute Cohen's d.\n",
    "\n",
    "    # group1: Series or NumPy array\n",
    "    # group2: Series or NumPy array\n",
    "\n",
    "    # returns a floating point number \n",
    "\n",
    "    diff = group1.mean() - group2.mean()\n",
    "\n",
    "    n1, n2 = len(group1), len(group2)\n",
    "    var1 = group1.var()\n",
    "    var2 = group2.var()\n",
    "\n",
    "    # Calculate the pooled threshold as shown earlier\n",
    "    pooled_var = (n1 * var1 + n2 * var2) / (n1 + n2)\n",
    "    \n",
    "    # Calculate Cohen's d statistic\n",
    "    d = diff / np.sqrt(pooled_var)\n",
    "    \n",
    "    return d\n",
    "\n",
    "\n",
    "def plot_pdfs(cohen_d=2):\n",
    "    \"\"\"Plot PDFs for distributions that differ by some number of stds.\n",
    "    \n",
    "    cohen_d: number of standard deviations between the means\n",
    "    \"\"\"\n",
    "    group1 = scipy.stats.norm(0, 1)\n",
    "    group2 = scipy.stats.norm(cohen_d, 1)\n",
    "    xs, ys = evaluate_PDF(group1)\n",
    "    plt.fill_between(xs, ys, label='Group1', color='#ff2289', alpha=0.7)\n",
    "\n",
    "    xs, ys = evaluate_PDF(group2)\n",
    "    plt.fill_between(xs, ys, label='Group2', color='#376cb0', alpha=0.7)\n",
    "    \n",
    "    o, s = overlap_superiority(group1, group2)\n",
    "    print('overlap', o)\n",
    "    print('superiority', s)\n",
    "    \n",
    "    \n",
    "from scipy import stats\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "def one_sample_ttest(sample, popmean, alpha):\n",
    "\n",
    "    # Visualize sample distribution for normality \n",
    "    sns.set(color_codes=True)\n",
    "    sns.set(rc={'figure.figsize':(12,10)})\n",
    "    sns.distplot(sample)\n",
    "    \n",
    "    # Population mean \n",
    "    mu = popmean\n",
    "    \n",
    "    # Sample mean (x̄) using NumPy mean()\n",
    "    x_bar= sample.mean()\n",
    "\n",
    "    # Sample Standard Deviation (sigma) using Numpy\n",
    "    sigma = np.std(sample)\n",
    "    \n",
    "    # Degrees of freedom\n",
    "    df = len(sample) - 1\n",
    "    \n",
    "    #Calculate the critical t-value\n",
    "    t_crit = stats.t.ppf(1 - alpha, df=df)\n",
    "    \n",
    "    #Calculate the t-value and p-value\n",
    "    results = stats.ttest_1samp(a= sample, popmean= mu)         \n",
    "    \n",
    "    if (results[0]>t_crit) and (results[1]<alpha):\n",
    "        print (\"Null hypothesis rejected. Results are statistically significant with t-value =\", \n",
    "                round(results[0], 2), \"critical t-value =\", t_crit, \"and p-value =\", np.round((results[1]), 10))\n",
    "    else:\n",
    "        print (\"Null hypothesis is True with t-value =\", \n",
    "                round(results[0], 2), \", critical t-value =\", t_crit, \"and p-value =\", np.round((results[1]), 10))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Two Sampled T-Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#It is always a good idea to draw the probability distributions for samples to visually inspect the differences \n",
    "#present between mean and standard deviation. Plot both samples' distributions and inspect the overlap using seaborn \n",
    "#to get an idea how different the samples might be from one another.\n",
    "\n",
    "sns.set(color_codes=True)\n",
    "sns.set(rc={'figure.figsize':(12,10)})\n",
    "sns.distplot(control) # Blue distribution\n",
    "sns.distplot(experimental) # Green distribution\n",
    "\n",
    "\n",
    "\"\"\"As a reminder the five steps to performing a hypothesis test are:\n",
    "\n",
    "Set up null and alternative hypotheses\n",
    "Choose a significance level\n",
    "Calculate the test statistic\n",
    "Determine the critical or p-value (find the rejection region)\n",
    "Compare t-value with critical t-value to reject or fail to reject the null hypothesis\"\"\"\n",
    "\n",
    "\"\"\"Now, create some functions to calculate the t-statistic. The first function to create is one \n",
    "that calculates the variance for a single sample.\"\"\"\n",
    "\n",
    "def sample_variance(sample):\n",
    "    sample_mean = np.mean(sample)\n",
    "    return np.sum((sample - sample_mean) **2)/ (len(sample) -1)\n",
    "    \n",
    "#Using sample_variance, you can now write another function pooled_variance to calculate $S_{p}^{2}$\n",
    "\n",
    "def pooled_variance(sample1, sample2):\n",
    "    n_1, n_2 = len(sample1), len(sample2)\n",
    "    var_1, var_2 = sample_variance(sample1), sample_variance(sample2)\n",
    "    return ((n_1-1) * var_1 + (n_2-1)* var_2)/((n_1 + n_2)-2)\n",
    "\n",
    "def twosample_tstatistic(expr, ctrl):\n",
    "    exp_mean, ctrl_mean = np.mean(expr), np.mean(ctrl)\n",
    "    pool_var = pooled_variance(expr, ctrl)\n",
    "    n_e, n_c = len(expr), len(ctrl)\n",
    "    num = exp_mean - ctrl_mean\n",
    "    denom = np.sqrt(pool_var * ((1/n_e)+(1/n_c)))\n",
    "    return num / denom\n",
    "\n",
    "#Write a function visualize_t that uses matplotlib to display a standard t-distribution with vertical lines identifying each critical value that signifies the rejection region.\n",
    "\n",
    "# Visualize p_value\n",
    "\n",
    "def visualize_t(t_stat, n_control, n_experimental):\n",
    "    \n",
    "    \"\"\"\n",
    "    Visualize the critical t values on a t distribution\n",
    "    \n",
    "    Parameters\n",
    "    -----------\n",
    "    t-stat: float\n",
    "    n_control: int\n",
    "    n_experiment: int\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    None\n",
    "    \n",
    "    \"\"\"\n",
    "\n",
    "    # initialize a matplotlib \"figure\"\n",
    "    fig = plt.figure(figsize=(8,5))\n",
    "    ax = fig.gca()\n",
    "    # generate points on the x axis between -4 and 4:\n",
    "    xs = np.linspace(-4, 4, 500)\n",
    "\n",
    "    # use stats.t.pdf to get values on the probability density function for the t-distribution\n",
    "    \n",
    "    ys= stats.t.pdf(xs, (n_control+n_experimental-2), 0, 1)\n",
    "    ax.plot(xs, ys, linewidth=3, color='darkred')\n",
    "\n",
    "    ax.axvline(t_stat, color='black', linestyle='--', lw=5)\n",
    "    ax.axvline(-t_stat, color='black', linestyle='--', lw=5)\n",
    "\n",
    "    plt.show()\n",
    "    return None\n",
    "\n",
    "#n_control = len(control)\n",
    "#n_experimental = len(experimental)\n",
    "#visualize_t(t_stat, n_control, n_experimental)\n",
    "\n",
    "\"\"\"Given a t-value and a degrees of freedom, you can use the \"survival function\" sf of scipy.stats.t \n",
    "(aka the complementary CDF) to compute the one-sided p-value. For the two-sided p-value, \n",
    "just double the one-sided p-value.\n",
    "\"\"\"\n",
    "## Calculate p_value\n",
    "# Lower tail comulative density function returns area under the lower tail curve\n",
    "lower_tail = stats.t.cdf(-1.89, (50+50-2), 0, 1)\n",
    "# Upper tail comulative density function returns area under upper tail curve\n",
    "upper_tail = 1. - stats.t.cdf(1.89, (50+50-2), 0, 1)\n",
    "\n",
    "p_value = lower_tail+upper_tail\n",
    "print(p_value)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TYPE 1 Error and Type 2 Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def type_1_error(population, num_tests, alpha_set):\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    population: ndarray\n",
    "        A random normal distribution\n",
    "    num_tests: int\n",
    "        The number of hypothesis tests to be computed\n",
    "    alpha_set: list\n",
    "        List of alpha levels\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    sig_tests : DataFrame\n",
    "        A dataframe containing the columns 'type_1_error', 'p_value', and 'alpha'\n",
    "    \"\"\"\n",
    "    columns = ['type_1_error','p_value','alpha']\n",
    "    sig_tests = pd.DataFrame(columns=columns)\n",
    "    counter = 0\n",
    "    \n",
    "    for i in range(1,num_tests+1):\n",
    "        \n",
    "        for alpha in alpha_set:\n",
    "            \n",
    "            # take two samples from the same population\n",
    "            samp1 = np.random.choice(population,100,replace=True)\n",
    "            samp2 = np.random.choice(population,100,replace=True)\n",
    "            \n",
    "            # test sample means\n",
    "            result = stats.ttest_ind(samp1, samp2)\n",
    "            \n",
    "            # evaluate whether null hypothesis is rejected or not\n",
    "            if result[1] < alpha:\n",
    "                 sig_tests.loc[counter] = [1, result[1], alpha]\n",
    "            else:\n",
    "                 sig_tests.loc[counter] = [0, result[1], alpha]\n",
    "\n",
    "            counter += 1\n",
    "            \n",
    "    return sig_tests\n",
    "\n",
    "\n",
    "def type_2_error(population, population_2, num_tests, alpha_set):\n",
    "    \n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    population: ndarray\n",
    "        A random normal distribution\n",
    "    population_2: ndarray\n",
    "        A different random normal distribution\n",
    "    num_tests: int\n",
    "        The number of hypothesis tests to be computed\n",
    "    alpha_set: list\n",
    "        List of alpha levels\n",
    "    \n",
    "    Returns\n",
    "    ----------\n",
    "    sig_tests : DataFrame\n",
    "        A dataframe containing the columns 'type_2_error', 'p_value', and 'alpha'\n",
    "    \"\"\"\n",
    "    \n",
    "    columns = ['type_2_error','p_val','alpha']\n",
    "    sig_tests = pd.DataFrame(columns=columns)\n",
    "    counter = 0\n",
    "    \n",
    "    for i in range(1,num_tests+1):\n",
    "        \n",
    "        for alpha in alpha_set:\n",
    "            \n",
    "            # take two samples from the same population\n",
    "            samp1 = np.random.choice(population,100,replace=True)\n",
    "            samp2 = np.random.choice(population_2,100,replace=True)\n",
    "            \n",
    "            # test sample means\n",
    "            result = stats.ttest_ind(samp1, samp2)\n",
    "            \n",
    "            # evaluate whether null hypothesis is rejected or not\n",
    "            if result[1] > alpha:\n",
    "                 sig_tests.loc[counter] = [1, result[1], alpha]\n",
    "            else:\n",
    "                 sig_tests.loc[counter] = [0, result[1], alpha]\n",
    "\n",
    "            counter += 1\n",
    "            \n",
    "    return sig_tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you decide to use a large value for alpha :\n",
    "\n",
    "Increases the chance of rejecting the null hypothesis\n",
    "The risk of a Type II error (false negative) is REDUCED\n",
    "Risk of a Type I error (false positive) is INCREASED\n",
    "Similarly, if you decide to use a very small value of alpha, it'll change the outcome as:\n",
    "\n",
    "Increases the chance of accepting the null hypothesis\n",
    "The risk of a Type I error (false positive) is REDUCED\n",
    "Risk of a Type II error (false negative) is INCREASED\n",
    "From above, you can see that in statistical hypothesis testing, the more you try and avoid a Type I error (false positive), the more likely a Type II error (false negative) will occur."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Key Takeaways\n",
    "Some of the key takeaways from this section include:\n",
    "\n",
    "It's important to have a sound approach to experimental design to be able to determine the significance of your findings\n",
    "Start by examining any existing research to see if it can shed light on the problem you're studying\n",
    "Start with a clear alternative and null hypothesis for your experiment to \"prove\"\n",
    "It's important to have a thoughtfully selected control group from the same population for your trial to distinguish effect from variations based on population, time or other factors\n",
    "Sample size needs to be selected carefully to ensure your results have a good chance of being statistically significant\n",
    "Your results should be reproducible by other people and using different samples from the population\n",
    "The p-value for an outcome determines how likely it is that the outcome could be due to chance\n",
    "The alpha value is the marginal threshold at which we're comfortable rejecting the null hypothesis\n",
    "An alpha of 0.05 is a common choice for many experiments\n",
    "Effect size measures just the size in difference between two groups under observation, whereas statistical significance combines effect size with sample size\n",
    "A one sample t-test is used to determine whether a sample comes from a population with a specific mean.\n",
    "A two-sample t-test is used to determine if two population means are equal\n",
    "Type 1 errors (false positives) are when we accept an alternative hypothesis which is actually false\n",
    "The alpha that we pick is the likelihood that we will get a type 1 error due to random chance\n",
    "Type 2 errors (false negatives) are when we reject an alternative hypothesis which is actually true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical Power\n",
    "The power of a statistical test is defined as the probability of rejecting the null hypothesis, given that it is indeed false."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Statsmodels has some convenient build in methods for calculating the power of a t-test and plotting power curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.stats.power import TTestIndPower, TTestPower\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "sns.set_style('darkgrid') #Nice background styling on plots\n",
    "\n",
    "power_analysis = TTestIndPower()\n",
    "\n",
    "power_analysis.plot_power(dep_var=\"nobs\",\n",
    "                          nobs = np.array(range(5,1500)),\n",
    "                          effect_size=np.array([.05, .1, .2,.3,.4,.5]),\n",
    "                          alpha=0.05)\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to plotting a full curve, you can also calculate specific values. Simply don't specify one of the four parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calculate power\n",
    "power_analysis.solve_power(effect_size=.2, nobs1=80, alpha=.05)\n",
    "\n",
    "#Calculate sample size required\n",
    "power_analysis.solve_power(effect_size=.2, alpha=.05, power=.8)\n",
    "\n",
    "#Calculate minimum effect size to satisfy desired alpha and power as well as respect sample size limitations\n",
    "power_analysis.solve_power(nobs1=25, alpha=.05, power=.8)\n",
    "\n",
    "#Calculate alpha (less traditional)\n",
    "power_analysis.solve_power(nobs1=25, effect_size=.3, power=.8)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welches T-Test\n",
    "\n",
    "Recall that z-tests are also appropriate for statistics, such as the mean, which can be assumed to be normally distributed. However, when sample sizes are low (n_observations<30), the t-test is more appropriate, as it has heavier tails. Even with this modification, remember that there are still several assumptions to the model. Most notably, traditional t-tests assume that sample sizes and sample variances between the two groups are equal. When these assumptions are not met, Welch's T-test is generally a more reliable test.\n",
    "\n",
    "the student's  tt -test assumes that the samples are of equal size and equal variance ( so , mean as well ). When these assumptions are not met, then Welch's  tt -test provides a more accurate p-value.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def welch_t(a, b):\n",
    "    \n",
    "    \"\"\" Calculate Welch's t statistic for two samples. \"\"\"\n",
    "\n",
    "    numerator = a.mean() - b.mean()\n",
    "    \n",
    "    # “ddof = Delta Degrees of Freedom”: the divisor used in the calculation is N - ddof, \n",
    "    #  where N represents the number of elements. By default ddof is zero.\n",
    "    \n",
    "    denominator = np.sqrt(a.var(ddof=1)/a.size + b.var(ddof=1)/b.size)\n",
    "    \n",
    "    return np.abs(numerator/denominator)\n",
    "\n",
    "\n",
    "def welch_df(a, b):\n",
    "    \n",
    "    \"\"\" Calculate the effective degrees of freedom for two samples. \"\"\"\n",
    "    \n",
    "    s1 = a.var(ddof=1) \n",
    "    s2 = b.var(ddof=1)\n",
    "    n1 = a.size\n",
    "    n2 = b.size\n",
    "    \n",
    "    numerator = (s1/n1 + s2/n2)**2\n",
    "    denominator = (s1/ n1)**2/(n1 - 1) + (s2/ n2)**2/(n2 - 1)\n",
    "    \n",
    "    return numerator/denominator\n",
    "\n",
    "\n",
    "import numpy as np\n",
    "import scipy.stats as stats\n",
    "\n",
    "\n",
    "\n",
    "def p_value_welch_ttest(a, b, two_sided=False):\n",
    "    \"\"\"Calculates the p-value for Welch's t-test given two samples.\n",
    "    By default, the returned p-value is for a one-sided t-test. \n",
    "    Set the two-sided parameter to True if you wish to perform a two-sided t-test instead.\n",
    "    \"\"\"\n",
    "    t = welch_t(a, b)\n",
    "    df = welch_df(a, b)\n",
    "    \n",
    "    p = 1-stats.t.cdf(np.abs(t), df)\n",
    "    \n",
    "    if two_sided:\n",
    "        return 2*p\n",
    "    else:\n",
    "        return p\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ANOVA \n",
    "\n",
    "Loading the Data\n",
    "As usual, we start by loading in a dataset of our sample observations. This particular table is of salaries in IT and has 4 columns:\n",
    "\n",
    "S - the individuals salary\n",
    "\n",
    "X - years of experience\n",
    "\n",
    "E - education level (1-Bachelors, 2-Masters, 3-PHD)\n",
    "\n",
    "M - management (0-no management, 1-yes management)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "formula = 'S ~ C(E) + C(M) + X'\n",
    "lm = ols(formula, df).fit()\n",
    "table = sm.stats.anova_lm(lm, typ=2)\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the Table¶\n",
    "For now , simply focus on the outermost columns. On the left, you can see our various groups, and on the right, the probability that the factor is indeed influential. Values < .05 (or whatever we set  αα  to) indicate rejection of the null hypothesis. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A 2-Category ANOVA F-Test is Equivalent to a 2-Tailed t-Test!\n",
    "\n",
    "Now, recalculate an ANOVA F-test with only the supplement variable. An ANOVA F-test between two categories is the same as performing a 2-tailed t-Test! So, the p-value in the table should be identical to your calculation above.\n",
    "\n",
    "Note: there may be a small fractional difference (>0.001) between the two values due to a rounding error between implementations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Your code here; conduct an ANOVA F-test of the oj and vc supplement groups.\n",
    "#Compare the p-value to that of the t-test above. \n",
    "#They should match (there may be a tiny fractional difference due to rounding errors in varying implementations)\n",
    "formula = 'len ~ C(supp)'\n",
    "lm = ols(formula, df).fit()\n",
    "table = sm.stats.anova_lm(lm, typ=2)\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Multiple T-Tests\n",
    "While the 2-category ANOVA test is identical to a 2-tailed t-Test, performing multiple t-tests leads to the multiple comparisons problem. To investigate this, look at the various sample groups you could create from the 2 features:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "for group in df.groupby(['supp', 'dose'])['len']:\n",
    "    group_name = group[0]\n",
    "    data = group[1]\n",
    "    print(group_name)\n",
    "('OJ', 0.5)\n",
    "('OJ', 1.0)\n",
    "('OJ', 2.0)\n",
    "('VC', 0.5)\n",
    "('VC', 1.0)\n",
    "('VC', 2.0)\n",
    "\n",
    "\n",
    "\"\"\"While bad practice, examine the effects of calculating multiple t-tests with the various\n",
    "combinations of these. To do this, generate all combinations of the above groups. \n",
    "For each pairwise combination, calculate the p-value of a 2 sided t-test. \n",
    "Print the group combinations and their associated p-value for the two-sided t-test.\n",
    "\"\"\"\n",
    "#Your code here; reuse your t-test code above to calculate the p-value for a 2-sided t-test\n",
    "#for all combinations of the supplement-dose groups listed above. \n",
    "#(Since there isn't a control group, compare each group to every other group.)\n",
    "\n",
    "from itertools import combinations\n",
    "\n",
    "groups = [group[0] for group in df.groupby(['supp', 'dose'])['len']]\n",
    "combos = combinations(groups, 2)\n",
    "for combo in combos:\n",
    "    supp1 = combo[0][0]\n",
    "    dose1 = combo[0][1]\n",
    "    supp2 = combo[1][0]\n",
    "    dose2 = combo[1][1]\n",
    "    sample1 = df[(df.supp == supp1) & (df.dose == dose1)]['len']\n",
    "    sample2 = df[(df.supp == supp2) & (df.dose == dose2)]['len']\n",
    "    p = stats.ttest_ind(sample1, sample2, equal_var=False)[1]\n",
    "    print(combo, p)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Key Takeaways\n",
    "Remember that the section began where the last left off, examining the relationship between  αα , power, effect size and sample size. As you saw, these 4 quantities form a deterministic relationship; know any 3, and you can caulculate the fourth. While a lower alpha value will lead to fewer type I errors, and a higher power will lead to fewer type II errors, in practice these are often set to common default standards due to exploding sample sizes required to detect various effect sizes. Some common thresholds used are:\n",
    "\n",
    "Setting alpha equal to .05 (or .01)\n",
    "Requiring power values of .8 or greater\n",
    "After a thorough investigation of this relationship, you then also saw an alternative t-test, Welch's t-test which can be used for comparing samples of different sizes of with different variances. While the formula was a bit complicated, the most important piece to remember is that when the assumptions that sample size and sample variance are equal for the two samples is violated, use Welch's t-test rather then student's t-test.\n",
    "\n",
    "Aside from ensuring that the assumptions of a t-test are met, it's also important to know how type I errors are compounded if you perform multiple A/B tests. This is known as the multiple comparison problem and you saw that type I errors compound under multiple tests. So while the probability of a type I error is equal to  αα  for any one test, the collective probability that there is at least 1 type I error continues to increase as you perform more tests, further detracting from the confidence that you have uncovered a meaningful relationship. In order to account for this, you can use stricter criteria when defining  αα  such as the Bonferroni Correction. Alternatively, ANOVA is equivalent to a 2 sided t-test when comparing 2 groups, but also generalizes appropriately to multiple group comparisons.\n",
    "\n",
    "Summary\n",
    "Remember that simply observing a low p-value is not meaningful in and of itself. There are a number of factors to take into consideration when interpreting the results of a statistical test, from alpha, power, sample size, effect size, and the formulation of the problem itself. Good hypothesis testing requires careful thought and design."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note that parametric means numeric , non-parametric is non-numeric may be ranks or some sort of value\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}