METER_Sample_Size.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "183f8782-7b20-4c50-9b24-330b84c51dee",
   "metadata": {},
   "source": [
    "# Sample Size - How much data is enough for your experiment?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b976912-2b5d-4ae1-be2a-2f1f40cb1d3e",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>Alert:</b> If you're running this on <b>Google Colab</b>, then uncomment and run the next two cells.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d1c4fa-2969-419e-8d60-1e9075cb3076",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !git clone https://github.com/Mark-Kramer/METER-Units.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8658c043-55e3-4ef9-83e2-078809684b7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import sys\n",
    "# sys.path.insert(0,'/content/METER-Units')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c27513a-502c-42e2-8a0a-a079e474ba3e",
   "metadata": {},
   "source": [
    "## 0 - Setup & Introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ae7402f-3399-4340-a306-0c557f42893d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load modules\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import pearsonr\n",
    "import statsmodels.api as sm\n",
    "# Load custom functions\n",
    "from sample_size_functions import *"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f512e22f-0af3-4e6e-8c7f-d5d4d68263f5",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "*Introduction*:\n",
    "\n",
    "- Imagine we are studying a potential new *genetic biomarker for longevity*, substance $x$. \n",
    "- Our goal is to design and perform an experiment to test the predictive power of this novel biomarker. To do so, we need to compute the **sample size** for our experiment, the number of subjects from whom to collect data, to determine whether the biomarker is useful.\n",
    "- **That's the goal of this unit:** perform a sample size calculation.\n",
    "- To start, we'll provide limited information about substance $x$ and longevity, and ask you to determine the sample size.\n",
    "- **No idea how to start a sample size calculation?** Don't worry! The goal of this unit is to teach you a general approach.\n",
    "- By the end of this unit, you'll have a deeper understanding of what sample size means, and a general, practical approach to compute it.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ff77b0-4c41-477a-be4f-dd5de75d4ff9",
   "metadata": {},
   "source": [
    "## 1 - Just Google it?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8cf43eb9-a571-4a0b-aef3-be10c159c701",
   "metadata": {},
   "source": [
    "- Imagine that we have a scientific hypothesis based on previous work that suggests that substance $x$ is a genetic biomarker for longevity (i.e., age at death). \n",
    "- Before conducting an experiment to test the predictive power of this novel biomarker, we need to compute the **sample size** for our experiment.\n",
    "- So, let's do it.\n",
    "- We will see that the sample size required to generate data that can support a scientific hypothesis depends directly on the prior beliefs and knowledge about that hypothesis.\n",
    "- Imagine that we have the following limited information about substance $x$ and longevity:\n",
    "    - *People have a normal distribution of expression of substance $x$.*\n",
    "    - *Individuals at the high end of expression levels tend to live about 5 years longer than people at the low end.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f39fded2-9a36-466f-8db5-696cea8b430f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Given this information, from how many individuals do we need to collect data to have a reasonable chance of demonstrating this hypothesis is correct? (I.e., What is the **sample size**?)\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ad9dedf5-ea2d-44bf-99be-35ca9e0e6a69",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>Alert: Wait, I have no idea how to answer this?</b>\n",
    "\n",
    "</p>\n",
    "\n",
    "- Don't worry!\n",
    "- The goal of this unit is to teach you to tackle this problem.\n",
    "- Let’s first come up with any approach to compute a sample size, even if we’re not confident in the results.\n",
    "\n",
    "A few possible places to start:\n",
    "\n",
    "- *Take an educated guess*: Perhaps you have taken part in or read about similar research before. What order of magnitude seems right for this sort of experiment?\n",
    "- *Find a source*: Sample size estimation is a common topic in introductory statistics textbooks. These often include formulas that students can use to compute sample size for specific categories of questions.\n",
    "- *Google it*: : There are many web-based resources (including online calculators) that are designed to enable sample size calculations. Search engines provide a starting point for finding such resource Doing so, you might end up at a website [like this](https://researchmethodsresources.nih.gov/grt-calculator) or [like this](https://www.abs.gov.au/websitedbs/d3310114.nsf/home/sample+size+calculator).\n",
    "\n",
    "Can you estimate the sample size using any of these approaches? What obstacles did you encounter along the way. (NOTE: This isn’t always easy or obvious, even for veteran researchers!! \n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e934f9a5-daba-4e5d-96e5-ea50c1a7a20a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Given the description of the scientific hypothesis and experiment, think about what data you would collect and what analyses you would perform to test the hypothesis.\n",
    "\n",
    "- What types of values do you expect for each variable? What are their distributions, do you think?\r",
    "- How do you expect the variables to be related?\r",
    "- Try drawing a sketch of what you imagine a successful result might look like?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adad8561-5a97-4175-921b-1eef24c6f18e",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** We provided very little information and asked you to compute the sample size. What other information do you think would be helpful to estimate the sample size?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9712a9-ae79-4450-9f59-68966a7e2fad",
   "metadata": {},
   "source": [
    "## 2- Underpowered experiments are doomed to failure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c0bb810-3a0e-4927-afe6-2439f483de9e",
   "metadata": {},
   "source": [
    "Now that you've determined (or guessed) the sample size `N` for your experiment, let's perform the experiment.\n",
    "\n",
    "You collect `N` samples of data, so that you receive from each individual:\n",
    "\n",
    "* `x` - a measure of the proposed biomarker for longevity,\n",
    "\n",
    "* `lifespan` - the individual's age at death.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99e47f6b-8302-4529-bc03-a8ffb9898eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 100                               # Here, learner will input N they found in Mini 1.\n",
    "x,lifespan = load_data(N)             # Use this by default.\n",
    "# x,lifespan = load_data_Colab(N)     # Use this if on !!GOOGLE COLAB!!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a0628f-693e-43b4-ae58-32acd1b18624",
   "metadata": {},
   "source": [
    "Let's start by plotting the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fbee484-9a1a-4c45-8ab0-9d8850b8ce09",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x,lifespan)\n",
    "plt.xlabel('Genetic biomarker x')\n",
    "plt.ylabel('Lifespan (years)');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9855a6b9-dc64-415c-855e-96209ee2bbb5",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What do you observe? Does the hypothesized relationship between the biomarker $x$ and lifespan appear present in the data?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab1e61e0-832d-4dd3-b169-76721af901f8",
   "metadata": {},
   "source": [
    "Let's assess the relationship between the biomarker `x` and `lifespan` beyond visual inspection.\n",
    "\n",
    "There are many ways to do so.\n",
    "\n",
    "Here, we'll fit a line to the data and compute the slope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "820dd5bd-b6d2-4d9e-bbb1-e11f8a1dcc8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Estimate a line from the data.\n",
    "\n",
    "from statsmodels.formula.api import ols\n",
    "\n",
    "dat                = {\"x\": x, \"lifespan\": lifespan}\n",
    "regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c22000f-d414-4617-ba28-da3fa045353a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** If this code is new to you, don't worry. Can you see the equation for the line in the code above?\n",
    "\n",
    "**A:**\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f09478d-a504-4492-8189-73f8e1f56218",
   "metadata": {},
   "source": [
    "Now, with the line estimated, we can print the estimated slope and its p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d2d9850-b0a3-40e4-ac13-1ed19ca7f0c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Slope estimate =',regression_results.params[1])\n",
    "print('p-value        =',regression_results.pvalues[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5566da0f-8cce-450c-8511-d6aa313a77e4",
   "metadata": {},
   "source": [
    "** Interpert slope, +/- 2 SD of slope."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2909e622-5e33-46b7-960a-a179742fc1dd",
   "metadata": {},
   "source": [
    "Let's also **visualize** the estimated line by plotting it with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa632f47-a947-43fb-835a-de8f055e2ab8",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pred   = regression_results.get_prediction().summary_frame()\n",
    "mn     = pred['mean']\n",
    "ci_low = pred['mean_ci_lower'] \n",
    "ci_upp = pred['mean_ci_upper']\n",
    "\n",
    "# And plot it.\n",
    "indices_sorted = np.argsort(x,0)\n",
    "plt.scatter(x,lifespan)\n",
    "plt.plot(x[indices_sorted[:,0]],mn[indices_sorted[:,0]], 'r')\n",
    "plt.plot(x[indices_sorted[:,0]],ci_low[indices_sorted[:,0]], ':r')\n",
    "plt.plot(x[indices_sorted[:,0]],ci_upp[indices_sorted[:,0]], ':r')\n",
    "plt.xlabel('Genetic biomarker x')\n",
    "plt.ylabel('Lifespan (years)');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "454310ca-3aee-4aff-9b5d-50937f094df3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Do you find a significant relationship between the genetic biomarker `x` and `lifespan`?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80d929d6-99ab-4e69-8fdb-6a5b011d23e1",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>Alert: Wait, this doesn't make sense!</b>\n",
    "\n",
    "</p>\n",
    "\n",
    "- We've applied a standard approach to compute sample size `N` and performed the experiment using this sample size.\n",
    "\n",
    "- We see a trend supporting the hypothesized relationship, but it's not significant.\n",
    "\n",
    "- Has the experiment failed?\n",
    "\n",
    "What's going on?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ccac4e8-fb87-4def-ba04-e6532e450a2f",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "*Moment of tension*:\n",
    "\n",
    "- Hook the learner - \"something isn't right and I want to know why.\"\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0964c13-f65c-4d5d-ac56-3bdfa56e1ae0",
   "metadata": {},
   "source": [
    "## 3- What is the effect size?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f0fbb8-8057-46df-9e75-06dbc437ecec",
   "metadata": {},
   "source": [
    "In Minis 1 & 2, we determined a sample size `N`, collected data with that sample size, and tested for a relationship between longevity and biomarker $x$.\n",
    "\n",
    "Our results failed to support the hypothesis with sufficient statistical significance!\n",
    "\n",
    "Despite this failure, these data can still be useful.\n",
    "\n",
    "In this Mini, we'll see how to use these data to estimate the **effect size**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdee27a8-a8e7-4e7e-b8bb-8664df42bd66",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What is the effect size for the data you analyzed in Mini 2?\n",
    "- Come up with a numerical measure that describes how meaningful the relationship between substance $x$ and lifespan is. What is the value of this measure for the data you obtained in Unit 2?\n",
    "- What does this effect size mean in words?\r",
    "- Is it consistent with your hypothesis?\r",
    "- If this effect size were correct, would it be scientifically meaningful? If so, why is the effect not statistically significant?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9957035-f702-4632-8e5c-24527bd412b6",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Imagine you repeated the experiment. Would you expect to see the same effect size?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6cf4ddc-77bb-42bc-8c41-17d1fabebbec",
   "metadata": {},
   "source": [
    "## 4- With resampling you can repeat any experiment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a18726-c789-43ff-9168-cf8829d2d904",
   "metadata": {},
   "source": [
    "- The data provided in Mini 2 represent one instantiation of the experiment.\n",
    "- Maybe we were unlucky and repeating the experiment with sample size `N` would produce significant results.\n",
    "- But, repeating the experiment is expensive.\n",
    "- An alternative is to resample your data in hand.\n",
    "\n",
    "In this Mini, we'll implement a resampling procedure (with fixed sample size `N`) and examine variability in the effect size. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d8d692-4e5c-4289-b604-1f6769ea5e9f",
   "metadata": {},
   "source": [
    "Our resampling procedure consists of 3 steps:\n",
    "\n",
    "1) Draw a new (random) set of labels we can use to index our data (biomarker $x$ and lifespan).\n",
    "2) Use these indices to create a resampled data set.\n",
    "3) Compute the relationship (and its statistical significance) in our resampled data (i.e., between the biomarker $x$ and lifespan).\n",
    "\n",
    "We'll now describe each step. For a related example, [see this video](https://youtu.be/mqDEJyW_z4c?si=heigY8z5PqAjnwKZ)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc056da-ec81-4893-9d07-aa69ac3698ec",
   "metadata": {},
   "source": [
    "**First**, we need to draw a set of labels we can use to index our data.\n",
    "\n",
    "To do so, we'll create a random list of indices, of size equal to the length of our data.\n",
    "\n",
    "We'll do so *with replacement*, so that the same index may be listed once, twice, or more, or not at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f83ed80-3448-4187-b74b-7f566dac587e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ind = np.random.choice(np.size(x), np.size(x))\n",
    "print(ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d822ceb-678f-495c-8a43-77f0d4905d03",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Look at the values in `ind`. What do they mean?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403535ef-d037-4e42-942e-b17102f3d481",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Run the code to generate `ind` again. What do you find? (I.e., is it the same or different than the first time?)\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd291613-73bb-42ac-8483-0c1bb077e2fa",
   "metadata": {},
   "source": [
    "**Second**, we'll used these indices to generate the resampled data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27c0df42-1d27-444c-a99b-9a060044e3cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_resampled = x[ind]\n",
    "lifespan_resampled = lifespan[ind]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d49311e6-9aaf-4179-9d5d-2121355d11cc",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Look at the values in `x_resampled` and `lifespan_resampled`. Do they make sense?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c881df81-4f0c-4597-a709-a370cfae4324",
   "metadata": {},
   "source": [
    "**Third**, we determine the relationship between the resampled biomarker $x$ and the resampled lifespan.\n",
    "\n",
    "To do so, we'll fit the same linear model to our new resampled data, and again compute the slope and significance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d32e4b19-83ae-423b-8d28-4a564a70cde0",
   "metadata": {},
   "outputs": [],
   "source": [
    "dat                = {\"x\": x_resampled, \"lifespan\": lifespan_resampled}\n",
    "regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "\n",
    "print('Slope estimate =',regression_results.params[1])\n",
    "print('p-value        =',regression_results.pvalues[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8814f4c9-84e1-41a4-b287-c96d064fdda5",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Repeat these steps to generate results from multiple \"experiments\". Do you ever find a significant result? How often do the p-values you find reach your desired level of statistical significance?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b36be1a-d4ed-443d-a883-7c8bdd24a479",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "*Conclusion*:\n",
    "\n",
    "- Using the sample size `N` and resampling the data we get qualitatively similar - but quantitatively different - results.\n",
    "- The effect size (i.e., slope) is consistently positive across resamples. That's consistent with our hypothesis.\n",
    "- But, we do not (or very infrequently) find a significant relationship between lifespan and $x$.\n",
    "- The significance (p-value) changes for each resample. \n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be0fc241-9c23-4afe-b0c0-0e70ecd37a5f",
   "metadata": {},
   "source": [
    "## 5- You have the power!  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "298ecca4-06c9-4683-8e69-ea704ddd0c9e",
   "metadata": {},
   "source": [
    "Resampling provides a direct, computational approach to compute the **power**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98be7354-5771-4400-880f-b21181be8967",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "Define power.\n",
    "    \n",
    "- It's a mysterious parameter in the online calculator.\n",
    "- What does Power = 0.8 mean?\n",
    "- What does alpha = 0.05 mean?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eb06322-0ec0-426e-83eb-9e92dba806c9",
   "metadata": {},
   "source": [
    "The procedure to compute the power using resampling is simple. Here are the steps in general:\n",
    "\n",
    "1. Resample the data.\n",
    "2. Compute the significance of the proposed effect.\n",
    "3. Repeat 1-2 many times.\n",
    "\n",
    "The power is the number of times we detect a significant effect, divided by the total number of repetitions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d0e580-1fe3-43e5-adfd-40ee1c16973f",
   "metadata": {},
   "source": [
    "Let's apply this procedure to our data and effect of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a3146d1-1410-415c-b9cb-1db7c23caa8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_repetitions = 1000\n",
    "p_value = np.zeros(number_of_repetitions)\n",
    "for k in np.arange(number_of_repetitions):\n",
    "    ind = np.random.choice(np.size(x), np.size(x))\n",
    "    x_resampled = x[ind]\n",
    "    lifespan_resampled = lifespan[ind]\n",
    "    dat                = {\"x\": x_resampled, \"lifespan\": lifespan_resampled}\n",
    "    regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "    p_value[k] = regression_results.pvalues[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78c28d4c-7143-4fe0-8c50-9eb9d0f49272",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Does the code above make sense? Can you see the data resampled, and the estimated model?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c1c1fd-ccc2-4102-9dfd-2ff3472c6158",
   "metadata": {},
   "source": [
    "Now, compute the power."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5acb69b-3d3d-43b3-8ce0-c12e2fccef2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "Power = np.sum(p_value < 0.05)/number_of_repetitions\n",
    "print(Power)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e0398d2-ead5-47ef-8c8b-5ad750444c85",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What value do you find for `Power`?\n",
    "\n",
    "- Interpert this value.\n",
    "- Do you have enough data to detect the effect?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b1b594-e32d-48cd-93c9-dd17a8a54d9a",
   "metadata": {},
   "source": [
    "## 5* (Optional Extension)- You have the power!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c829aa-5eff-4c3c-bc89-dcbad7d81915",
   "metadata": {},
   "source": [
    "In this optional Mini, we consider an **alternative strategy to compute the power**.\n",
    "\n",
    "Instead of resampling the data, we **fit a model to the data**, then use that model to generate new data samples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74f2ecf9-81dd-43e1-8b4e-8084606422a1",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "Let's begin by fitting a model to the data.\n",
    "\n",
    "We'll use a simple model: a line.\n",
    "\n",
    "We've already fit this model to estimate the effect size and its significance.\n",
    "\n",
    "We'll now fit the model, save the model parameters, then use this model to generate new data samples.\n",
    "\n",
    "Let's begin by fitting the model to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81a64bc3-e7d6-47e1-bff5-de84ec32afa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model (a line) to the data.\n",
    "\n",
    "from statsmodels.formula.api import ols\n",
    "dat   = {\"x\": x, \"lifespan\": lifespan}\n",
    "model = ols(\"lifespan ~ 1 + x\", data=dat).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61361434-36eb-4d4a-ab59-3f89e8199773",
   "metadata": {},
   "source": [
    "The model estimates two parameters: the `slope` and `intercept`.\n",
    "\n",
    "Let's get those two parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a149b27f-81aa-45df-a5ad-0f3fc5ce825e",
   "metadata": {},
   "outputs": [],
   "source": [
    "intercept = model.params[0]\n",
    "slope     = model.params[1]\n",
    "print('Intercept estimate = ',intercept)\n",
    "print('Slope estimate     = ',slope)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac15591-f459-4eba-a596-669305345c4f",
   "metadata": {},
   "source": [
    "We'll need one more estimate from the model fit: the dispersion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efbaaa8d-6d3f-4051-836f-4c982f4800e3",
   "metadata": {},
   "source": [
    "\n",
    "The **dispersion parameter** describes the amount of uncertainty in our ability to predinct each data point. In this case, it is the residual standard deviation of the lifespan after we have tried to predict it using the expression level of substance $x$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50278031-7b2b-40d7-9a75-5a635813c3ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "dispersion = np.sqrt(model.scale)\n",
    "print('Dispersion parameter = ',dispersion)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "db7f8d51-9109-4761-8470-6b2de249c3f5",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "[Add some text interpreting these three estimated values…]\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2929498-f259-4acf-aeb0-66e2e7b050b7",
   "metadata": {},
   "source": [
    "With the 3 estimated parameters, we can now simulate realizations of the model. \n",
    "\n",
    "To do so, we'll evaluate this model:\n",
    "\n",
    "`lifespan_modeled = intercept + slope * x + np.random.normal(loc=0.0, scale=dispersion, size=N)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7989c36c-9474-4d3b-9335-84a81136126c",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Describe - in words - each term in the equation.\n",
    "- What variables do you recognize?\n",
    "- What variables are now?\n",
    "- What is the equation doing???\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b390f90-1664-4a11-b9fb-ef173e16026a",
   "metadata": {},
   "source": [
    "Now, let's see what the model does.\n",
    "\n",
    "To do so, we'll evaluate the model, and compare the `lifespan_modeled` to the original `lifespan`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f938ff87-2289-4e2f-9dfe-aade886899e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lifespan_modeled = intercept + slope * x + np.random.normal(loc=0.0, scale=dispersion, size=[N,1])\n",
    "plt.scatter(x,lifespan, label='original data')\n",
    "plt.scatter(x,lifespan_modeled, label='lifespan modeled')\n",
    "plt.legend()\n",
    "plt.grid()\n",
    "plt.ylim([45, 105]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1bb3679-35a9-4347-9f16-f248054d404a",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Compare the original `lifespan` data and the `lifespan_modeled` data.\n",
    "- How do they look similar?\n",
    "- How do they look different?\n",
    "- Do the modeled data provide a \"good\" representation of the original data?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c54bd9-d2ce-4a38-a01d-ddf50ec6f6c8",
   "metadata": {},
   "source": [
    "With the model estimated, we can use it to compute the power. Here are the steps in general:\n",
    "\n",
    "1. Use the model to generate `lifespan_modeled` data.\n",
    "2. Compute the significance of the proposed effect.\n",
    "3. Repeat 1-2 many times.\n",
    "\n",
    "The power is the number of times we detect a significant effect, divided by the total number of repetitions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93fef8b-6d01-4605-9409-26c3e4b9f32e",
   "metadata": {},
   "source": [
    "Let's apply this procedure to our data and effect of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e7f88d7-13b2-4601-8885-eddbf5ca8dc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_repetitions = 1000\n",
    "p_value = np.zeros(number_of_repetitions)\n",
    "for k in np.arange(number_of_repetitions):\n",
    "    lifespan_modeled = intercept + slope * x + np.random.normal(loc=0.0, scale=dispersion, size=[N,1])\n",
    "    dat                = {\"x\": x, \"lifespan\": lifespan_modeled}\n",
    "    regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "    p_value[k] = regression_results.pvalues[1]\n",
    "Power = np.sum(p_value < 0.05)/number_of_repetitions\n",
    "print('Power = ',Power)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf0d3810-8ae2-4a92-97a9-e9e4b92f6232",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Does the code above make sense? Can you see the modeled lifespan data, and the estimated model?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d169bd-14e4-4b7c-8abb-f390152b1e90",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What value do you find for `Power`?\n",
    "\n",
    "- Interpert this value.\n",
    "- Do you have enough data to detect the effect?\n",
    "- How do these results compare to the resampling approached described in Mini 5?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e371a04-3a3a-4060-8a69-c186c5ae1d1e",
   "metadata": {},
   "source": [
    "## 6- You have the sample size!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c76961b2-e88a-4f5e-b9be-c12123c379e0",
   "metadata": {},
   "source": [
    "- In Mini 5, we used resampling with a fixed sample size to compute the power.\n",
    "- We can use the same resampling approach to compute the **sample size** for a fixed power."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf20ce25-41f7-48ff-b98a-8d62993fb7fd",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "Provide some intuition for this calculation.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3a5b17-8f4d-41c8-b964-80190e17e82c",
   "metadata": {},
   "source": [
    "Let's do it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1627daf9-3802-4a01-a6a8-29f6be074a75",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_resampled = 1000;\n",
    "number_of_repetitions = 1000\n",
    "p_value = np.zeros(number_of_repetitions)\n",
    "for k in np.arange(number_of_repetitions):\n",
    "    ind = np.random.choice(np.size(x), N_resampled)\n",
    "    x_resampled = x[ind]\n",
    "    lifespan_resampled = lifespan[ind]\n",
    "    dat                = {\"x\": x_resampled, \"lifespan\": lifespan_resampled}\n",
    "    regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "    p_value[k] = regression_results.pvalues[1]\n",
    "Power = np.sum(p_value < 0.05)/number_of_repetitions\n",
    "print('N=',N_resampled,'Power=',Power)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3643a5-553f-4d6b-abae-96c2372318c7",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Compare this code to the code in Mini 5. What has changed?\n",
    "\n",
    "**A:** ... we include `N_resampled` as a new variable, and use it in the code.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb0b017d-c306-4bb8-9d85-a8094baf7fc7",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** How does increasing the sample size `N_resampled` impact the power?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74580d8a-d275-40c6-93ed-dbfd13c6c8e9",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** At what value of `N_resampled` is the power just above 0.8?\n",
    "\n",
    "- How does this compare to the results from your online calculator?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b63c087-4bda-449e-8ec7-38b0c99ab679",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 6* (Optional Extension) - You have the sample size!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d51cb28e-e7de-444f-8b79-ae3674177ee6",
   "metadata": {},
   "source": [
    "In this optional Mini, we consider an **alternative strategy to compute the sample size**.\n",
    "\n",
    "Instead of resampling the data, we **fit a model to the data**, then use that model to generate new data samples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f29b244-371a-468b-b5df-afe48606c21e",
   "metadata": {},
   "source": [
    "---\n",
    "If you haven't done so, please first complete *Mini 5\\* (Extensions)*.\n",
    "\n",
    "In that optional Mini, we introduce the modeling approach, and apply it to compute the power."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0206a6-a420-4ec7-941b-93278a68e03a",
   "metadata": {},
   "source": [
    "---\n",
    "Let's now use this same strategy to compute the sample size.\n",
    "\n",
    "The first step is to estimate the model. We use the same code as in *Mini 5\\* (Extensions)*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2573f16-3b8d-4306-b95e-5c572066ecee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model (a line) to the data.\n",
    "from statsmodels.formula.api import ols\n",
    "dat   = {\"x\": x, \"lifespan\": lifespan}\n",
    "model = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "\n",
    "intercept  = model.params[0]\n",
    "slope      = model.params[1]\n",
    "dispersion = np.sqrt(model.scale)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3638a909-1255-4eac-af08-62c816b78606",
   "metadata": {},
   "source": [
    "We'll now use the model to generate new samples of data, given a sample size `N_modeled`.\n",
    "\n",
    "To do so, we'll re-use the code from Mini 6, with some small changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21a6a095-9482-4e66-8c9d-734db04e3a98",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_modeled = 800;                                  # This is the a part!\n",
    "number_of_repetitions = 1000\n",
    "p_value = np.zeros(number_of_repetitions)\n",
    "for k in np.arange(number_of_repetitions):\n",
    "    ind = np.random.choice(np.size(x), N_modeled)  # There's something new here, and in the next line.\n",
    "    lifespan_modeled = intercept + slope * x[ind] + np.random.normal(loc=0.0, scale=dispersion, size=[N_modeled,1])\n",
    "    dat                = {\"x\": x[ind], \"lifespan\": lifespan_modeled}\n",
    "    regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "    p_value[k] = regression_results.pvalues[1]\n",
    "Power = np.sum(p_value < 0.05)/number_of_repetitions\n",
    "print('N=',N_modeled,'Power=',Power)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14299131-f678-447e-8634-b8fb67130e15",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>Alert:</b>\n",
    "</p>\n",
    "\n",
    "- Notice the use of `ind` and `x[ind]` in the code. Why do we need this here?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d961f5-81bf-47ec-9beb-4ef8e929c924",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Compare this code to the code in Mini 6. What has changed?\n",
    "\n",
    "**A:** ... we include `N_modeled` as a new variable.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dabd5e4-18f6-46fe-90e6-2dabf86de5cf",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** At what value of `N_modeled` is the power just above 0.8?\n",
    "\n",
    "- How does this compare to the results from your online calculator?\n",
    "- How does this result compare to your resampling approach in Mini 6?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b0d6513-ce83-4e8f-b2a5-285ae9ae5ab2",
   "metadata": {},
   "source": [
    "## 7- Well powered experiments can provide strong evidence"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db6374d-cf7d-4781-9b36-e1163e224b7b",
   "metadata": {},
   "source": [
    "- We've now used (at least) two approaches to calculate the sample size\n",
    "  1) online calculator\n",
    "  2) resampling\n",
    "- The result from the resampling analysis suggests a much larger sample size is required than the estimated sample size many researchers will obtain from naïve online searches.\r",
    "- Let’s now “collect new data” and see if we detect a significant effect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "099fe13a-721f-44ac-a55a-8c068d053dc5",
   "metadata": {},
   "source": [
    "Let's begin by collecting the new data, now using the sample size you found through resampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c489a3-b950-4f63-94e1-8b087bd7427c",
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 1000                     #NOTE: Learner inputs sample size, based on results in previous Minis.\n",
    "x,lifespan = load_data(N)             # Use this by default.\n",
    "# x,lifespan = load_data_Colab(N)     # Use this if on !!GOOGLE COLAB!!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "942a8b66-3722-484f-b479-cbe115f10fa1",
   "metadata": {},
   "source": [
    "Let's plot it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f315412-72c0-4e8e-9fa8-ec373f2244f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x,lifespan)\n",
    "plt.xlabel('Genetic biomarker x')\n",
    "plt.ylabel('Lifespan (years)');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8014187-15f0-4f8b-9169-0a1ba676051b",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Compare the plot of these new data (with `N=800`) to the plot of the original data. How are the plots similar or different?\n",
    "\n",
    "- Do you see the same trend in both plots?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a99d171d-358b-41d3-9dab-c4b52842f08c",
   "metadata": {},
   "source": [
    "Now, let's test our hypothesis in this new data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdf5b390-752c-472b-bac7-ac5b69811260",
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.formula.api import ols\n",
    "\n",
    "dat                = {\"x\": x, \"lifespan\": lifespan}\n",
    "regression_results = ols(\"lifespan ~ 1 + x\", data=dat).fit()\n",
    "print('Slope estimate =',regression_results.params[1])\n",
    "print('p-value        =',regression_results.pvalues[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4761ace-4c77-4faa-b208-0462756fefa4",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What is the effect size and significance?\n",
    "\n",
    "- How do these results compare to the original data set?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55240b1c-bdab-4bd1-9a6d-517c3016fccf",
   "metadata": {},
   "source": [
    "Let's also **visualize** the estimated line by plotting it with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecfc78e3-8de5-49d9-a8fc-1cc2a2312fb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred   = regression_results.get_prediction().summary_frame()\n",
    "mn     = pred['mean']\n",
    "ci_low = pred['mean_ci_lower'] \n",
    "ci_upp = pred['mean_ci_upper']\n",
    "\n",
    "# And plot it.\n",
    "indices_sorted = np.argsort(x,0)\n",
    "plt.scatter(x,lifespan)\n",
    "plt.plot(x[indices_sorted[:,0]],mn[indices_sorted[:,0]], 'r')\n",
    "plt.plot(x[indices_sorted[:,0]],ci_low[indices_sorted[:,0]], ':r')\n",
    "plt.plot(x[indices_sorted[:,0]],ci_upp[indices_sorted[:,0]], ':r')\n",
    "plt.xlabel('Genetic biomarker x')\n",
    "plt.ylabel('Lifespan (years)');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c61310f-0dca-4663-af22-46396fe732fc",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** How do these results compare to the original data set?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e67739-960d-4a6b-8ba4-8cf73c53ae7b",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** What do you conclude about the relationship between the genetic biomarker `x` and `lifespan`?\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3c863e-38fe-4f1a-b200-33e92a3bdc2e",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "\n",
    "*Conclusion*:\n",
    "\n",
    "- The resampling procedure allowed us to compute a large enough sample size, so our experiment was sufficiently powered, and we detected a significant effect.\n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6109d93d-6b05-455e-941a-401447e4bcfb",
   "metadata": {},
   "source": [
    "## 8- Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694361d5-4d5d-470c-bf3c-7076bf25c578",
   "metadata": {},
   "source": [
    "- Internet tools and statistics textbooks can provide useful formulas and calculators to estimate sample size, but uncertainty about which tools to use and how to provide correct prior knowledge about effect size can lead to underpowered experiments. In this example, many users will incorrectly arrive at a sample size of 10-100.\r",
    "- There are a number of reasons that researchers might arrive at the wrong answer in this example:\n",
    "    - It’s very easy to underestimate variability in lifespan, which leads to an underestimate of sample size.\r",
    "    - Many textbooks and online calculators focus on predictors that can only take on a small number of values. Here, substance $x$ has a range of expression levels and many users will end up using the wrong tools.\r",
    "    - The prior knowledge about the effect of substance $x$ on lifespan was originally described in terms of the difference between individuals at the lowest vs. the highest expression levels. Most data points collected for the experiment will be somewhere between these expression levels.\n",
    " \n",
    "- However, the actual required effect size is more like 500-1000\r",
    "    - This is due to the large variability in human lifespans compared to the relatively small effect size.\r",
    "- Although the effect size is meaningful scientifically, it is small compared to the measurement variability."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e35c7c2e-0d4a-4236-94d1-21ca374e6df2",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>Warning:</b>\n",
    "</p>\n",
    "In this example, we were lucky that the initial draw of a small sample size produced the expected effect. An unlucky sample may have produced (by chance) an opposite effect. In that case, resampling will not produce meaningful power/sample size results. Preliminary data is often important for future experimental design, but it’s important to consider how variability in a small, preliminary dataset can influence power and sample size estimates.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2274d3-4bf7-4cdb-82c5-4ff1d9647fd3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "**Q:** Consider a new preliminary data set (provided). Use these data to estimate the sample size required to achieve 80% power.\n",
    "\n",
    "**A:**\n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}