multi_data_files_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Formatting the Output in a Table\n",
    "\n",
    "This continues the previous lesson, which got a bit long for one lesson. Recall that our task was to \n",
    "read three data files in the `data` directory, each RT data from a different participant, and combine them in a DataFrame named `df`. In the previous lessons we computed means and 95% CIs for individual participants and the group. In this final lesson in the series, we will print a table that includes all of the results: with columns for mean RT, lower 95% CI, and upper 95% CI, and with one row for each participant and the bottom row showing the mean and CIs across participants.\n",
    "\n",
    "Since this is a new notebook file, we need to import the libraries we will use, and import the data. To do this, we've copied and pasted the code from the previous lesson into the code cell below (a few lines were deleted for simplicity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in three files from the data folder, whose names start with \"s\" and end in \"csv\"\n",
    "# concatenate them into one dataframe\n",
    "\n",
    "import pandas as pd\n",
    "import glob\n",
    "\n",
    "# use glob to get all the files that start with \"s\" and end with \"csv\"\n",
    "# glob returns a list of file names\n",
    "filenames = glob.glob(\"data/s*.csv\")\n",
    "\n",
    "# read in the files and concatenate them into one dataframe\n",
    "dataframes = []\n",
    "for f in filenames:\n",
    "    dataframes.append(pd.read_csv(f))\n",
    "df = pd.concat(dataframes, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Formatting the output nicely"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final instruction was, \"Finally, show a table that includes all of the results: with columns for mean RT, lower 95% CI, and upper 95% CI, and with one row for each participant and the bottom row showing the mean and CIs across participants.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a decent first pass, but the CIs are rounded to 4, not 2, decimal places, and the table is missing the mean across participants. These are two different problems, so let's try to solve them one at a time. We'll start with rounding. Our prompt above was pretty specific in asking Copilot to round each of the stated values to 2 decimal places, so maybe instead of prompt engineering we can ask Chat for help. Let's try this prompt in Copilot Chat: `modify the code in the selected cell so that the CIs are rounded to 2 decimal places`. The result is:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To modify the code to round the CIs to 2 decimal places, you can change the `round()` method to round to 2 decimal places instead of the default 4 decimal places. Here's the modified code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "mean_rt = df.groupby('participantID')['RT'].mean().round(2)\n",
    "sem_rt = df.groupby('participantID')['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci})\n",
    "print(ci_table)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In this code, the `round()` method is used to round the mean RT and SEM values to 2 decimal places, and the `round()` method is also used to round the upper and lower CI values to 2 decimal places.\n",
    ">\n",
    "> The `upper_ci` and `lower_ci` variables calculate the upper and lower CIs by adding and subtracting the `ci_rt` value from the `mean_rt` value, respectively. The `round()` method is used to round the resulting values to 2 decimal places.\n",
    "> \n",
    "> Finally, the `ci_table` variable creates a table of the mean RT, lower CI, and upper CI values using the `pd.DataFrame()` function, and the `print()` function is used to print the table.\n",
    "> \n",
    "> This code should give you the desired output of a table containing the mean RT, lower CI, and upper CI values rounded to 2 decimal places.\n",
    "\n",
    "And indeed, if you paste the code below, you can see it works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's address the problem of the missing mean across participants. Based on what we learned earlier in the course, this might not be surprising. While we can use `.groupby()` to split the data by participantID, or simply `.mean()` to get the average across participantIDs, there is no way of generating both split and averaged data using the same command. \n",
    "\n",
    "Instead, what we could try doing is create the across-subject data as a separate table, with code to round to two decimal places, and then merge that table with the by-participant table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to merge the two tables. In the cell below, I only had to type the first one-line prompt, and Copilot generated all the subsequent code and prompts/comments. However, I had to hit `Enter` twice after the first prompt, and after some other lines (where you see blank lines below) to get all the code. I would tell when Copilot had finished because it started regenerating the same prompts and code it had already generated.\n",
    "\n",
    "Unfortunately when we try to run the cell below we get an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge the two tables that are the output of each of the two code cells above\n",
    "\n",
    "# calculate the mean RT for each participantID\n",
    "mean_rt = df.groupby('participantID')['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT for each participantID\n",
    "sem_rt = df.groupby('participantID')['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci})\n",
    "\n",
    "# calculate the mean RT across all participantID\n",
    "mean_rt = df['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT across all participantID\n",
    "sem_rt = df['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table2 = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci}, index=['RT'])\n",
    "\n",
    "# merge the two tables\n",
    "ci_table.append(ci_table2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error message is that `'DataFrame' object has no attribute 'append'`. This means that the code is trying to append the across-subject table to the by-participant table, but that the `append()` method doesn't exist for DataFrames. \n",
    "\n",
    "If you do a Web search for \"pandas dataframe append\", you may find this error puzzling, because there are many descriptions of how to use this method, including in the official pandas documentation. However, what's not immediately obvious is that the `.append()` method in pandas was **depricated** in [November, 2021](https://github.com/pandas-dev/pandas/pull/44539). \"Depricated\" in a programming language means that a feature/function/method/etc. was made obsolete — typically replaced by a newer feature, or perhaps because it was already redundant with another way of doing the same thing. Since Copilot is trained on a vast amount of code, this will inevitably include **legacy code** that is no longer used. \n",
    "\n",
    "We'll ask Copilot Chat for help again, this time telling it what the error message is, and asking how to fix it:\n",
    "\n",
    "`the selected cell generates an error: \"'DatFrame' object has no attribute 'append'\". how can i fix this error?`\n",
    "\n",
    "The response is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The error message \"'DataFrame' object has no attribute 'append'\" suggests that the `append()` method is not available for the `DataFrame` object in the code.\n",
    "> \n",
    "> To fix this error, you can use the `concat()` function instead of the `append()` method to concatenate the two dataframes vertically. Here's the modified code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# merge the two tables that are the output of each of the two code cells above\n",
    "\n",
    "# calculate the mean RT for each participantID\n",
    "mean_rt = df.groupby('participantID')['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT for each participantID\n",
    "sem_rt = df.groupby('participantID')['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci})\n",
    "\n",
    "# calculate the mean RT across all participantID\n",
    "mean_rt = df['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT across all participantID\n",
    "sem_rt = df['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table2 = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci}, index=['RT'])\n",
    "\n",
    "# concatenate the two tables vertically\n",
    "ci_table = pd.concat([ci_table, ci_table2])\n",
    "\n",
    "# print the merged table\n",
    "print(ci_table)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In this code, the `concat()` function is used to concatenate the two dataframes `ci_table` and `ci_table2` vertically. The resulting dataframe contains the rows of both dataframes stacked on top of each other.\n",
    "> \n",
    "> Finally, the `print()` function is used to print the merged table.\n",
    "> \n",
    "> This code should fix the error and give you the desired output of a merged table containing the mean RT, lower CI, and upper CI values for each participant, as well as the mean RT, lower CI, and upper CI values across all participants.\n",
    "\n",
    "When we copy and paste this code below, it works. The only lingering issue is that the last row is labelled `RT`, which does not tell us the values represent the mean across all participants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can as Chat to do this for us: `modify the code in the selected cell so that the last row in the final dataframe is labelled \"Average\"`\n",
    "\n",
    "> To modify the code to label the last row in the final dataframe as \"Average\", you can change the index label of the last row in the `ci_table2` dataframe to \"Average\" before concatenating the two dataframes. Here's the modified code:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# merge the two tables that are the output of each of the two code cells above\n",
    "\n",
    "# calculate the mean RT for each participantID\n",
    "mean_rt = df.groupby('participantID')['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT for each participantID\n",
    "sem_rt = df.groupby('participantID')['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci})\n",
    "\n",
    "# calculate the mean RT across all participantID\n",
    "mean_rt = df['RT'].mean().round(2)\n",
    "\n",
    "# calculate the 95% confidence intervals for the mean RT across all participantID\n",
    "sem_rt = df['RT'].sem().round(2)\n",
    "ci_rt = sem_rt * 1.96\n",
    "upper_ci = (mean_rt + ci_rt).round(2)\n",
    "lower_ci = (mean_rt - ci_rt).round(2)\n",
    "ci_table2 = pd.DataFrame({'Mean RT': mean_rt, 'Lower CI': lower_ci, 'Upper CI': upper_ci}, index=['Average'])\n",
    "\n",
    "# concatenate the two tables vertically\n",
    "ci_table = pd.concat([ci_table, ci_table2])\n",
    "\n",
    "# print the merged table\n",
    "print(ci_table)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In this code, the `index` parameter of the `pd.DataFrame()` function is used to set the index label of the last row in the `ci_table2` dataframe to \"Average\".\n",
    "> \n",
    "> The `concat()` function is then used to concatenate the two dataframes `ci_table` and `ci_table2` vertically. The resulting dataframe contains the rows of both dataframes stacked on top of each other.\n",
    "> \n",
    "> Finally, the `print()` function is used to print the merged table.\n",
    "> \n",
    "> This code should give you the desired output of a merged table containing the mean RT, lower CI, and upper CI values for each participant, as well as the mean RT, lower CI, and upper CI values across all participants, with the last row labelled as \"Average\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the code above and then paste it in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We've now completed the task of calculating the mean RT and 95% CIs for each participant, and for all participants. We've also learned a lot about how to use Copilot to help us write code, and how to use Copilot Chat to help us understand errors and fix them.\n",
    "\n",
    "As you've seen, the process of using AI to generate code isn't always smooth, nor are the results always accurate. It's critical to critically assess the outputs you see, and to understand what you are asking Copilot to do — as well as what the code it generates is doing. In this lesson, for the sake of expediency and length we did not provide detailed explanations of what the code was doing every step of the way. However, you've seen that Copilot can be useful at explaining code for you. \n",
    "\n",
    "There were also many points in this lesson where, as an experienced coder, I knew what the problem was and how to fix it. However, I deliberately did not fix the problems myself, because I recognize that as a novice coder you may not know how to fix the problems. Hopefully this lesson has shown you how to use Copilot effectively, but also underscored the value of actually knowing Python and being able to read and debug code yourself.\n",
    "\n",
    "The best approach to learning to use Python for data science is to write code yourself when you can,  because this helps build and reinforce your knowledge (remember, coding is a procedural skill that you only learn by doing — and you lose the knowledge if you don't use it). But, when you encounter a problem you've never solved before, or are getting errors in your Code, Copilot can be a huge help to deepen your knowledge and solve problems. As we've seen, Copilot Chat can be much more useful in this regard than simply writing prompts in code cells of your notebook, because it gives you both code, and detailed explanations of what that code is doing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "<h2>Copilot in Later Chapters of this Book</h2>\n",
    "\n",
    "Throughout the rest of this book, we focus on teaching you to write and understand code yourself, rather than using Copilot. We believe — and feel this lesson has demonstrated — that learning to write code yourself is the best way to learn to code. However, whether you're taking this course for credit and have assignments that Copilot may be useful in helping with, or simply using this book to learn how to apply Python to your own data, we recognize that Copilot can be a useful tool. We hope this chapter has given you a good introduction to how to use Copilot effectively, and that you will continue to use it on your coding journey.\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ncil",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}