explore-analyze-data-with-R/solution/challenge-Data_Exploration.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## **Flights Data Exploration Challenge**\n",
                "\n",
                "A significant part of a data scientist's role is to explore, analyze, and visualize data. In this challenge, you'll explore a real-world dataset containing flights data from the US Department of Transportation.\n",
                "\n",
                "Let's start by loading the required packages.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Load packages that you'll require in this adventure\n",
                "suppressPackageStartupMessages({\n",
                "  library(tidyverse)\n",
                "  library(summarytools)\n",
                "  library(glue)\n",
                "  library(patchwork)\n",
                "  })\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now, we can import the data into R and start doing some data science on it!\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Load and view the data\n",
                "df_flights <- read_csv(\"https://raw.githubusercontent.com/MicrosoftDocs/ml-basics/master/challenges/data/flights.csv\", show_col_types = FALSE)\n",
                "\n",
                "df_flights %>%\n",
                "  slice_head(n = 7)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The dataset contains observations of US domestic flights in 2013, and consists of the following fields:\n",
                "\n",
                "-   **Year**: The year of the flight (all records are from 2013)\n",
                "\n",
                "-   **Month**: The month of the flight\n",
                "\n",
                "-   **DayofMonth**: The day of the month on which the flight departed\n",
                "\n",
                "-   **DayOfWeek**: The day of the week on which the flight departed - from 1 (Monday) to 7 (Sunday)\n",
                "\n",
                "-   **Carrier**: The two-letter abbreviation for the airline.\n",
                "\n",
                "-   **OriginAirportID**: A unique numeric identifier for the departure aiport\n",
                "\n",
                "-   **OriginAirportName**: The full name of the departure airport\n",
                "\n",
                "-   **OriginCity**: The departure airport city\n",
                "\n",
                "-   **OriginState**: The departure airport state\n",
                "\n",
                "-   **DestAirportID**: A unique numeric identifier for the destination aiport\n",
                "\n",
                "-   **DestAirportName**: The full name of the destination airport\n",
                "\n",
                "-   **DestCity**: The destination airport city\n",
                "\n",
                "-   **DestState**: The destination airport state\n",
                "\n",
                "-   **CRSDepTime**: The scheduled departure time\n",
                "\n",
                "-   **DepDelay**: The number of minutes departure was delayed (flight that left ahead of schedule have a negative value)\n",
                "\n",
                "-   **DelDelay15**: A binary indicator that departure was delayed by more than 15 minutes (and therefore considered \"late\")\n",
                "\n",
                "-   **CRSArrTime**: The scheduled arrival time\n",
                "\n",
                "-   **ArrDelay**: The number of minutes arrival was delayed (flight that arrived ahead of schedule have a negative value)\n",
                "\n",
                "-   **ArrDelay15**: A binary indicator that arrival was delayed by more than 15 minutes (and therefore considered \"late\")\n",
                "\n",
                "-   **Cancelled**: A binary indicator that the flight was cancelled\n",
                "\n",
                "Your challenge is to explore the flight data to analyze possible factors that affect delays in departure or arrival of a flight.\n",
                "\n",
                "1.  Start by cleaning the data.\n",
                "\n",
                "    -   Identify any null or missing data, and impute appropriate replacement values.\n",
                "\n",
                "    -   Identify and eliminate any outliers in the **DepDelay** and **ArrDelay** columns.\n",
                "\n",
                "2.  Explore the cleaned data.\n",
                "\n",
                "    -   View summary statistics for the numeric fields in the dataset.\n",
                "\n",
                "    -   Determine the distribution of the **DepDelay** and **ArrDelay** columns.\n",
                "\n",
                "    -   Use statistics, aggregate functions, and visualizations to answer the following questions:\n",
                "\n",
                "        -   *What are the average (mean) departure and arrival delays?*\n",
                "\n",
                "        -   *How do the carriers compare in terms of arrival delay performance?*\n",
                "\n",
                "        -   *Is there a noticable difference in arrival delays for different days of the week?*\n",
                "\n",
                "        -   *Which departure airport has the highest average departure delay?*\n",
                "\n",
                "        -   *Do **late** departures tend to result in longer arrival delays than on-time departures?*\n",
                "\n",
                "        -   *Which route (from origin airport to destination airport) has the most **late** arrivals?*\n",
                "\n",
                "        -   *Which route has the highest average arrival delay?*\n",
                "\n",
                "Sometimes, when we have a lot of columns in our data, it may difficult to get a grip of the data at first sight using `slice_head`\n",
                "\n",
                "`glimpse` produces a transposed version where columns run down the page, and data runs across. This makes it possible to see every column in a data frame. Into the bargain, it also shows the dimension of the tibble and underlying data types of the columns.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Get a glimpse of your data\n",
                "df_flights %>%\n",
                "  glimpse()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Clean missing values\n",
                "\n",
                "Once you have imported your data, it is always a good idea to clean it. Sadly, this is often chronically underestimated, yet it's a fundamental step required for the subsequent operations in data analysis.\n",
                "\n",
                "Let's find how many null values there are for each column.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Find how many null values there are for each column.\n",
                "colSums(is.na(df_flights))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Hmm, looks like there are some NA (missing values) `late departure` indicators (DepDel15 column). Departures are considered late if the delay is 15 minutes or more, so let's see the delays for the ones with an NA late indicator:\n",
                "\n",
                "**Question 1.**\n",
                "\n",
                "Starting with `df_flights`, **select** columns `DepDelay` and `DepDel15` then **filter** to obtain rows where the value of `DepDel15` is `NA`. Assign the results in a variable name `flights_depdel`.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Select columns DepDelay and DepDel15\n",
                "# and then Filter the tibble to obtain\n",
                "# observations where there is a missing value of DepDel15\n",
                "\n",
                "flights_depdel <- df_flights %>%\n",
                "  select(...., ....) %>%\n",
                "  filter(is.na(....))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 1.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Good job! Now, let's `glimpse` at `flights_depdel`.\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "flights_depdel %>%\n",
                "  glimpse()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "From the first few observations, it looks like the observations in `DepDel15` (a binary indicator that departure was delayed by more than 15 minutes) all have a corresponding delay of 0 in `DepDelay`(the number of minutes departure was delayed). Let's check by looking at the summary statistics for the `DepDelay` records:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Get summary statistics using summary function\n",
                "df_flights %>%\n",
                "  filter(rowSums(is.na(.)) > 0) %>%\n",
                "  select(DepDelay) %>%\n",
                "  summary()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The min, max, and mean are all 0; so it seems that none of these were actually *late* departures.\n",
                "\n",
                "**Question 2.**\n",
                "\n",
                "Starting with `df_flights`, replace the missing values in the column **DepDel15** with a `0`. Assign this to a variable name `df_flights`.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Replace missing values in DepDel15 with 0\n",
                "df_flights <- df_flights %>%\n",
                "  mutate(DepDel15 = ....)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 2.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Good job! No missing values now. Let's take this a little further.\n",
                "\n",
                "### **Clean outliers**\n",
                "\n",
                "An outlier is a data point that differs significantly from other observations. Let's create a function that shows the distribution and summary statistics for a given column.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Function to show summary stats and distribution for a column\n",
                "show_distribution <- function(var_data, binwidth) {\n",
                "\n",
                "  # Get summary statistics by first extracting values from the column\n",
                "  min_val <- min(pull(var_data))\n",
                "  max_val <- max(pull(var_data))\n",
                "  mean_val <- mean(pull(var_data))\n",
                "  med_val <- median(pull(var_data))\n",
                "  mod_val <- statip::mfv(pull(var_data))\n",
                "\n",
                "  # Print the stats\n",
                "  stats <- glue::glue(\n",
                "  \"Minimum: {format(round(min_val, 2), nsmall = 2)}\n",
                "   Mean: {format(round(mean_val, 2), nsmall = 2)}\n",
                "   Median: {format(round(med_val, 2), nsmall = 2)}\n",
                "   Mode: {format(round(mod_val, 2), nsmall = 2)}\n",
                "   Maximum: {format(round(max_val, 2), nsmall = 2)}\"\n",
                "  )\n",
                "\n",
                "  theme_set(theme_light())\n",
                "  # Plot the histogram\n",
                "  hist_gram <- ggplot(var_data) +\n",
                "  geom_histogram(aes(x = pull(var_data)), binwidth = binwidth,\n",
                "                 fill = \"midnightblue\", alpha = 0.7, boundary = 0.4) +\n",
                "\n",
                "  # Add lines for the statistics\n",
                "  geom_vline(xintercept = min_val, color = \"gray33\",\n",
                " linetype = \"dashed\", size = 1.3) +\n",
                "  geom_vline(xintercept = mean_val, color = \"cyan\",\n",
                " linetype = \"dashed\", size = 1.3) +\n",
                "  geom_vline(xintercept = med_val, color = \"red\",\n",
                " linetype = \"dashed\", size = 1.3) +\n",
                "  geom_vline(xintercept = mod_val, color = \"yellow\",\n",
                " linetype = \"dashed\", size = 1.3) +\n",
                "  geom_vline(xintercept = max_val, color = \"gray33\",\n",
                " linetype = \"dashed\", size = 1.3) +\n",
                "\n",
                "  # Add titles and labels\n",
                "  ggtitle(\"Data Distribution\") +\n",
                "  xlab(\"\") +\n",
                "  ylab(\"Frequency\") +\n",
                "  theme(plot.title = element_text(hjust = 0.5))\n",
                "\n",
                "  # Plot the box plot\n",
                "  bx_plt <- ggplot(data = var_data) +\n",
                "  geom_boxplot(mapping = aes(x = pull(var_data), y = 1),\n",
                "               fill = \"#E69F00\", color = \"gray23\", alpha = 0.7) +\n",
                "\n",
                "    # Add titles and labels\n",
                "  xlab(\"Value\") +\n",
                "  ylab(\"\") +\n",
                "  theme(plot.title = element_text(hjust = 0.5))\n",
                "\n",
                "\n",
                "  # To return multiple outputs, use a `list`\n",
                "  return(\n",
                "\n",
                "    list(stats,\n",
                "         hist_gram / bx_plt)) # End of returned outputs\n",
                "\n",
                "} # End of function\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Question 3.** Starting with the `df_flights` data, only keep the **`DepDelay`** column. Assign this to a variable name `df_col`.\n",
                "\n",
                "Once you have this figured out, call the function `show_distribution` with the arguments names and corresponding values as follows: `var_data = df_col` and `binwidth = 100`\n",
                "\n",
                "From the function output, what's the distribution of **DepDelay** (the number of minutes departure was delayed)?\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Select DepDelay column\n",
                "df_col <- df_flights %>%\n",
                "  ....\n",
                "\n",
                "# Call the function show_distribution\n",
                "show_distribution(var_data = df_col, binwidth = 100)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 3.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now, let's investigate the distribution of **ArrDelay** (the number of minutes arrival was delayed)\n",
                "\n",
                "**Question 4.** Starting with the `df_flights` data, only keep the **`ArrDelay`** column. Assign this to a variable name `df_col`.\n",
                "\n",
                "Once you have this figured out, call the function `show_distribution` with the arguments names and corresponding values as follows: `var_data = df_col` and `binwidth = 100` (value of the width of each bin along the X-axis)\n",
                "\n",
                "From the function output, what's the distribution of **ArrDelay**?\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Select DepDelay column\n",
                "df_col <- df_flights %>%\n",
                "  ....\n",
                "\n",
                "# Call the function show_distribution\n",
                "show_distribution(var_data = df_col, binwidth = 100)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 4.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "From both outputs, there are outliers at the lower and upper ends of both variables. Let's trim the data so that we include only rows where the values for these fields are within the 1st and 90th percentile. We begin with the **ArrDelay** observation.\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Trim outliers for ArrDelay based on 1% and 90% percentiles\n",
                "# Produce quantiles corresponding to 1% and 90%\n",
                "arrdelay_01pcntile <- df_flights %>%\n",
                "  pull(ArrDelay) %>%\n",
                "  quantile(probs = 1 / 100, names = FALSE)\n",
                "\n",
                "arrdelay_90pcntile <- df_flights %>%\n",
                "  pull(ArrDelay) %>%\n",
                "  quantile(probs = 90 / 100, names = FALSE)\n",
                "\n",
                "# Print 1st and 90th quantiles respectively\n",
                "cat(arrdelay_01pcntile, \"\\n\", arrdelay_90pcntile)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now that we have quantiles corresponding to 1% and 90%, let's filter the `df_flights` data to only include rows whose Arrival delay falls within this range.\n",
                "\n",
                "**Question 5.** Starting with the `df_flights` data, filter to only include rows whose **ArrDelay** falls within 1st and 90th quantiles. Assign this to a variable name `df_flights`.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Filter data to remove outliers\n",
                "df_flights <- df_flights %>%\n",
                "  filter(ArrDelay > ...., ....)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 5.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Now, let's do the same for `DepDelay` column.\n",
                "\n",
                "**Question 6.** Starting with the `df_flights` data, obtain quantiles corresponding to 1% and 90%. Assign these values to the variable names `depdelay_01pcntile` and `depdelay_90pcntile` respectively.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Trim outliers for DepDelay based on 1% and 90% percentiles\n",
                "# Produce quantiles corresponding to 1% and 90%\n",
                "depdelay_01pcntile <- df_flights %>%\n",
                "  .... %>%\n",
                "  ....\n",
                "\n",
                "depdelay_90pcntile <- df_flights %>%\n",
                "  .... %>%\n",
                "  ....\n",
                "\n",
                "# Print 1st and 90th quantiles respectively\n",
                "cat(depdelay_01pcntile, \"\\n\", depdelay_90pcntile)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 6.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Good job!!\n",
                "\n",
                "Now that we have quantiles corresponding to 1% and 90%, let's filter the `df_flights` data to only include rows whose Departure delay falls within this range.\n",
                "\n",
                "**Question 7.** Starting with the `df_flights` data, filter to only include rows whose **DepDelay** falls within 1st and 90th quantiles. Assign this to a variable name `df_flights`.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Filter data to remove outliers\n",
                "df_flights <- df_flights %>%\n",
                "  ....\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 7.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "You rock!!\n",
                "\n",
                "Now, we can check the distribution of our two variables with outliers removed.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Distribution of DepDelay\n",
                "show_distribution(var_data = select(df_flights, DepDelay), binwidth = 2)\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Distribution of ArrDelay\n",
                "show_distribution(var_data = select(df_flights, ArrDelay), binwidth = 2)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Much better!\n",
                "\n",
                "Now that the data is all cleaned up, we can begin doing some exploratory analysis.\n",
                "\n",
                "## **Explore the data**\n",
                "\n",
                "Let's start with an overall view of the summary statistics for the numeric columns.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Obtain common summary statistics using summarytools package\n",
                "df_flights %>%\n",
                "  descr(stats = \"common\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **What are the mean departure and arrival delays?**\n",
                "\n",
                "**Question 8.** Starting with the `df_flights` data, use `across()` within `summarize()` to find the mean across `DepDelay` and `ArrDelay` columns. Assign this to `df_delays` variable name. What are the mean delays?\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Summarise the departure and arrival delays by finding the mean\n",
                "df_delays <- df_flights %>%\n",
                "  summarise(across(....))\n",
                "\n",
                "df_delays\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 8.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **How do the carriers compare in terms of arrival delay performance?**\n",
                "\n",
                "A box plot can be a good way for graphically depicting the distribution of groups of numerical data through their quantiles. The `geom` that takes care of box plots is `geom_boxplot`\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Compare arrival delay across different carriers\n",
                "df_flights %>%\n",
                "  ggplot() +\n",
                "  geom_boxplot(mapping = aes(x = Carrier, y = ArrDelay))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **How do the carriers compare in terms of departure delay performance?**\n",
                "\n",
                "Let's do the same for the departure delay performance.\n",
                "\n",
                "We can also try and rearrange the `Carrier` levels in ascending order of the delay time and sprinkle some color to the plots too.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "df_flights %>%\n",
                "  mutate(Carrier = fct_reorder(Carrier, DepDelay)) %>%\n",
                "  ggplot() +\n",
                "  geom_boxplot(mapping = aes(x = Carrier, y = DepDelay, color = Carrier),\n",
                "  show.legend = FALSE)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Alternatively, to create the above plots, we can use `purr::map()` which allows us to apply a function to each column. See `?map` for more details.\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "map(df_flights %>% select(ArrDelay, DepDelay), ~ ggplot(df_flights) +\n",
                "  geom_boxplot(mapping = aes(x = Carrier, y = .x)) + ylab(\"\"))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **Are some days of the week more prone to arrival delays than others?**\n",
                "\n",
                "Again, let's make use of a box plot to visually inspect the distribution of arrival delays depending on the day of the week. To successfully accomplish this, we have to first encode the week days to `categorical` variables, i.e. `factors` .\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Encode day of the week as a categorical and make boxplots\n",
                "df_flights %>%\n",
                "  mutate(DayOfWeek = factor(DayOfWeek)) %>%\n",
                "  ggplot() +\n",
                "  geom_boxplot(mapping = aes(x = DayOfWeek, y = ArrDelay),\n",
                "  show.legend = FALSE)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **Are some days of the week more prone to departure delays than others?**\n",
                "\n",
                "Now, over to you.\n",
                "\n",
                "**Question 9.** Let's investigate whether some days of the week (x axis) are more prone to departure delays (y axis) than others. Start by encoding day of the week as a categorical variable.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Encode day of the week as a categorical variable\n",
                "df_flights <- df_flights %>%\n",
                "  mutate(....)\n",
                "\n",
                "# Make a box plot of DayOfWeek and DepDelay\n",
                "dep_delay_plot <- df_flights %>%\n",
                "  ggplot() +\n",
                "  geom_boxplot(mapping = aes(x = ...., y = ...., color = DayOfWeek),\n",
                "  show.legend = FALSE) +\n",
                "  scale_color_brewer(palette = \"Dark2\")\n",
                "\n",
                "dep_delay_plot\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "What can you make out of this?\n",
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 9.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Great progress you are having!\n",
                "\n",
                "### **Which departure airport has the highest average departure delay?**\n",
                "\n",
                "To answer this, we have to first **group** the data **by** `OriginAirportName` and then **summarize** the observations by the **mean** of their Departure delay `DepDelay` and then **arrange** this in **descending** order of the mean departure delays.\n",
                "\n",
                "Let's put this into code.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Use group_by %>% summarize to find airports with highest avg DepDelay\n",
                "mean_departure_delays <- df_flights %>%\n",
                "  group_by(OriginAirportName) %>%\n",
                "  summarize(mean_dep_delay_time = mean(DepDelay)) %>%\n",
                "  arrange(desc(mean_dep_delay_time))\n",
                "\n",
                "# Print the first 7 rows\n",
                "mean_departure_delays %>%\n",
                "  slice_head(n = 7)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Fantastic!\n",
                "\n",
                "Let's represent this using bar plots.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "mean_departure_delays %>%\n",
                "  # Sort factor levels in descending order of delay time\n",
                "  mutate(OriginAirportName = fct_reorder(OriginAirportName,\n",
                " desc(mean_dep_delay_time))) %>%\n",
                "  ggplot() +\n",
                "  geom_col(mapping = aes(x = OriginAirportName, y = mean_dep_delay_time),\n",
                " fill = \"midnightblue\", alpha = 0.7) +\n",
                "  theme(\n",
                "    # Rotate X markers so we can read them\n",
                "    axis.text.x = element_text(angle = 90)\n",
                "  )\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Could you try and guess why Chicago Airport has most departure delay time or why Long Beach has the least?\n",
                "\n",
                "### **Do late departures tend to result in longer arrival delays than on-time departures?**\n",
                "\n",
                "**Question 10.** Starting with the `df_flights` data, first encode `DepDel15` column (a binary indicator that departure was delayed by more than 15 minutes) as categorical.\n",
                "\n",
                "Use a **box plot** to investigate whether late departures (x-axis) tend to result in longer arrival delays (y-axis) than on-time departures. Map the fill aesthetic to the `DepDel15` variable.\n",
                "\n",
                "> You can colour a box plot using either the `colour` aesthetic (like in the previous exercises), or, more usefully with the `fill` aesthetic.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Encode DepDel15 as a categorical variable\n",
                "df_flights <- df_flights %>%\n",
                "  mutate(DepDel15 = factor(DepDel15))\n",
                "\n",
                "arr_delay_plot <- df_flights %>%\n",
                "  ggplot() +\n",
                "  geom_boxplot(mapping <- aes(x = ...., y = ...., fill = ....))\n",
                "\n",
                "arr_delay_plot\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Does this surprise you?\n",
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 10.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **Which route (from origin airport to destination airport) has the most late arrivals?**\n",
                "\n",
                "Finally, let's investigate travel routes. We'll start by adding a column `Route` that indicates the Origin and Destination airports.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Add a \"Route\" column\n",
                "df_flights <- df_flights %>%\n",
                "  mutate(Route = paste(OriginAirportName, DestAirportName, sep = \">\"))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Great! Now we can use `group_by()`, `summarize()` and `arrange()` to find the routes with the most late arrivals\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Make grouped summaries to find the total delay\n",
                "# associated with a particular route\n",
                "df_flights %>%\n",
                "  group_by(Route) %>%\n",
                "  summarize(ArrDel15 = sum(ArrDel15)) %>%\n",
                "  arrange(desc(ArrDel15))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### **Which route has the highest average arrival delay time?**\n",
                "\n",
                "Over to you!\n",
                "\n",
                "**Question 11.** Starting with the `df_flights` data, group the observations by `Route` then create a summary tibble with a column name `ArrDelay` which represents the mean arrival delay time. Arrange this in descending order.\n",
                "\n",
                "Assign your results to a variable name `df_route_arrdelay`.\n",
                "\n",
                "Fill in the placeholder `....` with the right code."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Create grouped summaries of the arrival delay time\n",
                "df_route_arrdelay <- df_flights %>%\n",
                "  .... %>%\n",
                "  summarise(ArrDelay = ....) %>%\n",
                "  ....(desc(....))\n",
                "\n",
                "\n",
                "# Print the first 5 rows\n",
                "df_route_arrdelay %>%\n",
                "  slice_head(n = 5)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Test your answer:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                ". <- ottr::check(\"tests/Question 11.R\")\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Congratulations on finishing the first challenge! We'll wrap it at that for now. Of course there are other ways to approach this challenge. So please feel free to experiment, google and share your solutions with friends.\n",
                "\n",
                "See you in the next module where we get started with Machine Learning!\n",
                "\n",
                "Happy Learning,\n",
                "\n",
                "[Eric,](https://twitter.com/ericntay) Gold Microsoft Learn Student Ambassador.\n"
            ]
        }
    ],
    "metadata": {
        "anaconda-cloud": "",
        "kernelspec": {
            "display_name": "R",
            "langauge": "R",
            "language": "R",
            "name": "ir"
        },
        "language_info": {
            "codemirror_mode": "r",
            "file_extension": ".r",
            "mimetype": "text/x-r-source",
            "name": "R",
            "pygments_lexer": "r",
            "version": "4.1.2"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}