workflow/4_dataset_agreement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c40e77ba-d6e7-4f9c-93cb-43ecd70f7f60",
   "metadata": {},
   "source": [
    "# Agreement Evaluation between Data Sets\n",
    "\n",
    "Now that we have computed the error variances in the [TC notebook](2_TC_application.ipynb) and covariances in the [EC notebook](3_EC_application.ipynb), let's compare the differences (i.e., relative bias) between data sets over all time/seasons. We can then compare this relative bias to the estimated errors to see if the data sets are in statistical agreement. If the data sets are not in agreement based off their relative bias and random errors, then the choice of ET data set could have implications and propagated biases on resulting products modeled from the chosen ET data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dc5892d-4c23-4647-8758-26382db9d4d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hvplot.xarray\n",
    "import panel as pn\n",
    "import numpy as np\n",
    "import xarray as xr\n",
    "from xarray_einstats import linalg, stats\n",
    "from scipy.stats import norm\n",
    "import itertools\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33df4dc4-b6e3-40eb-a198-528766f10d0a",
   "metadata": {},
   "source": [
    "## Combine Data Sets in Xarray\n",
    "First, we need to load in our ET data sets and limit them to a common date range. Since biases will be between two data sets, we will restrict the data ranges of all data sets to have the beginning date of the second oldest starting date and ending data of the second most recent ending date. This choice allows us to save some memory usage, while also utilizing the largest amount of data. When computing biases for data sets with a more restricted date range, the missing data should propagate and not give us a value on those dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d532e7dc-7f1d-4080-b40d-7922a29259c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "files = [\n",
    "    \"../Data/ssebop/ssebop_aet_regridded.nc\",\n",
    "    \"../Data/gleam/gleam_aet.nc\",\n",
    "    \"../Data/era5/era5_aet_regridded.nc\",\n",
    "    \"../Data/nldas/nldas_aet_regridded.nc\",\n",
    "    \"../Data/terraclimate/terraclimate_aet_regridded.nc\",\n",
    "    \"../Data/wbet/wbet_aet_regridded.nc\",\n",
    "]\n",
    "dataset_name = [\"SSEBop\", \"GLEAM\", \"ERA5\", \"NLDAS\", \"TerraClimate\", \"WBET\"]\n",
    "\n",
    "date_ranges = {}\n",
    "for file, name in zip(files, dataset_name):\n",
    "    ds_temp = xr.open_dataset(\n",
    "        file, engine=\"netcdf4\", chunks={\"lon\": -1, \"lat\": -1, \"time\": -1}\n",
    "    )\n",
    "    date_ranges[name] = [ds_temp.time.min().values, ds_temp.time.max().values]\n",
    "\n",
    "# Take the second oldest start and second most recent end dates\n",
    "date_range = [\n",
    "    np.sort(np.array(list(date_ranges.values()))[:, 0])[1],\n",
    "    np.sort(np.array(list(date_ranges.values()))[:, 1])[-2],\n",
    "]\n",
    "date_range"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d64ebc9-ac5e-40ae-b0ad-7151943a51c1",
   "metadata": {},
   "source": [
    "Using the date range, we can now combine all of the data sets into a single `xarray.DataSet` for easy computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b880c50e-d29c-46e6-a338-39860d1aa474",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(ds):\n",
    "    \"\"\"\n",
    "    Keep only the specified time range for each file.\n",
    "    \"\"\"\n",
    "    return ds.sel(time=slice(date_range[0], date_range[1]))\n",
    "\n",
    "\n",
    "ds = xr.open_mfdataset(\n",
    "    files,\n",
    "    engine=\"netcdf4\",\n",
    "    preprocess=preprocess,\n",
    "    combine=\"nested\",\n",
    "    concat_dim=\"dataset_name\",\n",
    ")\n",
    "ds = ds.assign_coords({\"dataset_name\": dataset_name})\n",
    "ds.dataset_name.attrs[\"description\"] = \"Dataset name\"\n",
    "\n",
    "# Need time as first index for TC computation\n",
    "ds = ds.transpose(\"time\", ...)\n",
    "# The data set is less than 1GiB, so let's read it into memory vs keeping as a dask array\n",
    "ds = ds.compute()\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8a730d-5436-41e7-9d7e-e6f1c9a79284",
   "metadata": {},
   "source": [
    "## Relative Bias Estimation\n",
    "\n",
    "Next, we will want to compute the relative bias for all 15 possible pairs of our six data sets. So, let's generate those pairs or combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e79f241-28b3-4964-9c27-f58e15dc1669",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a list of the combinations\n",
    "combos = list(itertools.combinations(dataset_name, 2))\n",
    "combos = [list(combo) for combo in combos]\n",
    "combos"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8056b43-5244-4eb6-b245-81b52ee44c5d",
   "metadata": {},
   "source": [
    "Now that we have our data set combinations, let's compute the relative biases!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ed0db85-2221-4de0-8e5e-2f0bc042e61b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_bias = []\n",
    "for combo in combos:\n",
    "    ds_combo = ds.sel(dataset_name=combo)\n",
    "\n",
    "    da_combo_bias = (\n",
    "        ds_combo.aet\n",
    "        .diff(dim=\"dataset_name\")\n",
    "        .squeeze(dim=\"dataset_name\")\n",
    "        .drop_vars(\"dataset_name\")\n",
    "        .expand_dims(dataset_pair=[\" \".join(combo)])\n",
    "        .to_dataset()\n",
    "        .rename({\"aet\": \"rel_bias\"})\n",
    "    )\n",
    "\n",
    "    ds_bias.append(da_combo_bias)\n",
    "\n",
    "ds_bias = xr.concat(ds_bias, dim=\"dataset_pair\")\n",
    "\n",
    "ds_bias.rel_bias.attrs[\"description\"] = (\n",
    "    \"Relative bias (i.e., difference) between two data sets listed in dataset_pairs\"\n",
    ")\n",
    "ds_bias.dataset_pair.attrs[\"description\"] = \"Dataset pair used in difference.\"\n",
    "ds_bias.rel_bias.attrs[\"units\"] = \"mm.month-1\"\n",
    "\n",
    "ds_bias"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c90e36-cb5c-434a-9ce6-68b2fb89e565",
   "metadata": {},
   "source": [
    "Now, let's see how the resulting biases look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8e8c0fc-2036-423b-9a47-70cd1f88e27c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt = ds_bias.rel_bias.hvplot(\n",
    "    groupby=[\"dataset_pair\", \"time\"],\n",
    "    geo=True,\n",
    "    coastline=True,\n",
    "    clim=(-75, 75),\n",
    "    cmap=\"PuOr\",\n",
    ").opts(frame_width=500)\n",
    "\n",
    "pn.panel(plt, widget_location=\"top\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c5299e-980b-441b-bb39-3558ce5fdfe8",
   "metadata": {},
   "source": [
    "## Data Set Agreement Discussion\n",
    "\n",
    "Looking at the biases, we can see a large temporal variation in each estimate. However, while being able to check this relative bias temporally is in itself interesting, our goal is to compare the bias with the errors estimated from EC. Therefore, a single bias product that is averaged over time like the EC error estimates will make this comparison easier and more consistent. To that end, we will temporally average the bias estimates (both over all time and each season) and use these averages to compare with the errors.\n",
    "\n",
    "Since the EC error covariance matrix estimates utilized a more limited date range than the current bias estimates (date range limited by four data sets versus two), we limit the temporal average of the biases to use the same date range restriction as used in the EC estimates for consistency. To do this, we will make a function to select the common date range from a list of data set quadruplets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e61f67d5-b053-4441-85a7-b0d08aab9ac9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def common_date_range(ds, combo):\n",
    "    \"\"\"Return the common date slice of the datasets.\"\"\"\n",
    "    old_common_date = []\n",
    "    recent_common_date = []\n",
    "    for name in combo:\n",
    "        old_common_date.append(date_ranges[name][0])\n",
    "        recent_common_date.append(date_ranges[name][1])\n",
    "\n",
    "    return slice(np.max(old_common_date), np.min(recent_common_date))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eedfd38e-7295-42e4-9f3f-acb42caa833f",
   "metadata": {},
   "source": [
    "Now that we have the common date range function, we can calculate the average biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc044649-ad03-4df6-a12a-41d18c01f839",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create list of seasons\n",
    "seasons = [\"All\"] + list(np.unique(ds.time.dt.season))\n",
    "\n",
    "# Read in the EC results to get the combo list for limiting the dates\n",
    "ec_est_averages = xr.open_dataset(\"../Data/EC_errs.nc\")\n",
    "ec_est_averages = ec_est_averages.rename({\"covar_pair\": \"dataset_pair\"})\n",
    "\n",
    "ds_mean_bias = []\n",
    "ds_median_bias = []\n",
    "ds_var_bias = []\n",
    "ds_count_bias = []\n",
    "est_pair = []\n",
    "for season in seasons:\n",
    "    if season == \"All\":\n",
    "        ds_season = ds_bias\n",
    "    else:\n",
    "        ds_season = ds_bias.isel(time=(ds_bias.time.dt.season == season))\n",
    "\n",
    "    for dataset_pair in ec_est_averages.dataset_pair.data:\n",
    "        # Let's keep track of the other two data sets used to limit the date range\n",
    "        if season == \"All\":\n",
    "            est_pair.append(\n",
    "                ec_est_averages.sel(dataset_pair=dataset_pair).est_pair.drop_vars(\n",
    "                    \"est_pair\"\n",
    "                )\n",
    "            )\n",
    "\n",
    "        for i, indep_pair in enumerate(\n",
    "            ec_est_averages.sel(dataset_pair=dataset_pair).est_pair.data\n",
    "        ):\n",
    "            combo = (dataset_pair + \" \" + indep_pair).split()\n",
    "            ds_combo = ds_season.sel(\n",
    "                time=common_date_range(ds_bias, combo), dataset_pair=dataset_pair\n",
    "            ).drop_vars(\"dataset_pair\")\n",
    "\n",
    "            mean_bias = ds_combo.rel_bias.mean(\n",
    "                dim=\"time\", skipna=True, keep_attrs=True\n",
    "            ).expand_dims(season=[season], est_idx=[i], dataset_pair=[dataset_pair])\n",
    "            mean_bias.name = \"mean_bias\"\n",
    "            mean_bias.attrs[\"description\"] = (\n",
    "                \"Mean bias estimate for all common time steps between data sets.\"\n",
    "            )\n",
    "            mean_bias.attrs[\"units\"] = \"mm.month-1\"\n",
    "            ds_mean_bias.append(mean_bias)\n",
    "\n",
    "            median_bias = ds_combo.rel_bias.median(\n",
    "                dim=\"time\", skipna=True, keep_attrs=True\n",
    "            ).expand_dims(season=[season], est_idx=[i], dataset_pair=[dataset_pair])\n",
    "            median_bias.name = \"median_bias\"\n",
    "            median_bias.attrs[\"description\"] = (\n",
    "                \"Median bias estimate for all common time steps between data sets.\"\n",
    "            )\n",
    "            median_bias.attrs[\"units\"] = \"mm.month-1\"\n",
    "            ds_median_bias.append(median_bias)\n",
    "\n",
    "            var_bias = ds_combo.rel_bias.var(\n",
    "                dim=\"time\", ddof=1, skipna=True, keep_attrs=True\n",
    "            ).expand_dims(season=[season], est_idx=[i], dataset_pair=[dataset_pair])\n",
    "            var_bias.name = \"var_bias\"\n",
    "            var_bias.attrs[\"description\"] = (\n",
    "                \"Temporal variance of the bias estimates for all common \"\n",
    "                \"time steps between data sets.\"\n",
    "            )\n",
    "            var_bias.attrs[\"units\"] = \"mm.month-1\"\n",
    "            ds_var_bias.append(var_bias)\n",
    "\n",
    "            count_bias = (\n",
    "                np.isfinite(ds_combo.rel_bias)\n",
    "                .sum(dim=\"time\")\n",
    "                .expand_dims(season=[season], est_idx=[i], dataset_pair=[dataset_pair])\n",
    "            )\n",
    "            count_bias.name = \"counts\"\n",
    "            count_bias.attrs[\"description\"] = (\n",
    "                \"Number of data points used in the average bias \"\n",
    "                \"estimates (i.e., number of finite time values in a given pixel).\"\n",
    "            )\n",
    "            count_bias.attrs[\"units\"] = \"counts\"\n",
    "            ds_count_bias.append(count_bias)\n",
    "\n",
    "ds_mean_bias = xr.combine_by_coords(ds_mean_bias)\n",
    "ds_median_bias = xr.combine_by_coords(ds_median_bias)\n",
    "ds_var_bias = xr.combine_by_coords(ds_var_bias)\n",
    "ds_count_bias = xr.combine_by_coords(ds_count_bias)\n",
    "\n",
    "# Compile these DataSets into one and save\n",
    "bias_averages = xr.merge(\n",
    "    [ds_mean_bias, ds_median_bias, ds_var_bias, ds_count_bias], join=\"exact\"\n",
    ")\n",
    "\n",
    "# Include the list of the other two data sets used in the date range\n",
    "bias_averages = bias_averages.assign_coords(\n",
    "    est_pair=xr.concat(est_pair, dim=\"dataset_pair\")\n",
    ")\n",
    "\n",
    "bias_averages.est_pair.attrs[\"description\"] = (\n",
    "    \"Names of the other two data sets used in the date range selection.\"\n",
    ")\n",
    "bias_averages.season.attrs[\"description\"] = (\n",
    "    \"Season of the year given by the first letter of \"\n",
    "    'each month within the season. The full year is given by \"All\".'\n",
    ")\n",
    "bias_averages.est_idx.attrs[\"description\"] = (\n",
    "    \"Index of the other two data sets used \" \"in the date range selection in est_pair.\"\n",
    ")\n",
    "bias_averages.dataset_pair.attrs[\"description\"] = (\n",
    "    \"Data set pair used in relative bias evaluation.\"\n",
    ")\n",
    "\n",
    "# Match the index of the EC results\n",
    "bias_averages = bias_averages.reindex_like(ec_est_averages)\n",
    "\n",
    "if not os.path.isfile(\"../Data/avg_bias.nc\"):\n",
    "    _ = bias_averages.to_netcdf(\n",
    "        path=\"../Data/avg_bias.nc\", format=\"NETCDF4\", engine=\"netcdf4\"\n",
    "    )\n",
    "\n",
    "bias_averages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60159322-1f36-47d0-8e11-0636a9e6387b",
   "metadata": {},
   "source": [
    "Now that we have our average biases, let's generate some plots that show how the biases compare to the errors. If we see that the bias between the data sets is in agreement within the errors, then it indicates that the uncertainty in ET data sets are more important than the relative biases between them. Conversely, they do not agree, then the choice of ET data set could have implications and propagated biases on resulting products modeled from the ET data.\n",
    "\n",
    "To show this comparison and check for agreement, we can use an analytical method, since EC assumes a normal distribution for the errors. This method consists of:\n",
    "\n",
    "1. Calculate the absolute relative bias ($\\textrm{bias}_\\textrm{abs} = | \\textrm{dataset}_A - \\textrm{dataset}_B |$),\n",
    "2. [Propagate the data sets' error variances and covariance](https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulae) to the bias uncertainty ($\\sigma_{\\varepsilon_{\\rm bias}} = \\sqrt{\\sigma_{\\varepsilon_A}^2 + \\sigma_{\\varepsilon_B}^2 - 2 \\sigma_{\\varepsilon_{AB}}}$),\n",
    "3. Assuming a normal distribution of mean $\\textrm{bias}_\\textrm{abs}$ and variance $\\sigma_{\\varepsilon_{\\rm bias}}^2$, calculate the probability density of this bias distribution that is less than or equal 0 (a bias of 0 would indicate the data sets are the same).\n",
    "\n",
    "> We use the absolute relative bias versus the relative bias since we don't care which data set is larger than the other. Rather we just want to know how well they agree within their errors.\n",
    "\n",
    "With a given probability density, we can then estimate agreement from its value, since it is the probability that the absolute difference between the two data sets is less than or equal to 0. If the probability is low (i.e., say <0.1 or whatever significance level we choose), then the data sets are likely not in agreement. If the probability is near 0.5, then the data sets are in good agreement (i.e., the distribution is relatively equally spread around 0). Given that we have bias and error maps, we will be able to calculate probability densities for each pixel in the maps. Therefore, we can check the agreement of each data set pair across CONUS from these (what we will term) \"agreement probability maps\".\n",
    "\n",
    "To run this method and calculate the agreement probability maps, we will first merge the average EC `Dataset` generated in the [EC notebook](3_EC_application.ipynb#EC-Discussion) with the average biases `Dataset` to get a unified data set for the density calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bcf5587-8210-4b9e-8056-543ede3ef34c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the cross-correlation variable as we don't need it\n",
    "ec_est_averages = ec_est_averages.drop_vars([\"rho\"])\n",
    "# Rename the covar_pair to match the bias dataset_pair\n",
    "ec_est_averages = ec_est_averages.rename(\n",
    "    {\"covar_pair_idx_1\": \"dataset_pair_idx_1\", \"covar_pair_idx_2\": \"dataset_pair_idx_2\"}\n",
    ")\n",
    "\n",
    "bias_ec_averages = xr.merge(\n",
    "    [bias_averages.rename({\"counts\": \"counts_bias\"}), ec_est_averages]\n",
    ")\n",
    "bias_ec_averages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0532a4fb-49be-4588-a4dc-7cfa8532d705",
   "metadata": {},
   "source": [
    "Now that we have the error covariance matrices matched with the bias estimates, we can calculate the probabilities for checking agreement using the method outlined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7daac2c0-3fc1-42d7-82df-811f99705aba",
   "metadata": {},
   "outputs": [],
   "source": [
    "variances = bias_ec_averages[\"covar\"].linalg.diagonal(\n",
    "    dims=[\"dataset_pair_idx_1\", \"dataset_pair_idx_2\"],\n",
    "    offset=0,\n",
    ")\n",
    "covariances = (\n",
    "    bias_ec_averages[\"covar\"]\n",
    "    .linalg.diagonal(\n",
    "        dims=[\"dataset_pair_idx_1\", \"dataset_pair_idx_2\"],\n",
    "        offset=1,\n",
    "    )\n",
    "    .squeeze()\n",
    ")\n",
    "bias_ec_averages[\"sigma_bias\"] = np.sqrt(\n",
    "    variances.sum(dim=\"dataset_pair_idx_1\") - 2 * covariances\n",
    ")\n",
    "\n",
    "norm_dist = stats.XrContinuousRV(\n",
    "    norm,\n",
    "    loc=np.abs(bias_ec_averages[\"median_bias\"]),\n",
    "    scale=bias_ec_averages[\"sigma_bias\"],\n",
    ")\n",
    "# Set and name it as a DataArray as the attributes of the coordinates are not kept\n",
    "agreement_probability = norm_dist.cdf(0)\n",
    "agreement_probability.name = \"agreement_probability\"\n",
    "\n",
    "# Merge to preserve coordinate attributes\n",
    "bias_ec_averages = xr.merge([bias_ec_averages, agreement_probability])\n",
    "bias_ec_averages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483de3e8-ea51-406b-805f-a59e6d09a730",
   "metadata": {},
   "source": [
    "Finally, it is time to generate some plots that show how the biases compare to the errors. We will make a few plots (1) the agreement probabilities, (2) median bias, (3) the median agreement probabilities across the multiple covariance estimates, and (4) the \"SNR\" (median/std) of the agreement probabilities across the multiple covariance estimates. These last two should give us a good overview of how the agreement probabilities vary across CONUS as there will be plenty of `NaN` values in the covariance matricies as discussed in the [EC notebook](3_EC_application.ipynb#EC-Discussion). So, let's focus our discussion on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7980a4c7-c869-47d5-8d4b-50c1db02a11b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt = (\n",
    "    (\n",
    "        bias_ec_averages[\"agreement_probability\"]\n",
    "        .hvplot(\n",
    "            groupby=[\"season\", \"dataset_pair\", \"est_idx\"],\n",
    "            geo=True,\n",
    "            coastline=True,\n",
    "            clim=(0, 0.5),\n",
    "            cmap=\"Purples\",\n",
    "            title=\"Agreement Probability\",\n",
    "        )\n",
    "        .opts(frame_width=500)\n",
    "    )\n",
    "    + (\n",
    "        bias_ec_averages[\"agreement_probability\"]\n",
    "        .median(dim=\"est_idx\")\n",
    "        .rename(\"median_agreement_probability\")\n",
    "        .hvplot(\n",
    "            groupby=[\"season\", \"dataset_pair\"],\n",
    "            geo=True,\n",
    "            coastline=True,\n",
    "            clim=(0, 0.5),\n",
    "            cmap=\"Purples\",\n",
    "            title=\"Median Agreement Probability\",\n",
    "        )\n",
    "        .opts(frame_width=500)\n",
    "    )\n",
    "    + (\n",
    "        bias_ec_averages[\"median_bias\"]\n",
    "        .hvplot(\n",
    "            groupby=[\"season\", \"dataset_pair\", \"est_idx\"],\n",
    "            geo=True,\n",
    "            coastline=True,\n",
    "            clim=(-50, 50),\n",
    "            cmap=\"PuOr\",\n",
    "            title=\"Relative Bias\",\n",
    "        )\n",
    "        .opts(frame_width=500)\n",
    "    )\n",
    "    + (\n",
    "        np.log10(\n",
    "            bias_ec_averages[\"agreement_probability\"].median(dim=\"est_idx\")\n",
    "            / bias_ec_averages[\"agreement_probability\"].std(dim=\"est_idx\")\n",
    "        )\n",
    "        .rename(\"log10_snr_agreement_probability\")\n",
    "        .hvplot(\n",
    "            groupby=[\"season\", \"dataset_pair\"],\n",
    "            geo=True,\n",
    "            coastline=True,\n",
    "            clim=(-2, 2),\n",
    "            cmap=\"PuOr\",\n",
    "            title=\"log10(SNR of Agreement Probability [Median/Std])\",\n",
    "        )\n",
    "        .opts(frame_width=500)\n",
    "    )\n",
    ")\n",
    "\n",
    "pn.panel(plt.cols(2), widget_location=\"top\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e509b93b-dbd1-4313-a3c1-8d671e8cc8b1",
   "metadata": {},
   "source": [
    "Two things to reiterate when looking at the median agreement probabilities is that the errors used in its calculation are likely lower limits on the uncertainty and the bias is a temporal median. Therefore, if we see a good agreement between data sets, this is likely true for the majority of the monthly ET data (i.e., 50% of the monthly relative bias is below the median). Additionally, if there is a lack of agreement, then it is still possible the data sets agree if the errors are drastically underestimated.\n",
    "\n",
    "With that in mind, looking at the agreement probabilites for each data set we can see that overall the SNR is relatively high, indicating the agreement is consistent across different EC estimates. Individually:\n",
    "\n",
    "1. **SSEBop** - The agreement probability with all data sets is typically high across CONUS, except in mountainous regions like the Rocky Mountains. Looking at seasonal data, winter shows minimal agreement with the other data sets. However, that is expected as winter typically has almost no ET, and therefore has small errors, which make agreement difficult. As for spring, summer, and fall, the agreement patterns vary, with spring mainly having agreement in the Southeast, summer having agreement in the Midwest, and fall having agreement in the East to Southeast for with the other data sets.\n",
    "\n",
    "2. **GLEAM** - The agreement probability with all data sets is typically high for all of the CONUS, excluding the Pacific Northwest, where the probability can be below 0.05. As for seasonal data, the spatial pattern of agreement is not very consistent across data sets in the summer and spring, with some data sets showing agreement in southern CONUS, while others do not. However, most data sets show very low agreement in northern CONUS during these seasons. Looking at fall, most data sets show agreement in the South to Southeast, but still show low agreement probabilites in the North.\n",
    "\n",
    "3. **ERA5** - Overall, the agreement probability with all data sets is typically high for all of the CONUS, with some lower agreement happening across CONUS with NLDAS. Seasonal subsets show high agreement in the summer and fall for most data sets in the South to Southeast. However, agreement is low in the northern half of CONUS during these months, as well as practically all of CONUS in the spring.\n",
    "\n",
    "4. **NLDAS** - The agreement probability is spatially varied when comparing between the other data sets. However, most data sets show agreement across CONUS at reasonable probability levels (i.e., >0.1). Some lower agreement probabilities across data sets occurs in eastern CONUS, especially the southeastern coast. In terms of seasonal variation, the southwest and centeral US typically have high agreement in the summer and fall, but the majority of CONUS shows low agreement in the spring.\n",
    "\n",
    "5. **TerraClimate** - Overall, the agreement probability with all data sets is high for all of the CONUS, with relatively lower (but still reasonably high) probabilities in the Southeast and Pacific Northwest. Across seasons, we typically see low agreement in all but the Southwest during spring. Agreement is more spatially varied in the summer, with some data sets showing more agreement in the North and others the South. As for fall, most data sets show excellent agreement in the western half of CONUS, with mixed agreement in the eastern half.\n",
    "\n",
    "6. **WBET** - On average, the agreement probabilities are high across CONUS, but there is some variation between data sets. The seasonal subsets follow a similar trend, with agreement varying spatially between data sets. Spring shows the lowest overall agreement, which only has strong agreement in the extreme Southwest and Florida Peninsula. Summer shows better agreement, primarily in the MidWest. Finally, fall shows overall high agreement between data sets for most of CONUS except with SSEBop and TerraClimate, which show low agreement in the western half and eastern half of CONUS, respectively.\n",
    "\n",
    "From these summaries, we can draw a few conclusions about the agreement between the data sets. In general, most data sets agree across CONUS when not separating the data by seasons. However, each have certain regions where the agreement with the other data sets is low. These regions indicate one of two things with the data set, either 1) the lack of consensus with the other data sets shows the ET data set is truly biased in these regions, or 2) the uncertainty is underestimated in these regions. Either way, the lack of agreement indicates that the ET data set is not optimally performing in these given regions. As for seasonalities (excluding winter as low agreement is expected as stated above), there is a lot of variation in agreement based on the data set and the corresponding bias pair. In general though, it appears that spring has the lowest overall agreement, increasing into summer and further increasing in the fall. Again, each data set has certain regions where the seasonal agreement with the other data sets is high and low. Therefore, we will explore this regional variation further in a [Regional Analysis notebook](5_regional_analysis.ipynb), where we explore how each ET data sets performs across three regions in CONUS where quality ET data is critical to hydrological modeling and water availability."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}