workflow/0_compile_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bdf842e6-d027-4526-8452-99f49681bb7f",
   "metadata": {},
   "source": [
    "# Compile ET Monthly Data Sets\n",
    "\n",
    "Below are the Python codes needed to download and process six ET datasets with monthly time steps. On my Dell Precision 3570 with a 12th Gen Intel i7-1255U 1.70 GHz processor and a 250Mb/s internet connection, each of these datasets takes less than 15 minutes to download and process, except for the WBET data set. This data set can take ~24 hours to fully process due to the high resolution and large file sizes. While the other datasets use at most 2GB of disk space each, the WBET data set can use ~100GB at peak usage and ~50GB for the final processed file. This can be reduced if the full date range is not utilized.\n",
    "\n",
    "> NOTE: The links and files names on the web services are accurate as of 10/04/2024. It is possible that the location of the data has been updated, the web service has changed the path to the data set, or the files have been removed since this date. If you try to run these codes and they fail, it is recommended to follow the shared links and confirm the url and download methods have not changed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fa99f2e-2bd3-47e9-ba2e-729dcf01bb81",
   "metadata": {},
   "source": [
    "## SSEBop MODIS Data Set\n",
    "\n",
    "The SSEBop data can be accessed from the [USGS Early Warning and Environmental Monitoring Program](https://earlywarning.usgs.gov/ssebop) (via the [ftp site](https://edcintl.cr.usgs.gov/downloads/sciweb1/shared/uswem/web/conus/eta/modis_eta/monthly/downloads/)). The monthly data is stored in individual GeoTIFFs files that are then zipped for compression. We retrieve the zip files with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/#). We selected to retrieve all of the monthly data for the available years of 2000-2022. The preprocessing steps included extracting the monthly GeoTIFFs from the zip files and combining them all into a single NetCDF file. This process had to be done separately for dates before and after July 2016 as dates after this have fewer lat/lon indices values than those before. To have a unified data set these date groups were processed by aligning and only keeping the lat/lon range common to both date groups. Additionally after July 2016, bodies of water have a fill value of 0 rather than NaN. We replace the 0 fill with NaN using a mask from before July 2016, as 0s can be present on land surfaces in the winter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3749be2-9cdc-412d-835b-e5d2786a014b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# SSEBop Dataset\n",
    "import fsspec\n",
    "import rioxarray\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "import zipfile\n",
    "import io\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "fs = fsspec.filesystem(\"https\", timeout=3600)\n",
    "url = \"https://edcintl.cr.usgs.gov/downloads/sciweb1/shared/uswem/web/conus/eta/modis_eta/monthly/downloads/\"\n",
    "\n",
    "# Monthly date range to use\n",
    "dates = pd.date_range(\"2000-01-01\", \"2022-12-31\", freq=\"MS\")\n",
    "\n",
    "# Make a directory for holding and extracting the zip files\n",
    "if not os.path.isdir(\"../Data/ssebop\"):\n",
    "    os.makedirs(\"../Data/ssebop\")\n",
    "\n",
    "# Read the files to xarray and compile to monthly data\n",
    "for date in dates:\n",
    "    # Download the zipfiles\n",
    "    # (Do this individually vs all files in a list as server may disconnect\n",
    "    # downloading for list)\n",
    "    fs.get(url + \"m\" + date.strftime(\"%Y%m\") + \".zip\", \"../Data/ssebop/\")\n",
    "\n",
    "    # Access zip file without unzipping\n",
    "    zfile = zipfile.ZipFile(\"../Data/ssebop/m\" + date.strftime(\"%Y%m\") + \".zip\")\n",
    "\n",
    "    # Select the GeoTIFF files from zip_file_list and extract\n",
    "    geotiff_file = \"m\" + date.strftime(\"%Y%m\") + \".modisSSEBopETactual.tif\"\n",
    "    zfile.extractall(\"../Data/ssebop/\", [geotiff_file])\n",
    "\n",
    "    # Delete downloaded zipfile as we no longer need it\n",
    "    os.remove(\"../Data/ssebop/m\" + date.strftime(\"%Y%m\") + \".zip\")\n",
    "\n",
    "    # Read in GeoTIFF\n",
    "    ds_monthly = rioxarray.open_rasterio(\n",
    "        f\"../Data/ssebop/{geotiff_file}\",\n",
    "        chunks={},\n",
    "        band_as_variable=True,\n",
    "    )\n",
    "\n",
    "    # Rename coords to corresponding names. Assign the date to the Dataset\n",
    "    ds_monthly = (\n",
    "        ds_monthly.rename({\"x\": \"lon\", \"y\": \"lat\", \"band_1\": \"aet\"})\n",
    "        .assign_coords(time=date)\n",
    "        .expand_dims(dim=\"time\")\n",
    "    )\n",
    "\n",
    "    # Save monthly compiled dataset\n",
    "    ds_monthly.to_netcdf(\n",
    "        path=\"../Data/ssebop/\" + date.strftime(\"%Y-%m\") + \".modisSSEBopETactual.nc\",\n",
    "        engine=\"netcdf4\",\n",
    "    )\n",
    "\n",
    "    # Delete extracted geotiff as we no longer need it\n",
    "    os.remove(f\"../Data/ssebop/{geotiff_file}\")\n",
    "\n",
    "# Read in monthly netcdf Datasets into full Dataset.\n",
    "# Dates after 2016-07 have less lat/lon indices values than those before.\n",
    "# Read these date groups in seperately, slice, and align for concatenating\n",
    "# into single file. Additionally after 2016-07, bodies of water have a fill\n",
    "# value of 0 rather than NaN. Replace the 0 fill with NaN using a mask from\n",
    "# before 2016-07 as 0 can be present on land surface in winter.\n",
    "pre_201608 = pd.date_range(\"2001-01-01\", \"2016-7-31\", freq=\"MS\")\n",
    "ds1 = xr.open_mfdataset(\n",
    "    [\n",
    "        \"../Data/ssebop/\" + date + \".modisSSEBopETactual.nc\"\n",
    "        for date in pre_201608.strftime(\"%Y-%m\")\n",
    "    ],\n",
    "    engine=\"netcdf4\",\n",
    ")\n",
    "\n",
    "post_201608 = pd.date_range(\"2016-08-01\", \"2022-12-31\", freq=\"MS\")\n",
    "ds2 = xr.open_mfdataset(\n",
    "    [\n",
    "        \"../Data/ssebop/\" + date + \".modisSSEBopETactual.nc\"\n",
    "        for date in post_201608.strftime(\"%Y-%m\")\n",
    "    ],\n",
    "    engine=\"netcdf4\",\n",
    ")\n",
    "\n",
    "# reindex lat and lon to match, values are off by floating point rounding errors\n",
    "# Match lat/lon range\n",
    "ds1 = ds1.sel(lon=ds2[\"lon\"], lat=ds2[\"lat\"], method=\"nearest\", tolerance=1e-10)\n",
    "ds1, ds2 = xr.align(ds1, ds2, join=\"override\", exclude=\"time\")\n",
    "\n",
    "# Concatenate, remove spatial_ref var, and convert dtype to float32\n",
    "ds = xr.concat([ds1, ds2], dim=\"time\")\n",
    "ds = ds.drop_vars(\"spatial_ref\")\n",
    "ds = ds.astype(\"float32\")\n",
    "\n",
    "# Set 0 fill values to NaNs\n",
    "ds = ds.where(~np.isnan(ds.aet.isel(time=0)))\n",
    "\n",
    "# Add new metadata attributes\n",
    "ds[\"aet\"].attrs[\"description\"] = (\n",
    "    \"Actual evaporation from SSEBop MODIS, monthly total\"\n",
    ")\n",
    "ds[\"aet\"].attrs[\"dimensions\"] = \"lon lat time\"\n",
    "ds[\"aet\"].attrs[\"standard_name\"] = \"Actual evaporation\"\n",
    "ds[\"aet\"].attrs[\"long_name\"] = \"Actual evaporation\"\n",
    "ds[\"aet\"].attrs[\"units\"] = \"mm.month-1\"\n",
    "\n",
    "# Add some coordinate metadata attributes\n",
    "ds[\"lat\"].attrs[\"units\"] = \"degrees_north\"\n",
    "ds[\"lat\"].attrs[\"description\"] = \"Latitude of the center of the grid cell\"\n",
    "ds[\"lat\"].attrs[\"long_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"standard_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"axis\"] = \"Y\"\n",
    "\n",
    "ds[\"lon\"].attrs[\"units\"] = \"degrees_east\"\n",
    "ds[\"lon\"].attrs[\"description\"] = \"Longitude of the center of the grid cell\"\n",
    "ds[\"lon\"].attrs[\"long_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"standard_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"axis\"] = \"X\"\n",
    "\n",
    "ds[\"time\"].attrs[\"long_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"standard_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"axis\"] = \"T\"\n",
    "\n",
    "# Create chunksizes from coords, with the time dim chunked\n",
    "chunksizes = list(ds.coords.sizes.values())\n",
    "chunksizes[list(ds.coords.sizes).index('time')] = 1\n",
    "\n",
    "# Save full Dataset\n",
    "ds.to_netcdf(\n",
    "    path=\"../Data/ssebop/ssebop_aet.nc\",\n",
    "    format=\"NETCDF4\",\n",
    "    engine=\"netcdf4\",\n",
    "    # Chunk file and compress as is its pretty big uncompressed (~20 GB)\n",
    "    encoding={\"aet\": {\"zlib\": True, \"complevel\": 4, 'chunksizes': chunksizes}},\n",
    ")\n",
    "\n",
    "# Remove intermediate monthly files\n",
    "for date in dates.strftime(\"%Y-%m\"):\n",
    "    os.remove(\"../Data/ssebop/\" + date + \".modisSSEBopETactual.nc\")\n",
    "\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/ssebop/ssebop_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9584dd08-4d51-4b0c-b602-54e8e3c2e42c",
   "metadata": {},
   "source": [
    "## GLEAM Data Set\n",
    "\n",
    "The GLEAM v3.7b data can be accessed from at the [GLEAM website](https://www.gleam.eu/) by signing up for login access to their SFTP site. The monthly data is stored in a single NetCDF file which we retrieve with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/#). We selected to retrieve all of the monthly data for the available years of 2003-2022. The preprocessing steps included (1) limiting the latitude and longitude to CONUS range as the file holds global data (i.e., limit latitudes from $24^\\circ$ to $53^\\circ$ and longitudes from $-126^\\circ$ to $-66^\\circ$); and (2) shifting the month date to be the first of the month vs the end.\n",
    "\n",
    "> NOTE: You will need a [GLEAM Login](https://www.gleam.eu/#downloads) to access this data. Once you have an login, you can use it to access the SFTP site. This is done by setting your username, password, and port as the os enviromental variables (e.g., in your `~/.bashrc` file) of `GLEAM_USERNAME`, `GLEAM_PASSWORD`, and `GLEAM_PORT`, respectively.\n",
    ">\n",
    "> Additionally, since the time of gathering this data set, the GLEAM model has been updated to v4.1. As stated on the GLEAM webpage, \"datasets are typically updated and extended once a year and are generally released around April. When a new version of a dataset is released, the older version becomes obsolete and is removed from the server. However, previous versions are still available upon request.\" Therefore, the v3.7b data is not accessible anymore on the GLEAM server and will have to be requested. Also, if one wants to use the newer 4.1b data set (currently unavailable as of 10/04/2024) the `filepath` below can be updated to the new version as it comes out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f46e3ce-32f2-4028-8381-6ab88aef54fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GLEAM v3.7b Dataset\n",
    "import fsspec\n",
    "import xarray as xr\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "sftp_host = \"sftp://hydras.ugent.be\"\n",
    "gleam_creds_sftp = dict(\n",
    "    username=os.environ[\"GLEAM_USERNAME\"],\n",
    "    password=os.environ[\"GLEAM_PASSWORD\"],\n",
    "    port=int(os.environ[\"GLEAM_PORT\"]),\n",
    ")\n",
    "\n",
    "if not os.path.isdir(\"../Data/gleam\"):\n",
    "    os.makedirs(\"../Data/gleam\")\n",
    "\n",
    "# Download the GLEAM data\n",
    "# host input excludes the sftp prefix\n",
    "fs = fsspec.filesystem(\"sftp\", host=sftp_host[7:], **gleam_creds_sftp)\n",
    "\n",
    "filepath = \"/data/v3.7b/monthly/E_2003-2022_GLEAM_v3.7b_MO.nc\"\n",
    "path = sftp_host + filepath\n",
    "\n",
    "fs.get(path, \"../Data/gleam/\")\n",
    "\n",
    "# Open the file\n",
    "ds = xr.open_dataset(\n",
    "    f\"../Data/gleam/{os.path.basename(filepath)}\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "\n",
    "# Only keep CONUS range of data\n",
    "ds = ds.sel(lat=slice(53, 24))\n",
    "ds = ds.sel(lon=slice(-126, -66))\n",
    "\n",
    "# Adjust month dates to be first of month rather than end of month for consistency\n",
    "# with other datasets\n",
    "ds = ds.reindex(\n",
    "    {\"time\": ds.get_index(\"time\").shift(periods=-1, freq=\"MS\")}, method=\"backfill\"\n",
    ")\n",
    "\n",
    "# Rename variable to common name and add new metadata attributes\n",
    "ds = ds.rename({\"E\": \"aet\"})\n",
    "ds[\"aet\"].attrs[\"description\"] = \"Actual total evaporation from GLEAM 3.7b\"\n",
    "ds[\"aet\"].attrs[\"long_name\"] = \"Actual evaporation\"\n",
    "ds[\"aet\"].attrs[\"dimensions\"] = \"lon lat time\"\n",
    "\n",
    "# Add some coordinate metadata attributes\n",
    "ds[\"lat\"].attrs[\"units\"] = \"degrees_north\"\n",
    "ds[\"lat\"].attrs[\"description\"] = \"Latitude of the center of the grid cell\"\n",
    "ds[\"lat\"].attrs[\"long_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"standard_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"axis\"] = \"Y\"\n",
    "\n",
    "ds[\"lon\"].attrs[\"units\"] = \"degrees_east\"\n",
    "ds[\"lon\"].attrs[\"description\"] = \"Longitude of the center of the grid cell\"\n",
    "ds[\"lon\"].attrs[\"long_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"standard_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"axis\"] = \"X\"\n",
    "\n",
    "ds[\"time\"].attrs[\"long_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"standard_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "ds[\"time\"].attrs[\"axis\"] = \"T\"\n",
    "\n",
    "ds.to_netcdf(\n",
    "    path=\"../Data/gleam/gleam_aet.nc\",\n",
    "    format=\"NETCDF4\",\n",
    "    engine=\"netcdf4\",\n",
    "    # Ensure no chunking as file is very small (~20 MB)\n",
    "    encoding={\"aet\": {'chunksizes': list(ds.coords.sizes.values())}}\n",
    ")\n",
    "\n",
    "# Remove downloaded file to reduce storage, as the data is now in the new netcdf\n",
    "os.remove(\"../Data/gleam/E_2003-2022_GLEAM_v3.7b_MO.nc\")\n",
    "\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/gleam/gleam_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "314b3c0f-047f-4a50-8f0e-36d859b5ea04",
   "metadata": {},
   "source": [
    "## ERA-5 Data Set\n",
    "\n",
    "The ERA-5 data can be found at the [Climate Data Store](https://cds.climate.copernicus.eu/) (CDS). The monthly data is stored in a cloud hosted format and needs to be retrieved using the [CDS API](https://cds.climate.copernicus.eu/how-to-api). We selected to retrieve the mean monthly ET data for the years spanning 1950-2022. The preprocessing steps included (1) converting the data to positive values as negatives are used to indicate the inverse of precipitation; (2) setting any then negative values to zero; and (3) converting units from meters per day to millimeters per month.\n",
    "\n",
    "> NOTE: You will need a CDS account to access this data. Once you have an account, make sure to [configure `cdsapi`](https://github.com/ecmwf/cdsapi#configure) for the download to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e960d505-bdf8-49cd-960d-f8ab8462c273",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ERA-5 Dataset\n",
    "import xarray as xr\n",
    "import cdsapi\n",
    "import os\n",
    "import zipfile\n",
    "\n",
    "c = cdsapi.Client()\n",
    "\n",
    "if not os.path.isdir(\"../Data/era5\"):\n",
    "    os.makedirs(\"../Data/era5\")\n",
    "\n",
    "# This command is generated from Climate Data Store\n",
    "# (https://cds.climate.copernicus.eu/datasets/reanalysis-era5-land-monthly-means?tab=download)\n",
    "c.retrieve(\n",
    "    \"reanalysis-era5-land-monthly-means\",\n",
    "    {\n",
    "        \"product_type\": \"monthly_averaged_reanalysis\",\n",
    "        \"variable\": \"total_evaporation\",\n",
    "        \"year\": [\n",
    "            \"1950\",\n",
    "            \"1951\",\n",
    "            \"1952\",\n",
    "            \"1953\",\n",
    "            \"1954\",\n",
    "            \"1955\",\n",
    "            \"1956\",\n",
    "            \"1957\",\n",
    "            \"1958\",\n",
    "            \"1959\",\n",
    "            \"1960\",\n",
    "            \"1961\",\n",
    "            \"1962\",\n",
    "            \"1963\",\n",
    "            \"1964\",\n",
    "            \"1965\",\n",
    "            \"1966\",\n",
    "            \"1967\",\n",
    "            \"1968\",\n",
    "            \"1969\",\n",
    "            \"1970\",\n",
    "            \"1971\",\n",
    "            \"1972\",\n",
    "            \"1973\",\n",
    "            \"1974\",\n",
    "            \"1975\",\n",
    "            \"1976\",\n",
    "            \"1977\",\n",
    "            \"1978\",\n",
    "            \"1979\",\n",
    "            \"1980\",\n",
    "            \"1981\",\n",
    "            \"1982\",\n",
    "            \"1983\",\n",
    "            \"1984\",\n",
    "            \"1985\",\n",
    "            \"1986\",\n",
    "            \"1987\",\n",
    "            \"1988\",\n",
    "            \"1989\",\n",
    "            \"1990\",\n",
    "            \"1991\",\n",
    "            \"1992\",\n",
    "            \"1993\",\n",
    "            \"1994\",\n",
    "            \"1995\",\n",
    "            \"1996\",\n",
    "            \"1997\",\n",
    "            \"1998\",\n",
    "            \"1999\",\n",
    "            \"2000\",\n",
    "            \"2001\",\n",
    "            \"2002\",\n",
    "            \"2003\",\n",
    "            \"2004\",\n",
    "            \"2005\",\n",
    "            \"2006\",\n",
    "            \"2007\",\n",
    "            \"2008\",\n",
    "            \"2009\",\n",
    "            \"2010\",\n",
    "            \"2011\",\n",
    "            \"2012\",\n",
    "            \"2013\",\n",
    "            \"2014\",\n",
    "            \"2015\",\n",
    "            \"2016\",\n",
    "            \"2017\",\n",
    "            \"2018\",\n",
    "            \"2019\",\n",
    "            \"2020\",\n",
    "            \"2021\",\n",
    "            \"2022\",\n",
    "        ],\n",
    "        \"month\": [\n",
    "            \"01\",\n",
    "            \"02\",\n",
    "            \"03\",\n",
    "            \"04\",\n",
    "            \"05\",\n",
    "            \"06\",\n",
    "            \"07\",\n",
    "            \"08\",\n",
    "            \"09\",\n",
    "            \"10\",\n",
    "            \"11\",\n",
    "            \"12\",\n",
    "        ],\n",
    "        \"time\": \"00:00\",\n",
    "        \"data_format\": \"netcdf\",\n",
    "        \"area\": [\n",
    "            53,\n",
    "            -126,\n",
    "            24,\n",
    "            -66,\n",
    "        ],\n",
    "    },\n",
    "    \"../Data/era5/era5.zip\",\n",
    ")\n",
    "\n",
    "# Access zip file without unzipping\n",
    "zfile = zipfile.ZipFile(\"../Data/era5/era5.zip\")\n",
    "\n",
    "# Select the GeoTIFF files from zip_file_list and extract\n",
    "zfile.extractall(\"../Data/era5/\", ['data_0.nc', 'data_1.nc'])\n",
    "\n",
    "ds = xr.open_mfdataset(\n",
    "    \"../Data/era5/data_*.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "\n",
    "# Rename the coordinates to a common format and add some metadata attributes\n",
    "ds = ds.rename({\"valid_time\": \"time\", \"longitude\": \"lon\",\n",
    "                \"latitude\": \"lat\", \"e\": \"aet\"})\n",
    "ds = ds.drop_vars(['number', 'expver'])\n",
    "\n",
    "# Data values are negative to indicate inverse of precipitation (see docs) and\n",
    "# in meters. We want to switch to positive values and mm, along with setting\n",
    "# any then negative values to 0. Additionally, from the documentation, the monthly\n",
    "# means have units that include \"per day\". We want \"per month\". So, we need to\n",
    "# multiply each month by the number of days in it.\n",
    "ds = -1e3 * ds\n",
    "ds = ds.where(~(ds < 0), 0)\n",
    "ds = ds * ds.get_index(\"time\").days_in_month.values.reshape(\n",
    "    len(ds.get_index(\"time\")), 1, 1\n",
    ")\n",
    "\n",
    "# Add new metadata attributes\n",
    "ds[\"aet\"].attrs[\"units\"] = \"mm.month-1\"\n",
    "ds[\"aet\"].attrs[\"description\"] = (\n",
    "    \"Accumulated amount of water that has evaporated from the Earth's surface, \"\n",
    "    + \"including a simplified representation of transpiration (from vegetation), \"\n",
    "    + \"into vapour in the air above.\"\n",
    ")\n",
    "ds[\"aet\"].attrs[\"long_name\"] = \"Total Evaporation\"\n",
    "ds[\"aet\"].attrs[\"dimensions\"] = \"lon lat time\"\n",
    "\n",
    "ds[\"lat\"].attrs[\"description\"] = \"Latitude of the center of the grid cell\"\n",
    "ds[\"lat\"].attrs[\"standard_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"axis\"] = \"Y\"\n",
    "\n",
    "ds[\"lon\"].attrs[\"description\"] = \"Longitude of the center of the grid cell\"\n",
    "ds[\"lon\"].attrs[\"standard_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"axis\"] = \"X\"\n",
    "\n",
    "ds[\"time\"].attrs[\"standard_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"axis\"] = \"T\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "\n",
    "# Convert to float32 (we do not need 64 bit precision) and save.\n",
    "ds = ds.astype(\"float32\")\n",
    "ds.to_netcdf(\n",
    "    path=\"../Data/era5/era5_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    format=\"NETCDF4\",\n",
    "    # Ensure no chunking as file is small (~0.5 GB)\n",
    "    encoding={\"aet\": {'chunksizes': list(ds.coords.sizes.values())}}\n",
    ")\n",
    "\n",
    "os.remove(\"../Data/era5/era5.zip\")\n",
    "os.remove(\"../Data/era5/data_0.nc\")\n",
    "os.remove(\"../Data/era5/data_1.nc\")\n",
    "\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/era5/era5_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c45e04df-a49b-470a-b530-1a30c417ff7a",
   "metadata": {},
   "source": [
    "## NLDAS Data Set\n",
    "\n",
    "The NLDAS data can be found at the [Goddard Earth Sciences Data and Information Services Center](https://daac.gsfc.nasa.gov/) (GESDISC). The monthly data is stored in individual NetCDF files which we retrieve with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/#). We selected to retrieve all of the monthly data for the available years of 1979-2022. The preprocessing steps included (1) extracting the ET data from the collection of variables; (2) drop January of 1979 as it starts on the 2nd of the month vs the 1st like all other months; and (3) converting units from kilograms per meter squared per month to millimeters per month (really just a name change assuming a water density of 1 g.cm-3)\n",
    "\n",
    "> NOTE: You will need a [EarthData Login](https://wiki.earthdata.nasa.gov/display/EL/How+To+Register+For+an+EarthData+Login+Profile) to access this data. Once you have an login, make sure to [link the login to the NASA GESDISC Data Archive](https://disc.gsfc.nasa.gov/earthdata-login) for the download to work. You will then need to set your username and password as the os enviromental variables (e.g., in your `~/.bashrc` file) of `NASA_EARTHDATA_USERNAME` and `NASA_EARTHDATA_PASSWORD`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f6000bf-f519-4037-b76e-7e828c5bbc2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NLDAS Dataset\n",
    "import fsspec\n",
    "import xarray as xr\n",
    "import aiohttp\n",
    "import os\n",
    "\n",
    "if not os.path.isdir(\"../Data/nldas\"):\n",
    "    os.makedirs(\"../Data/nldas\")\n",
    "\n",
    "# If compiled netcdf is not made from downloads, make it\n",
    "# Requires an account to access the data\n",
    "# Username and Password are given as OS environmental variables\n",
    "# (NASA_EARTHDATA_USERNAME and NASAS_EARTHDATA_PASSWORD)\n",
    "fs = fsspec.filesystem(\n",
    "    \"https\",\n",
    "    timeout=3600,\n",
    "    client_kwargs={\n",
    "        \"auth\": aiohttp.BasicAuth(\n",
    "            os.environ[\"NASA_EARTHDATA_USERNAME\"],\n",
    "            password=os.environ[\"NASA_EARTHDATA_PASSWORD\"],\n",
    "        )\n",
    "    },\n",
    ")\n",
    "base_url = (\n",
    "    \"https://data.gesdisc.earthdata.nasa.gov/data/NLDAS/NLDAS_NOAH0125_M.2.0/\"\n",
    ")\n",
    "\n",
    "months = range(1, 13)\n",
    "years = range(1979, 2023)\n",
    "\n",
    "# Make list of paths\n",
    "paths = []\n",
    "for year in years:\n",
    "    for month in months:\n",
    "        # Create full URL\n",
    "        filepath = (\n",
    "            f\"{year}/NLDAS_NOAH0125_M.A{year}\" + str(month).zfill(2) + \".020.nc\"\n",
    "        )\n",
    "        paths.append(base_url + filepath)\n",
    "\n",
    "fs.get(paths, \"../Data/nldas/\")\n",
    "\n",
    "# Open first year to get list of variables to drop\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/nldas/NLDAS_NOAH0125_M.A197901.020.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={}\n",
    ")\n",
    "drop_vars = [var for var in list(ds.data_vars) if var != \"Evap\"]\n",
    "\n",
    "# Open all files and combine. Use one chunk as file is only 200MB total\n",
    "ds = xr.open_mfdataset(\n",
    "    [\n",
    "        \"../Data/nldas/\"\n",
    "        + f\"NLDAS_NOAH0125_M.A{year}\"\n",
    "        + str(month).zfill(2)\n",
    "        + \".020.nc\"\n",
    "        for year in years\n",
    "        for month in months\n",
    "    ],\n",
    "    drop_variables=drop_vars,\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={\"lat\": -1, \"lon\": -1, \"time\": -1},\n",
    ")\n",
    "\n",
    "# Drop January of 1979 as it starts on the 2nd. See NLDAS docs for details.\n",
    "ds = ds.where(ds.time != ds.time[0], drop=True)\n",
    "\n",
    "# Rename variable and coords to common names and add attributes\n",
    "ds = ds.rename({\"Evap\": \"aet\"})\n",
    "\n",
    "# Units are in kg.m-2.month-1, which is equivalent to mm.month-1 assuming a water\n",
    "# density of 1g.cm-3 (mm = kg.m-2 / g.cm-3 * 1e3g.kg-1 * 1e-6m3.cm-3 * 1e3mm.m-1)\n",
    "ds[\"aet\"].attrs[\"units\"] = \"mm.month-1\"\n",
    "ds[\"aet\"].attrs[\"description\"] = \"Actual Total Evapotranspiration\"\n",
    "ds[\"aet\"].attrs[\"dimensions\"] = \"lon lat time\"\n",
    "\n",
    "# Add some metadata attributes\n",
    "ds[\"lat\"].attrs[\"description\"] = \"Latitude of the center of the grid cell\"\n",
    "ds[\"lat\"].attrs[\"axis\"] = \"Y\"\n",
    "\n",
    "ds[\"lon\"].attrs[\"description\"] = \"Longitude of the center of the grid cell\"\n",
    "ds[\"lon\"].attrs[\"axis\"] = \"X\"\n",
    "\n",
    "ds[\"time\"].attrs[\"standard_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"axis\"] = \"T\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "del ds[\"time\"].attrs[\"begin_date\"]\n",
    "del ds[\"time\"].attrs[\"begin_time\"]\n",
    "del ds[\"time\"].attrs[\"end_date\"]\n",
    "del ds[\"time\"].attrs[\"end_time\"]\n",
    "del ds[\"time\"].attrs[\"bounds\"]\n",
    "\n",
    "# Save dataset to netcdf\n",
    "ds.to_netcdf(\n",
    "    path=\"../Data/nldas/nldas_aet.nc\",\n",
    "    format=\"NETCDF4\",\n",
    "    engine=\"netcdf4\",\n",
    "    # Ensure no chunking as file is small (~0.2 GB)\n",
    "    encoding={\"aet\": {'chunksizes': list(ds.coords.sizes.values())}}\n",
    ")\n",
    "\n",
    "# Remove downloaded files to reduce storage, as the data is now in the combined netcdf\n",
    "for year in years:\n",
    "    for month in months:\n",
    "        os.remove(\n",
    "            \"../Data/nldas/\"\n",
    "            + f\"NLDAS_NOAH0125_M.A{year}\"\n",
    "            + str(month).zfill(2)\n",
    "            + \".020.nc\"\n",
    "        )\n",
    "\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/nldas/nldas_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fb6d5bd-5838-4aee-aa39-cf3441c3bc25",
   "metadata": {},
   "source": [
    "## TerraClimate Data Set\n",
    "\n",
    "The TerraClimate data can be found at the [northestknowledge.net](https://climate.northwestknowledge.net/) via the [Climatology Lab](https://www.climatologylab.org/terraclimate.html). The monthly data is stored in individual NetCDF files for each year spanning 1958-2022, which we retrieve with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/#). The preprocessing steps included limiting the latitude and longitude to CONUS range as the files hold global data (i.e., limit latitudes from $24^\\circ$ to $53^\\circ$ and longitudes from $-126^\\circ$ to $-66^\\circ$). No other preprocessing is needed as the units are already in millimeters per month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32511002-51c9-474a-9fa6-418bacdf109b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TerraClimate Dataset\n",
    "import fsspec\n",
    "import xarray as xr\n",
    "import os\n",
    "\n",
    "if not os.path.isdir(\"../Data/terraclimate\"):\n",
    "    os.makedirs(\"../Data/terraclimate\")\n",
    "\n",
    "# If compiled netcdf is not made from downloads, make it\n",
    "fs = fsspec.filesystem(\"https\", timeout=3600)\n",
    "url = \"https://climate.northwestknowledge.net/TERRACLIMATE-DATA/\"\n",
    "\n",
    "years = range(1958, 2023)\n",
    "\n",
    "# Download all of the individual year files. Do this recursively as fs may timeout if\n",
    "# a full list of files is called at once\n",
    "paths = []\n",
    "for year in years:\n",
    "    file = f\"TerraClimate_aet_{year}.nc\"\n",
    "    # Create full URL\n",
    "    paths.append(url + file)\n",
    "\n",
    "fs.get(paths, \"../Data/terraclimate/\")\n",
    "\n",
    "# Open the files and combine\n",
    "ds = xr.open_mfdataset(\n",
    "    [\"../Data/terraclimate/\" + f\"TerraClimate_aet_{year}.nc\" for year in years],\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={'lat': -1, 'lon': -1, 'time': 12},\n",
    ")\n",
    "\n",
    "# Only keep CONUS range of data\n",
    "ds = ds.sel(lat=slice(53, 24))\n",
    "ds = ds.sel(lon=slice(-126, -66))\n",
    "\n",
    "# Remove crs index\n",
    "ds = ds.drop_vars(\"crs\")\n",
    "\n",
    "# Replace unicode characters in summary (degree symbol)\n",
    "ds.attrs[\"summary\"] = ds.attrs[\"summary\"].replace(\n",
    "    ds.attrs[\"summary\"][64:66], \" deg\"\n",
    ")\n",
    "\n",
    "# Update aet units to include time span\n",
    "ds[\"aet\"].attrs[\"units\"] = \"mm.month-1\"\n",
    "ds[\"aet\"].attrs[\"long_name\"] = \"Total Actual Evapotranspiration\"\n",
    "\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "\n",
    "# Convert dtype to float32 as we do not need 64 bit precision, and it reduces\n",
    "# the size by half\n",
    "ds = ds.astype(\"float32\")\n",
    "\n",
    "# Create chunksizes from coords, with the time dim chunked\n",
    "chunksizes = list(ds.coords.sizes.values())\n",
    "chunksizes[list(ds.coords.sizes).index('time')] = 1\n",
    "\n",
    "# Save xarray dataset to netcdf\n",
    "_ = ds.to_netcdf(\n",
    "    path=\"../Data/terraclimate/terraclimate_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    format=\"NETCDF4\",\n",
    "    # Chunk file and compress as is its pretty big uncompressed (~3 GB)\n",
    "    encoding={\"aet\": {\"zlib\": True, \"complevel\": 4, 'chunksizes': chunksizes}},\n",
    ")\n",
    "\n",
    "# Remove downloaded files to reduce storage, as the data is now in the combined netcdf\n",
    "for year in years:\n",
    "    os.remove(\"../Data/terraclimate/\" + f\"TerraClimate_aet_{year}.nc\")\n",
    "\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/terraclimate/terraclimate_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b9e451-e65a-40bf-90f5-45ee24b497cb",
   "metadata": {},
   "source": [
    "## WBET (Rietz et al. 2023) Data Set\n",
    "\n",
    "The WBET data can be accessed from [ScienceBase](https://www.sciencebase.gov/catalog/item/64135576d34eb496d1ce3d2e). The monthly data is stored in individual GeoTIFFs files that are then zipped within decadal directories spanning years 1895-2018. We retrieve the zip files uing with [sciencebasepy](https://github.com/DOI-USGS/sciencebasepy). We selected to retrieve all of the monthly data for the available years. The preprocessing steps included (1) extracting the monthly GeoTIFFs from the zip files and combining them all into a single NetCDF file; and (2) converting the units from mm per day to mm per month.\n",
    "\n",
    "> NOTE: You will need a [SciencBase Account](https://www.sciencebase.gov/directory/newUser/create) to download this data via the Python code, since ScienceBase requires and account to access cloud hosted data. You will then need to set your username and password as the os enviromental variables (e.g., in your `~/.bashrc` file) of `SCIENCEBASE_USERNAME` and `SCIENCEBASE_PASSWORD`. If you don't have an account and do not want to create one, you will have to manually download [the data](https://www.sciencebase.gov/catalog/item/64135576d34eb496d1ce3d2e)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d20d5c5-6962-4c1c-91eb-8b269f2320dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sciencebasepy\n",
    "import os\n",
    "import re\n",
    "import rioxarray\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "import zipfile_deflate64 as zipfile\n",
    "import io\n",
    "\n",
    "if not os.path.isdir(\"../Data/wbet\"):\n",
    "    os.makedirs(\"../Data/wbet\")\n",
    "\n",
    "# Establish a session.\n",
    "sb = sciencebasepy.SbSession()\n",
    "\n",
    "# Login required to access cloud files via sciencebasepy\n",
    "sb.login(os.environ[\"SCIENCEBASE_USERNAME\"], os.environ[\"SCIENCEBASE_PASSWORD\"])\n",
    "\n",
    "years = range(1896, 2019)\n",
    "\n",
    "# Get list of files for monthly ET data\n",
    "file_list = sb.get_item_file_info(sb.get_item(\"64135576d34eb496d1ce3d2e\"))\n",
    "filenames = [\n",
    "    i[\"name\"]\n",
    "    for i in file_list\n",
    "    if re.search(\"ET.*_monthly.zip\", i[\"name\"]) is not None\n",
    "]\n",
    "\n",
    "file_date_ranges = [re.findall(r'\\d+', filename) for filename in filenames]\n",
    "filenames = [name for name, (low, high) in zip(filenames, file_date_ranges)\n",
    "             if (int(low) >= min(years)) and (int(high) <= max(years))]\n",
    "\n",
    "# Download the files (these files are big (6GB a piece), so this will take a while...)\n",
    "_ = sb.download_cloud_files(\n",
    "    filenames,\n",
    "    sb.generate_S3_download_links(\"64135576d34eb496d1ce3d2e\", filenames),\n",
    "    \"wbet\",\n",
    ")\n",
    "\n",
    "# Open the GeoTIFF files to xarray\n",
    "ds_monthly_list = []\n",
    "for zippedfiles in filenames:\n",
    "    # Access zip file without unzipping\n",
    "    zfile = zipfile.ZipFile(\"../Data/wbet/\" + zippedfiles)\n",
    "    zip_file_list = zfile.namelist()\n",
    "\n",
    "    # Select the GeoTIFF files from zip_file_list and extract\n",
    "    gtif_files = [\n",
    "        file for file in zip_file_list if re.search(\".*(\\.tif)$\", file) is not None\n",
    "    ]\n",
    "    zfile.extractall(\"../Data/wbet/\", gtif_files)\n",
    "\n",
    "    # Delete downloaded file to save disk space, since files are now extracted\n",
    "    os.remove(\"../Data/wbet/\" + zippedfiles)\n",
    "\n",
    "    for gtif in gtif_files:\n",
    "        # Read in each extracted GeoTIFF\n",
    "        ds_month = rioxarray.open_rasterio(\n",
    "            \"../Data/wbet/\" + gtif, chunks={}, band_as_variable=True\n",
    "        )\n",
    "\n",
    "        # Remove spatial_ref coord and rename coords to corresponding names.\n",
    "        # Assign the date to the Dataset 5th-8th characters of file indicate\n",
    "        # year, 10th-11th indicate month (index start at 0, characters at 1)\n",
    "        year, month = gtif[4:8], gtif[9:11]\n",
    "        date = year + \"-\" + month\n",
    "        ds_month = (\n",
    "            ds_month.rename({\"x\": \"lon\", \"y\": \"lat\", \"band_1\": \"aet\"})\n",
    "            .assign_coords(time=pd.to_datetime(date))\n",
    "            .expand_dims(dim=\"time\")\n",
    "        )\n",
    "        ds_month = ds_month.drop_vars(\"spatial_ref\")\n",
    "\n",
    "        # Stack the monthly Datasets to list for concatenating\n",
    "        ds_monthly_list.append(ds_month)\n",
    "        ds_month.close()\n",
    "\n",
    "    # Concatenate and save to netcdf\n",
    "    ds = xr.concat(ds_monthly_list, dim=\"time\")\n",
    "    ds.to_netcdf(\n",
    "        path=\"../Data/wbet/\" + zippedfiles[:-4] + \".nc\",\n",
    "        format=\"NETCDF4\",\n",
    "        engine=\"netcdf4\",\n",
    "        encoding={\"aet\": {\"zlib\": True, \"complevel\": 4}},\n",
    "    )\n",
    "\n",
    "    # Delete extracted files to save disk space, since files are now\n",
    "    # compiled to netcdf\n",
    "    for gtif in gtif_files:\n",
    "        os.remove(\"../Data/wbet/\" + gtif)\n",
    "\n",
    "    # Reset variables\n",
    "    del ds\n",
    "    del ds_monthly_list\n",
    "    ds_monthly_list = []\n",
    "\n",
    "# Open processed decade files as single dataset\n",
    "ds = xr.open_mfdataset(\n",
    "    [\"../Data/wbet/\" + file[:-4] + \".nc\" for file in filenames],\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={\"lat\": -1, \"lon\": -1, \"time\": 2},\n",
    ")\n",
    "\n",
    "# From the metadata xml file, the monthly data are in units of mm.day-1.\n",
    "# We want mm.month-1. So, we need to multiply each month by the number of days in it.\n",
    "ds = ds * ds.get_index(\"time\").days_in_month.values.reshape(\n",
    "    len(ds.get_index(\"time\")), 1, 1\n",
    ")\n",
    "\n",
    "# Add new metadata attributes to variable and coordinates\n",
    "ds[\"aet\"].attrs[\"unit\"] = \"mm.month-1\"\n",
    "ds[\"aet\"].attrs[\"description\"] = (\n",
    "    \"Actual Total Evapotranspiration via WBET from Reitz+2023\"\n",
    ")\n",
    "ds[\"aet\"].attrs[\"long_name\"] = \"Actual Evapotranspiration\"\n",
    "ds[\"aet\"].attrs[\"standard_name\"] = \"Actual Evapotranspiration\"\n",
    "ds[\"aet\"].attrs[\"dimensions\"] = \"lon lat time\"\n",
    "\n",
    "ds[\"lat\"].attrs[\"units\"] = \"degrees_north\"\n",
    "ds[\"lat\"].attrs[\"description\"] = \"Latitude of the center of the grid cell\"\n",
    "ds[\"lat\"].attrs[\"long_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"standard_name\"] = \"latitude\"\n",
    "ds[\"lat\"].attrs[\"axis\"] = \"Y\"\n",
    "\n",
    "ds[\"lon\"].attrs[\"units\"] = \"degrees_east\"\n",
    "ds[\"lon\"].attrs[\"description\"] = \"Longitude of the center of the grid cell\"\n",
    "ds[\"lon\"].attrs[\"long_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"standard_name\"] = \"longitude\"\n",
    "ds[\"lon\"].attrs[\"axis\"] = \"X\"\n",
    "\n",
    "ds[\"time\"].attrs[\"long_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"standard_name\"] = \"time\"\n",
    "ds[\"time\"].attrs[\"unit\"] = \"month\"\n",
    "ds[\"time\"].attrs[\"description\"] = (\n",
    "    \"Monthly time step indicated by the first day of the month.\"\n",
    ")\n",
    "ds[\"time\"].attrs[\"axis\"] = \"T\"\n",
    "\n",
    "# Convert dtype to float32 as we do not need 64 bit precision, and it reduces\n",
    "# the size by half\n",
    "ds = ds.astype(\"float32\")\n",
    "\n",
    "# Create chunksizes from coords, with the time dim chunked\n",
    "chunksizes = list(ds.coords.sizes.values())\n",
    "chunksizes[list(ds.coords.sizes).index('time')] = 1\n",
    "\n",
    "# Save dataset and remove processed files\n",
    "ds.to_netcdf(\n",
    "    path=\"../Data/wbet/wbet_aet.nc\",\n",
    "    format=\"NETCDF4\",\n",
    "    engine=\"netcdf4\",    \n",
    "    # Chunk file and compress as it is big uncompressed (~130 GB)\n",
    "    encoding={\"aet\": {\"zlib\": True, \"complevel\": 9, 'chunksizes': chunksizes}},\n",
    ")\n",
    "\n",
    "for file in filenames:\n",
    "    os.remove(\"../Data/wbet/\" + file[:-4] + \".nc\")\n",
    "\n",
    "# Open the saved netcdf\n",
    "ds = xr.open_dataset(\n",
    "    \"../Data/wbet/wbet_aet.nc\",\n",
    "    engine=\"netcdf4\",\n",
    "    chunks={},\n",
    ")\n",
    "ds"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}