modis-explore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This notebook \n",
    "began as a duplication of the MODIS notebook for exploring chlorophyll data at the site of the OOI Cabled Array and was utilized as a generic explorer for MODIS Satellite Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MODIS Satellite Data - Map Projection\n",
    "\n",
    "\n",
    "## Get Data\n",
    "\n",
    "* [NASA Earth Observations (NEO)](https://neo.sci.gsfc.nasa.gov/view.php?datasetId=MY1DMW_CHLORA&date=2019-02-01) site\n",
    "  - No bounding box\n",
    "  - Can select monthly averages or 8-day\n",
    "  - Clicking on `Download raw data` produces 30-45MB files at 0.1 deg resolution as NetCDF\n",
    "* MODIS Chlorophyll Level 1 and 2 data are [here](http://oceancolor.gsfc.nasa.gov/cgi/browse.pl?sen=am)\n",
    "* MODIS Chlorophyll Level 3 data are [here](http://oceancolor.gsfc.nasa.gov/cgi/l3)\n",
    "* These data files will be saved onto your computers harddrive and must be moved into Pangeo in order to run these cells\n",
    "\n",
    "\n",
    "## Set up Folders and Files\n",
    "* In working Pangeo server, create a 'data' folder and a 'modis' subfolder. Upload your MODIS files to this subfolder. Do not capitalize folder names.\n",
    "* Ensure that you have cloned a repo containing the 'golive_libary.py' file. It can be found in the robfatland/golive repo on GitHub.\n",
    "\n",
    "## Data Presentation \n",
    "\n",
    "This notebook takes a look at how to superimpose MODIS chlorophyll data on a map and explore different methods of presentation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up python environment\n",
    "* Install all necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = 'all'  # default is ‘last_expr’\n",
    "\n",
    "# automatically reload external modules such as golive_library if they have changed\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: utm in /srv/conda/envs/notebook/lib/python3.8/site-packages (0.5.0)\n",
      "Collecting pygeotools\n",
      "  Using cached pygeotools-0.6.0.tar.gz (80 kB)\n",
      "Requirement already satisfied: gdal in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pygeotools) (3.0.4)\n",
      "Requirement already satisfied: numpy in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pygeotools) (1.18.1)\n",
      "Requirement already satisfied: scipy in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pygeotools) (1.4.1)\n",
      "Requirement already satisfied: matplotlib in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pygeotools) (3.1.3)\n",
      "Requirement already satisfied: cycler>=0.10 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib->pygeotools) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib->pygeotools) (2.7.5)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib->pygeotools) (2.4.6)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib->pygeotools) (1.1.0)\n",
      "Requirement already satisfied: six in /srv/conda/envs/notebook/lib/python3.8/site-packages (from cycler>=0.10->matplotlib->pygeotools) (1.14.0)\n",
      "Requirement already satisfied: setuptools in /srv/conda/envs/notebook/lib/python3.8/site-packages (from kiwisolver>=1.0.1->matplotlib->pygeotools) (45.1.0.post20200119)\n",
      "Building wheels for collected packages: pygeotools\n",
      "  Building wheel for pygeotools (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for pygeotools: filename=pygeotools-0.6.0-py3-none-any.whl size=88056 sha256=3f3f0130ca6d1eba0757dfbb18cbaa15ba77593cdf045891f76ce0cd372a7947\n",
      "  Stored in directory: /home/jovyan/.cache/pip/wheels/49/e7/5b/66b229126d8bd59141a175f0cd6ae71750ad87eb6e477620bc\n",
      "Successfully built pygeotools\n",
      "Installing collected packages: pygeotools\n",
      "Successfully installed pygeotools-0.6.0\n",
      "Collecting tqdm\n",
      "  Using cached tqdm-4.43.0-py2.py3-none-any.whl (59 kB)\n",
      "Installing collected packages: tqdm\n",
      "Successfully installed tqdm-4.43.0\n",
      "Requirement already satisfied: netCDF4 in /srv/conda/envs/notebook/lib/python3.8/site-packages (1.5.3)\n",
      "Requirement already satisfied: numpy>=1.7 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from netCDF4) (1.18.1)\n",
      "Requirement already satisfied: cftime in /srv/conda/envs/notebook/lib/python3.8/site-packages (from netCDF4) (1.0.4.2)\n",
      "Requirement already satisfied: xarray in /srv/conda/envs/notebook/lib/python3.8/site-packages (0.15.0)\n",
      "Requirement already satisfied: pandas>=0.25 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from xarray) (1.0.1)\n",
      "Requirement already satisfied: numpy>=1.15 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from xarray) (1.18.1)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pandas>=0.25->xarray) (2.7.5)\n",
      "Requirement already satisfied: pytz>=2017.2 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pandas>=0.25->xarray) (2019.3)\n",
      "Requirement already satisfied: six>=1.5 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from python-dateutil>=2.6.1->pandas>=0.25->xarray) (1.14.0)\n",
      "Requirement already satisfied: matplotlib in /srv/conda/envs/notebook/lib/python3.8/site-packages (3.1.3)\n",
      "Requirement already satisfied: cycler>=0.10 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib) (1.1.0)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib) (2.4.6)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib) (2.7.5)\n",
      "Requirement already satisfied: numpy>=1.11 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from matplotlib) (1.18.1)\n",
      "Requirement already satisfied: six in /srv/conda/envs/notebook/lib/python3.8/site-packages (from cycler>=0.10->matplotlib) (1.14.0)\n",
      "Requirement already satisfied: setuptools in /srv/conda/envs/notebook/lib/python3.8/site-packages (from kiwisolver>=1.0.1->matplotlib) (45.1.0.post20200119)\n",
      "Requirement already satisfied: scipy in /srv/conda/envs/notebook/lib/python3.8/site-packages (1.4.1)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from scipy) (1.18.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install utm\n",
    "!pip install pygeotools\n",
    "!pip install tqdm\n",
    "!pip install netCDF4\n",
    "!pip install xarray\n",
    "!pip install matplotlib\n",
    "!pip install scipy\n",
    "#!pip install dask\n",
    "#!pip install utm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Python Standard Library built-in modules\n",
    "import sys\n",
    "import os\n",
    "import timeit\n",
    "from datetime import timedelta, datetime\n",
    "from random import randint as ri\n",
    "\n",
    "# Third-party modules\n",
    "from tqdm import tqdm\n",
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "import netCDF4\n",
    "import xarray as xr\n",
    "import dask\n",
    "import numpy as np\n",
    "from numpy import datetime64 as dt64, timedelta64 as td64\n",
    "from scipy import stats\n",
    "from scipy.signal import medfilt as mf\n",
    "from ftplib import FTP\n",
    "from ipywidgets import *                # interactivity\n",
    "from traitlets import dlink             # interactivity\n",
    "import utm\n",
    "# from osgeo import gdal \n",
    "# from pygeotools.lib import iolib,timelib\n",
    "\n",
    "# Local application/library specific imports\n",
    "from golive_library import GoliveLibrary as g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this cell is for the section of the notebook beyond the intermezzo remarks\n",
    "# verify the MODIS example file is present\n",
    "#modis_d = g.set_data_directory('modis')\n",
    "#image_d = g.set_data_directory('images')\n",
    "#g.lsal(modis_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open all files as an xarray dataset for processing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b = A20191532019160.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20191612019168.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20191692019176.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20191772019184.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20191852019192.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20191932019200.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192012019208.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192092019216.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192172019224.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192252019232.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192332019240.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192412019248.L3m_8D_CHL_chlor_a_4km.nc\n",
      "b = A20192492019256.L3m_8D_CHL_chlor_a_4km.nc\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "fnm_field_index = 8                                                 # index of the filename after the ls line is split\n",
    "m=[]                                                                # a list of available MODIS datasets\n",
    "r = os.popen('cd ..; cd data/modis; ls -al *.nc').readlines()       # Opens all MODIS data saved in appropriate folder\n",
    "#print (r[0])                                                       # Line for checking wether previous command works \n",
    "#print ('type r=', type(r[0]))                                      # confirms type of object saved as r\n",
    "for q in r:                                                         # opens each file and changes filename for easier use\n",
    "    b = q.split()[fnm_field_index].rstrip()                         # removes whitespace from files to make them more easily callable\n",
    "    print('b =', b)                                                # Prints each datafile name to check for success of previous line\n",
    "    m.append(xr.open_dataset('../data/modis/' + b))                 # Append the dataset with new, cleaned up filenames\n",
    "\n",
    "print(len(m))                                                       # confirms amount of files saved in dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring data\n",
    "\n",
    "Now that we've loaded our datasets we can explore them at our leisure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<xarray.Dataset>\n",
      "Dimensions:  (eightbitcolor: 256, lat: 4320, lon: 8640, rgb: 3)\n",
      "Coordinates:\n",
      "  * lat      (lat) float32 89.979164 89.9375 89.89583 ... -89.93751 -89.97918\n",
      "  * lon      (lon) float32 -179.97917 -179.9375 ... 179.93752 179.97917\n",
      "Dimensions without coordinates: eightbitcolor, rgb\n",
      "Data variables:\n",
      "    chlor_a  (lat, lon) float32 ...\n",
      "    palette  (rgb, eightbitcolor) uint8 ...\n",
      "Attributes:\n",
      "    product_name:                      A20191692019176.L3m_8D_CHL_chlor_a_4km.nc\n",
      "    instrument:                        MODIS\n",
      "    title:                             MODISA Level-3 Standard Mapped Image\n",
      "    project:                           Ocean Biology Processing Group (NASA/G...\n",
      "    platform:                          Aqua\n",
      "    temporal_range:                    8-day\n",
      "    processing_version:                2018.0\n",
      "    date_created:                      2019-08-06T01:56:59.000Z\n",
      "    history:                           l3mapgen par=A20191692019176.L3m_8D_CH...\n",
      "    l2_flag_names:                     ATMFAIL,LAND,HILT,HISATZEN,STRAYLIGHT,...\n",
      "    time_coverage_start:               2019-06-18T00:15:00.000Z\n",
      "    time_coverage_end:                 2019-06-26T02:59:59.000Z\n",
      "    start_orbit_number:                91071\n",
      "    end_orbit_number:                  91189\n",
      "    map_projection:                    Equidistant Cylindrical\n",
      "    latitude_units:                    degrees_north\n",
      "    longitude_units:                   degrees_east\n",
      "    northernmost_latitude:             90.0\n",
      "    southernmost_latitude:             -90.0\n",
      "    westernmost_longitude:             -180.0\n",
      "    easternmost_longitude:             180.0\n",
      "    geospatial_lat_max:                90.0\n",
      "    geospatial_lat_min:                -90.0\n",
      "    geospatial_lon_max:                180.0\n",
      "    geospatial_lon_min:                -180.0\n",
      "    latitude_step:                     0.041666668\n",
      "    longitude_step:                    0.041666668\n",
      "    sw_point_latitude:                 -89.979164\n",
      "    sw_point_longitude:                -179.97917\n",
      "    geospatial_lon_resolution:         4.6383123\n",
      "    geospatial_lat_resolution:         4.6383123\n",
      "    geospatial_lat_units:              degrees_north\n",
      "    geospatial_lon_units:              degrees_east\n",
      "    spatialResolution:                 4.64 km\n",
      "    number_of_lines:                   4320\n",
      "    number_of_columns:                 8640\n",
      "    measure:                           Mean\n",
      "    suggested_image_scaling_minimum:   0.01\n",
      "    suggested_image_scaling_maximum:   20.0\n",
      "    suggested_image_scaling_type:      LOG\n",
      "    suggested_image_scaling_applied:   No\n",
      "    _lastModified:                     2019-08-06T01:56:59.000Z\n",
      "    Conventions:                       CF-1.6 ACDD-1.3\n",
      "    institution:                       NASA Goddard Space Flight Center, Ocea...\n",
      "    standard_name_vocabulary:          CF Standard Name Table v36\n",
      "    naming_authority:                  gov.nasa.gsfc.sci.oceandata\n",
      "    id:                                A20191692019176.L3b_8D_CHL.nc/L3/A2019...\n",
      "    license:                           http://science.nasa.gov/earth-science/...\n",
      "    creator_name:                      NASA/GSFC/OBPG\n",
      "    publisher_name:                    NASA/GSFC/OBPG\n",
      "    creator_email:                     data@oceancolor.gsfc.nasa.gov\n",
      "    publisher_email:                   data@oceancolor.gsfc.nasa.gov\n",
      "    creator_url:                       http://oceandata.sci.gsfc.nasa.gov\n",
      "    publisher_url:                     http://oceandata.sci.gsfc.nasa.gov\n",
      "    processing_level:                  L3 Mapped\n",
      "    cdm_data_type:                     grid\n",
      "    identifier_product_doi_authority:  http://dx.doi.org\n",
      "    identifier_product_doi:            10.5067/AQUA/MODIS/L3M/CHL/2018\n",
      "    keywords:                          Earth Science > Oceans > Ocean Chemist...\n",
      "    keywords_vocabulary:               NASA Global Change Master Directory (G...\n",
      "    data_bins:                         9327283\n",
      "    data_minimum:                      0.005913759\n",
      "    data_maximum:                      99.95763\n",
      "\n",
      "Time coverage start: 2019-06-18T00:15:00.000Z\n"
     ]
    }
   ],
   "source": [
    "# Specify the dataset you want to explore\n",
    "ds = m[2]\n",
    "\n",
    "# Print the selected dataset to see what dimensions, coordinates, variables, and attributes it contains.\n",
    "print(ds)\n",
    "print()\n",
    "\n",
    "# Print desired attributes for easier reading\n",
    "print(\"Time coverage start:\", ds.attrs['time_coverage_start'])      # Confirm time of data collections for dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting\n",
    "\n",
    "### Map Plots\n",
    "\n",
    "Now we can begin generating plots of our MODIS satellite chlorophyll data. Overall, we're focused on three of the OOI Regional Cabeled Array locations at the Oregon Coast, so we'll eventually want to create a map that includes these coordinates.\n",
    "\n",
    "#### Interesting Locations and coordinates:\n",
    "\n",
    "Site Name            |   Lat       |        Lon\n",
    "------------------   |   ---       |        ---\n",
    "Hawaii - USA         |  19.8968    |      -155.5828\n",
    "Wedell Sea           |  -71.1879   |      -45.0000\n",
    "Strait of Gibralter  |  35.9982    |      -5.6879\n",
    "\n",
    "#### Things to improve on in this section:\n",
    "* write code that allows users to specify central location and then plots an area of desired size around said location\n",
    "\n",
    "For the zoomed out plots:\n",
    "\n",
    "* add titles to plots so viewers can see what data range each map plot corresponds to. Entering `title=modis.attrs[\"time_coverage_start\"]` into the `plot.contourf` line doesn't seem to work, nor does adding `modisplot.title(\"modis.attrs[\"time_coverage_start\"])` on the next line. Simply typing `print (modis.attrs[\"time_coverage_start\"]` in the plotting loop prints out all of the dates at once, before the plots have time to load.\n",
    "* place plots into subfigure locations to make viewing easier\n",
    "* add plot markers to indicate locations of RCA. Kept getting QuadMesh errors when trying to add markers before.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start with a zoomed in plot of sea surface chlorophyll in the approximate location of the OOI Axial Base shallow profiler. The following cell is working with the previously selected dataset.\n",
    "* location of axial base shallow profiler: -129.75333 45.83049 (from table above)\n",
    "* Resolution of 0.1 degrees with automated colorbar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Select the desired location to plot\n",
    "lats = 19.8968\n",
    "lons = -155.5828\n",
    "res = 1\n",
    "s_lat1 = (lats - 1)\n",
    "s_lat2 = (lats + 1)\n",
    "s_lon1 = (lons - 1)\n",
    "s_lon2 = (lons + 1)\n",
    "\n",
    "modis = ds.sel(lat=slice(s_lat2,s_lat1), lon=slice(s_lon2,s_lon1))   \n",
    "\n",
    "# Uncomment these to find the file name and details for the dataset you are plotting\n",
    "# print(modis.product_name)\n",
    "# print(modis.chlor_a)\n",
    "\n",
    "# Set up a plot with a specified figure size and set lat/lon to appropriate axes\n",
    "modisplot = modis.chlor_a.plot(x='lon', y='lat', figsize=(12,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, zoom out to look at the Oregon Coast region. White areas on the map represent missing data. The Orgeon coast can be seen on the right side of the map (colored white because there's no chlorophyll data on land). Cloud cover is responsible for white diamonds oversea and is varied across datasets. Some days have too much cloud cover to see significant trends, others have nearly no cloud cover and provide an excellent view.\n",
    "\n",
    "Here, we're looking at a region that encompasses the three OOI RCA locations in the table above. Setting the colorbar to a high resolution with a relatively low max value elucidates the offshore eddies and swirls of chlorophyll.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set colorbar values. May take trial and error to get the level of detail you are aiming for.\n",
    "vmin = 0.0\n",
    "vmax = 1.0\n",
    "\n",
    "# Set levels for resolution of colorbar. Change the 0.01 value for higher or lower resolution. \n",
    "lvl = np.arange(vmin, vmax, 0.01).tolist()\n",
    "\n",
    "# Create a plot for every open dataset\n",
    "for ds in m:\n",
    "    modis = ds.sel(lat=slice(47.5,44.4), lon=slice(-130,-123.5))    # Select location\n",
    "    #print(modis.product_name)\n",
    "    #print(modis.lat.values[0], modis.lon.values[0], modis.lat.values[-1], modis.lon.values[-1])\n",
    "    #print(modis.chlor_a)\n",
    "    modisplot = modis.chlor_a.plot.contourf(x='lon', y='lat', figsize=(12,12), vmin=0, vmax=1, levels=lvl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining datasets\n",
    "\n",
    "Now that we've explored the individual datasets, we will combine them to get a timeseries of chlorophyll data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This preprocessor ensures that each new dataset is given a unique time value for its \n",
    "#   new time coordinate variable.\n",
    "\n",
    "timeseries_timestamps = []\n",
    "\n",
    "def preprocessor(ds):\n",
    "    global timeseries_timestamps\n",
    "    ds1 = ds.sel(lat=slice(47.5,44.4), lon=slice(-130,-123.5))    # selecting only the desired lat and lon speeds up data processing\n",
    "    s = ds1.attrs['time_coverage_start']    # using time_coverage_start instead of date_created\n",
    "    this_timestamp = pd.Timestamp(s)\n",
    "    while this_timestamp in timeseries_timestamps: this_timestamp += timedelta(hours = 1)\n",
    "    timeseries_timestamps.append(this_timestamp)\n",
    "    ds1['time'] = xr.Variable('time', [this_timestamp])\n",
    "    return ds1\n",
    "\n",
    "\n",
    "# Combine all the open datasets into a multifile dataset along a new coordinate, time\n",
    "# Also create a list of dates \n",
    "dates_list = []\n",
    "for n in m:\n",
    "    name = n.attrs['product_name']\n",
    "    ds = xr.open_mfdataset('../data/modis/*.nc', combine='by_coords', preprocess = preprocessor)\n",
    "    dates_list.append(n.attrs['time_coverage_start'])\n",
    "print(\"Dates list:\")\n",
    "dates_list    # check that dates are chronological\n",
    "print()\n",
    "\n",
    "# View details of new combined dataset to check that location selection and combination along new coordinate worked\n",
    "print(\"New multifile dataset:\")\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time Series Plots\n",
    "\n",
    "Now let's make time series plots of the chlorophyll a values collected by MODIS at three locations. Missing data is due to cloud coverage. \n",
    "\n",
    "Again, the locations and coordinates of our three points of interest:\n",
    "\n",
    "Site                 |   Lat       |        Lon\n",
    "------------------   |   ---       |        ---\n",
    "Oregon Offshore      |  44.37415   |      -124.95648\n",
    "Oregon Slope Base    |  44.52897   |      -125.38966\n",
    "Axial Base           |  45.83049   |      -129.75326\n",
    "\n",
    "Keep in mind that we are plotting using `method=\"nearest\"` and the MODIS data may not contain data at the exact coordinates of the RCA locations. The satellite resolution is good enough that this should not be a major concern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a time series plot for Oregon Offshore MODIS data\n",
    "\n",
    "fig, axes = plt.subplots(figsize=(14,5))\n",
    "axes.set(xlabel = 'Date', ylabel = 'Concentration (mg m^-3)', title = 'Chlorophyll over Summer 2019 at Oregon Offshore')\n",
    "ds1 = ds.sel(lat=44.37415, lon=-124.95648, method=\"nearest\")    # select location\n",
    "axes.plot(ds1['time'], ds1.chlor_a, marker='o') \n",
    "\n",
    "print(ds1.coords)    # print the coordinates that were plotted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a table of [date | chlor] for Oregon Offshore\n",
    "# This cell is much faster after adding location selection to the preprocessor function! \n",
    "\n",
    "ds1 = ds.sel(lat=44.37415, lon=-124.95648, method=\"nearest\")\n",
    "chlor = ds1.chlor_a.values\n",
    "dates = ds1.time.values\n",
    "\n",
    "for i in range(len(ds1.time)):\n",
    "    print(\"date: \", ds1.time.values[i], \" chlor: \", ds1.chlor_a.values[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a time series plot for Oregon Slope Base MODIS data\n",
    "\n",
    "fig, axes = plt.subplots(figsize=(14,5))\n",
    "axes.set(xlabel = 'Date', ylabel = 'Concentration (mg m^-3)', title = 'Chlorophyll over Summer 2019 at Oregon Slope Base')\n",
    "ds1 = ds.sel(lat=44.52897, lon=-125.38966, method=\"nearest\") # select location\n",
    "axes.plot(ds1['time'], ds1.chlor_a, marker='o') \n",
    "\n",
    "print(ds1.coords)    # print the coordinates that were plotted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a table of [date | chlor] for Oregon Slope Base\n",
    "\n",
    "ds1 = ds.sel(lat=44.52897, lon=-125.38966, method=\"nearest\")\n",
    "chlor = ds1.chlor_a.values\n",
    "dates = ds1.time.values\n",
    "\n",
    "for i in range(len(ds1.time)):\n",
    "    print(\"date: \", ds1.time.values[i], \"chlor: \", ds1.chlor_a.values[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a time series plot for Axial Base MODIS data\n",
    "\n",
    "fig, axes = plt.subplots(figsize=(14,5))\n",
    "axes.set(xlabel = 'Date', ylabel = 'Concentration (mg m^-3)', title = 'Chlorophyll over Summer 2019 at Axial Base')\n",
    "ds1 = ds.sel(lat=45.83049, lon=-129.75326, method=\"nearest\")\n",
    "axes.plot(ds1['time'], ds1.chlor_a, marker='o') \n",
    "\n",
    "print(ds1.coords)    # print the coordinates that were plotted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a table of [date | chlor] for Axial Base MODIS data\n",
    "# Note that there are more data points in the table below than the plot above would imply - fixed this by adding markers to plots\n",
    "\n",
    "ds1 = ds.sel(lat=45.83049, lon=-129.75326, method=\"nearest\")\n",
    "chlor = ds1.chlor_a.values\n",
    "dates = ds1.time.values\n",
    "\n",
    "for i in range(len(ds1.time)):\n",
    "    print(\"date: \", ds1.time.values[i], \"chlor: \", ds1.chlor_a.values[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### intermezzo remarks\n",
    "\n",
    "- The image saver below does not work properly; I just get an empty image...\n",
    "- The nbextension command should *not* be necessary but it is for the record:\n",
    "  \n",
    "```\n",
    "jupyter nbextension enable --py --sys-prefix ipyleaflet\n",
    "```\n",
    "\n",
    "- install the MODIS WMS from [here](https://lpdaac.usgs.gov/data_access/web_map_services_wms)\n",
    "\n",
    "So basically lots to do here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bug: This does not work properly yet\n",
    "\n",
    "# plt.savefig(data + '/images/modis_SST.png')\n",
    "import matplotlib\n",
    "img=matplotlib.image.imread(image_d + 'modis_sst.png')\n",
    "imgplot = plt.imshow(img)\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The goal is to overlay an image (created earlier; .png file) with a controllable opacity\n",
    "# \n",
    "# How I got to opacity control through a sequence of print statements: \n",
    "#   print(thisMap)\n",
    "#   print(thisMap.layers)\n",
    "#   print(thisMap.layers[1])                    which is cheating by hardcoding the layer index as 1\n",
    "#   print(thisMap.layers[1].opacity)\n",
    "#   therefore setting thisMap.layers[1].opacity = 0.2 works just fine\n",
    "\n",
    "def ChangeMapOpacity(opacity):thisMap.layers[1].opacity = opacity\n",
    "\n",
    "from ipyleaflet import Map, ImageOverlay, WMSLayer\n",
    "opacity = 0.7\n",
    "thisMap = Map(center=(47, -129), zoom=5, layout=Layout(width='100%', height='600px'))\n",
    "# The path to the image for overlay is relative; I could not get an absolute path to work...\n",
    "sourceImage =  '../data/images/modis_sst.png'\n",
    "image_layer = ImageOverlay(url=sourceImage, bounds=((40.2, -138.58), (53.8, -122.45)), opacity=opacity)\n",
    "thisMap.add_layer(image_layer)\n",
    "\n",
    "\n",
    "interact(ChangeMapOpacity, opacity = widgets.FloatSlider(min=0., max=1., step=0.025, value=opacity, \n",
    "      continuous_update=False, display='Opacity'))\n",
    "\n",
    "# It seems that 'thisMap' must be the last line of code in the cell, hence placing interact() above\n",
    "thisMap\n",
    "\n",
    "# The following works but does not track zoom with new higher resolution tiles\n",
    "# wms = WMSLayer(url=\"https://demo.boundlessgeo.com/geoserver/ows?\",layers=\"nasa:bluemarble\")\n",
    "# thisMap.add_layer(wms)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell selects out MODIS-derived sea surface temperature in deg C at specific locations\n",
    "#   This is for comparative work with in situ sensors in the Regional Cabled Observatory, which is\n",
    "#   in turn part of the NSF-sponsored Ocean Observing Initiative\n",
    "\n",
    "# OSB data coughs up 44.52897 -125.38966\n",
    "\n",
    "eoLat, eoLon, eoDep          = 44. + 22./60. + 10./3600., -(124. + 57./60. + 15./3600.),  582.\n",
    "osbLat, osbLon, osbDep       = 44. + 30./60. + 55./3600., -(125. + 23./60. + 23./3600.), 2906.\n",
    "shrLat, shrLon, shrDep       = 44. + 34./60. +  9./3600., -(125  +  8./60. + 53./3600.),  778.\n",
    "axbLat, axbLon, axbDep       = 45. + 49./60. +  5./3600., -(129. + 45./60. + 13./3600.), 2605.\n",
    "ashesLat, ashesLon, ashesDep = 45. + 56./60. +  1./3600., -(130. +  0./60. + 50./3600.), 1543.\n",
    "axcLat, axcLon, axcDep       = 45. + 57./60. + 17./3600., -(130. +  0./60. + 32./3600.), 1528.\n",
    "axeLat, axeLon, axeDep       = 45. + 56./60. + 23./3600., -(129. + 58./60. + 27./3600.), 1516.\n",
    "axiLat, axiLon, axiDep       = 45. + 53./60. + 35./3600., -(129. + 58./60. + 44./3600.), 1520.\n",
    "ca = [('Endurance Offshore',                eoLat,    eoLon,    eoDep, 'Endurance Offshore'),\n",
    "      ('Oregon Slope Base',                osbLat,   osbLon,   osbDep, 'Oregon Slope Base'),\n",
    "      ('Southern Hydrate Ridge',           shrLat,   shrLon,   shrDep, 'Southern Hydrate Ridge'),\n",
    "      ('Axial Base',                       axbLat,   axbLon,   axbDep, 'Axial Base'),\n",
    "      ('Axial ASHES Vent Field',         ashesLat, ashesLon, ashesDep, 'Inferno vent'),\n",
    "      ('Axial Caldera Center',             axcLat,   axcLon,   axcDep, 'Axial Caldera Center'),\n",
    "      ('Axial Caldera East',               axeLat,   axeLon,   axeDep, 'Axial Caldera East'),\n",
    "      ('Axial International Vent Field',   axiLat,   axiLon,   axiDep, 'Axial International Vent Field')\n",
    "    ]\n",
    "\n",
    "# for i in ca: print(i[0], i[1], i[2])\n",
    "# for i in range(3): for j in range(3): print(m.MODISA_L3m_SST_2014_sst[0][i][j].values)\n",
    "# \n",
    "# testLat = ca[1][1]       #   44.51528\n",
    "# testLon = ca[1][2]       # -125.38972\n",
    "# print(testLat, testLon)\n",
    "\n",
    "sst = []\n",
    "for i in range(len(ca)):\n",
    "    # print(m.MODISA_L3m_SST_2014_sst.sel(lat=ca[i][1], lon=ca[i][2], method = 'nearest'))\n",
    "    temp = modis.MODISA_L3m_SST_2014_sst.sel(lat=ca[i][1], lon=ca[i][2], method = 'nearest').values\n",
    "    sst.append(round(float(temp[0]), 2))\n",
    "\n",
    "# Here are MODIS-derived day-time sea surface temperatures at the cabled array locations on 2016-01-01T00:15:10 \n",
    "print(sst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And this works properly albeit with limited tiling resolution\n",
    "from ipyleaflet import Map, WMSLayer\n",
    "wms = WMSLayer(\n",
    "url=\"https://demo.boundlessgeo.com/geoserver/ows?\",\n",
    "layers=\"nasa:bluemarble\"\n",
    ")\n",
    "m = Map(layers=(wms, ), center=(42.5531, -48.6914), zoom=3)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pp(ds):\n",
    "    ds['time'] = xr.Variable('time', [pd.Timestamp(ds.attrs['time_coverage_start']) + pd.Timedelta(15, unit='d')])\n",
    "    return ds\n",
    "\n",
    "cb=xr.open_mfdataset('modis_chlora_2017_*.nc', preprocess = pp, concat_dim='time').chlor_a.to_dataset()\n",
    "# cb.chlor_a shows units as mg / cubic meter, numerically equivalent to ug / liter\n",
    "\n",
    "p,a=plt.subplots(figsize=(13,9))\n",
    "\n",
    "for x in np.arange(-125.38966-0.1, -125.38966+0.1, 0.1):\n",
    "    for y in np.arange(44.52897-0.1, 44.52897+0.1, 0.1):\n",
    "        a.plot(cb.time.values, cb.chlor_a.sel(lon=x, lat=y, method='nearest').values, 'o-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cb.to_netcdf('modis_chlora_5_month_timeseries.nc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "latselect = np.arange(44.52897-1., 44.52897+1.00001, 0.10)\n",
    "lonselect = np.arange(-125.38966-1., -125.38966+1.00001, 0.10)\n",
    "local = cb.sel(lat = latselect, lon = lonselect, method = 'nearest')\n",
    "local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "p,a=plt.subplots(figsize=(13,9))\n",
    "for i in range(9,12):\n",
    "    for j in range(9,12):\n",
    "        a.plot(local.time.values, local.chlor_a.isel(lat=i,lon=j).values, 'o-')\n",
    "\n",
    "a.plot(local.time.values, local.chlor_a.isel(lat=10,lon=10).values, 'o-', linewidth=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.to_netcdf('local.nc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local=xr.open_dataset('local.nc')\n",
    "local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# p,a=plt.subplots(figsize=(13,9))\n",
    "# for t in l1.chlor_a: print(t)  \n",
    "# a.plot(local.time.values, t.values, 'o-')\n",
    "\n",
    "for i in range(21): \n",
    "    for j in range(21): \n",
    "        print(l1.chlor_a[0:5, i, j].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local = cb.reindex_like()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.chlor_a[0].plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=4.0)\n",
    "plt.xlim(-128,-124); plt.ylim(43.5,45.5); plt.plot(-125.38966, 44.52897, '^')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.chlor_a[1].plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=4.0)\n",
    "plt.xlim(-128,-124); plt.ylim(43.5,45.5); plt.plot(-125.38966, 44.52897, '^')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.chlor_a[2].plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=4.0)\n",
    "plt.xlim(-128,-124); plt.ylim(43.5,45.5); plt.plot(-125.38966, 44.52897, '^')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.chlor_a[3].plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=4.0)\n",
    "plt.xlim(-128,-124); plt.ylim(43.5,45.5); plt.plot(-125.38966, 44.52897, '^')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local.chlor_a[4].plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=4.0)\n",
    "plt.xlim(-128,-124); plt.ylim(43.5,45.5); plt.plot(-125.38966, 44.52897, '^')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca=xr.open_rasterio('./chlora.tif').to_dataset(name='chlora')\n",
    "ca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca.chlora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cad=ca.where(ca.chlora<1.4)\n",
    "cad.chlora.plot(figsize=(14,9),cmap=plt.cm.rainbow,vmin=0.00, vmax=1.0)\n",
    "plt.xlim(-128,-124)\n",
    "plt.ylim(43.5,45.5)\n",
    "plt.plot(-125.38966, 44.52897, '^')\n",
    "# a=p.axes()\n",
    "# a.set(xlim=(-155,-135))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "float(ca.sel(band=0,y=44.52897,x=-125.38966, method='nearest').values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in np.arange(-125.5, -125.0, 0.1):\n",
    "    for y in np.arange(44.3, 44.8, 0.1):\n",
    "        print(float(ca.sel(band=0,y=y,x=x, method='nearest').values))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}