Merge pull request #297 from gulfofmaine/fix/erddap-sort

Sort ERDDAP Dataframes by time before trying to select the latest value

Changelog.md

@@ -8,6 +8,8 @@ Changes:
 
 Fixes:
 
+- Sort dataframes from ERDDAP by time before use. IOOS Sensor ERDDAP has a tendency to return data from latest to earliest, rather than from earliest to latest like all other ERDDAPs that I've tested against. [#287](https://github.com/gulfofmaine/buoy_barn/issues/287)
+
 ## 0.1.17 - 02/09/2020
 
 Fixes:

app/deployments/utils/erddap_datasets.py

@@ -45,6 +45,9 @@ def setup_variables(
 
 
 def retrieve_dataframe(server, dataset: str, constraints, timeseries) -> DataFrame:
+    """ Returns a dataframe from ERDDAP for a given dataset
+
+    Attempts to sort the dataframe by time """
     e = setup_variables(
         server.connection(),
         dataset,
@@ -56,4 +59,11 @@ def retrieve_dataframe(server, dataset: str, constraints, timeseries) -> DataFra
         os.environ.get("RETRIEVE_DATAFRAME_TIMEOUT_SECONDS", 60)
     )
 
-    return e.to_pandas(parse_dates=True)
+    df = e.to_pandas(parse_dates=True)
+
+    try:
+        df = df.sort_values("time (UTC)")
+    except KeyError:
+        logger.warning(f"Unable to sort dataframe by `time (UTC)` for {dataset}")
+
+    return df