2019_Iberia01_ESSD.Rmd

---
title: "Iberia01 - A new gridded dataset of daily precipitation and temperatures over Iberia."
subtitle: "Paper notebook - submitted to Earth System Science Data (ESSD)"
author: "S. Herrera, R.M. Cardoso, P.M.M. Soares, F. Espírito-Santo & J. M Gutiérrez"
date: '`r Sys.Date()`'
output:
  pdf_document:
    fig_caption: yes
    highlight: pygments
    latex_engine: pdflatex
    pandoc_args:
    - --number-sections
    - --number-offset=0
    toc: yes
  html_document:
    fig_caption: yes
    highlight: pygments
    number_sections: yes
    theme: readable
    toc: yes
    toc_float: yes
encoding: UTF8
documentclass: article
abstract: Iberia01 is a gridded dataset of daily precipitation and mean, maximum and minimum daily temperatures over the Iberian Peninsula covering the period 1971-2015 at a 0.1º spatial resolution. This dataset has been developed based on previous works done for Spain (Spain02) and Portugal (PT02). The present annex includes the code, in R, to reproduce some of the figures shown in the paper Herrera et al. 2019, Earth System Science Data (ESSD). DOI -  http://dx.doi.org/10.20350/digitalCSIC/8641
urlcolor: blue
---

\fontfamily{cmr}
\fontsize{11}{22}
\selectfont

```{r include=FALSE}
knitr::opts_chunk$set(echo = TRUE,
                      highlight = TRUE,
                      message = FALSE,
                      fig.align = "center",
                      tidy = FALSE,
                      eval = FALSE,
                      fig.width = 7,
                      cache = TRUE,
                      cache.path = "./cache/",
                      fig.path = "./cache/figs")
# Thanks to this:
# https://www.r-bloggers.com/wrapper-of-knitrinclude_graphics-to-handle-urls-pdf-outputs/
# https://github.com/liao961120/linguisticsdown/blob/master/R/include_graphics2.R
```  

# R PACKAGE REQUIREMENTS 

Iberia01 is provided in NetCDF format and, therefore, can be accessed using a number of packages and libraries. In this example, we build on the climate4R (https://github.com/SantanderMetGroup/climate4R) R framework for Climate Data Access and Post-processing which allows to easily read the data, compute indices and validation metrics and plot the results. To ensure reproducibility, it is recommend to install the same versions of the packages that produced this document, even though in most cases more recent versions exist (see the last Session Information Section).

## Installing climate4R

The *climate4R* R framework can be installed using conda or docker. Individual packages can be also installed from the GitHub repositories using the function `install_github` from package `devtools`. For, instance, to install `transformeR` v0.0.14, it suffices with pointing to the specific version tag after the `@` symbol. 

Thus, in order to install the proper versions of the packages of the *climate4R* bundle needed to run these examples:
  
```{r,eval=FALSE}
devtools::install_github(c("SantanderMetGroup/transformeR", "SantanderMetGroup/convertR",
                           "SantanderMetGroup/loadeR", "SantanderMetGroup/visualizeR"))
```

```{r, eval=TRUE, echo = FALSE, message=FALSE}
rm(list = ls())
options(java.parameters = "-Xmx8000m")
library(transformeR)
library(convertR)
library(loadeR)
library(visualizeR)
```

```{r, message=FALSE, eval=FALSE}
rm(list = ls())
options(java.parameters = "-Xmx8000m")
library(loadeR, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
```

## R package for data loading

The R package `loadeR` will perform the data loading task, including authentication against the Santander User Data Gateway (UDG) server (see the [loadeR wiki](https://github.com/SantanderMetGroup/loadeR/wiki)) and enabling the access to any remote dataset via OPeNDAP, as well as creating and accesing datasets locally.

Prior to data access through the Santander Climate Data Service (via the User Data Gateway), authentication is required (see instructions for registration, [https://github.com/SantanderMetGroup/loadeR/wiki/Loading-data-from-the-User-Data-Gateway-(UDG)](https://github.com/SantanderMetGroup/loadeR/wiki/Loading-data-from-the-User-Data-Gateway-(UDG))). Once a user/password have been obtained, the authentication is performed in one step:

```{r,eval=FALSE}
loginUDG(username = "---", password = "---") # Replace "---" by the personal user/password
```

```{r,eval=TRUE, echo = FALSE, message=FALSE}
loginUDG(username = "SixtoHerrera", password = "carlosv") # Replace "---" by the personal user/password
```

## R package for data post-processing and plotting

In addition, the R package `transformeR` enables climate data transformation (plotting, aggregation, subsetting, PCA/EOF analysis ...). Further details on its capabilities and installation instructions are available in the [wiki page](https://github.com/SantanderMetGroup/transformeR/wiki). Moreover, the package `visualizeR` provides different ploting functions.

```{r, message=FALSE, eval=FALSE}
library(transformeR, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
library(convertR, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
library(visualizeR, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
```

### Colour Palettes:

The package `RColorBrewer` is used to replicate the spectral color palette used in the paper in the correlation maps. Next, the palette `veri.colors` is defined, that will be used in the verification maps:

```{r, message=FALSE, eval=FALSE}
library(RColorBrewer, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
cols <- brewer.pal(n = 11, name = "Spectral")
veri.colors <- colorRampPalette(rev(cols))
```

```{r, message=FALSE, eval=TRUE, echo = FALSE}
library(RColorBrewer)
cols <- brewer.pal(n = 11, name = "Spectral")
veri.colors <- colorRampPalette(rev(cols))
```

# Daily Precipitation Climatology:
First, we obtain the daily precipitation climatology and the differences between both datasets, Iberia01 and E-OBS v17.0. To this aim, the dataset and geographical domain should be defined:

```{r,echo=TRUE, message=FALSE, eval=TRUE}
eobsNcML <- "http://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/rr_0.25deg_reg_v17.0.nc"
ib01NcML <- "http://meteo.unican.es/tds5/dodsC/Iberia01/Iberia01_v1.0_010reg_aa_3d.ncml"
## Defining the temporal and geographical domain:
latLim <- c(34, 44)
lonLim <- c(-10, 6)
years <- c(1971:2010)
```

## E-OBS v17.0:
First, we obtain the climatology for the E-OBS v17.0 dataset. To this aim we should know the variables included in the dataset

```{r, echo=TRUE, eval=FALSE, message=FALSE}
di <- dataInventory(eobsNcML)
## Variables:
names(di)
```

and load the corresponding data:

```{r,eval=TRUE, echo = TRUE, message=FALSE}
pr.eobs <- loadGridData(eobsNcML, "rr", dictionary = FALSE, lonLim = lonLim, latLim = latLim, 
                           season = c(1:12), years = years, time = "DD", aggr.d = "sum")
setGridUnits(pr.eobs, "mm", var = "rr")
pr.eobs <- climatology(pr.eobs, clim.fun = list(FUN = "mean", na.rm = TRUE))
spatialPlot(pr.eobs, backdrop.theme = "countries", at = seq(-0.5,6,0.5), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Precipitation (1971-2010)")
```

## Iberia01:

Now, we consider the Iberia01 dataset:

```{r,eval=TRUE, echo = FALSE, message=FALSE}
loginUDG(username = "SixtoHerrera", password = "carlosv") # Replace "---" by the personal user/password
```

```{r,echo=TRUE, eval=FALSE, message=FALSE}
di <- dataInventory(ib01NcML)
## Variables:
names(di)
```

```{r,eval=TRUE, echo = FALSE, message=FALSE}
loginUDG(username = "SixtoHerrera", password = "carlosv") # Replace "---" by the personal user/password
```

```{r,echo=TRUE, eval=TRUE, message=FALSE}
pr.Ib01 <- loadGridData(ib01NcML, "pr", dictionary = FALSE, lonLim = lonLim, latLim = latLim, 
                           season = c(1:12), years = years, time = "DD", aggr.d = "sum")
setGridUnits(pr.Ib01, "mm", var = "pr")
pr.Ib01 <- climatology(pr.Ib01, clim.fun = list(FUN = "mean", na.rm = TRUE))
spatialPlot(pr.Ib01, backdrop.theme = "countries", at = seq(-0.5,6,0.5), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Precipitation (1971-2010)")
```

In order to obtain the difference between both datasets, we interpolate E-OBS v17.0 to the grid of the Iberian01 dataset:

```{r,eval=TRUE, echo = TRUE, message=FALSE}
## E-OBS climatology is regridded to the Iberian01 grid:
pr.eobs <- interpGrid(pr.eobs, new.coordinates = getGrid(pr.Ib01), method = "nearest")
```

## Differences between E-OBS v17.0 and Iberia01:
- Bias:
```{r,eval=TRUE, echo = TRUE, message=FALSE}
bias <- gridArithmetics(pr.eobs, pr.Ib01, operator = "-")
spatialPlot(bias, backdrop.theme = "countries", at = seq(-3,3,0.5), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Bias (1971-2010)")
```

- Relative Bias:

```{r,eval=TRUE, echo = TRUE, message=FALSE}
## Relative Bias
C <- 100
relBias <- gridArithmetics(pr.eobs, pr.Ib01, pr.Ib01, C, operator = c("-", "/", "*"))
spatialPlot(relBias, backdrop.theme = "countries", at = seq(-100,100,10), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Relative Bias (1971-2010)")
```

```{r,eval=TRUE, echo = FALSE, message=FALSE}
rm(list=c("pr.Ib01", "pr.eobs", "bias", "relBias"))
```

# Mean Daily Temperature:

Once the climatologies for some precipitation indices have been obtained we show the climatology of mean temperature. Note that, in this case, the datasets should be redefined, at least for E-OBS v17.0:

```{r,echo=TRUE, message=FALSE, eval = TRUE}
eobsNcML <- "http://opendap.knmi.nl/knmi/thredds/dodsC/e-obs_0.25regular/tg_0.25deg_reg_v17.0.nc"
ib01NcML <- "http://meteo.unican.es/tds5/dodsC/Iberia01/Iberia01_v1.0_010reg_aa_3d.ncml"
## Defining the temporal and geographical domain:
latLim <- c(34, 44)
lonLim <- c(-10, 6)
years <- c(1971:2010)
```

## E-OBS v17.0:

First, we obtain the climatology for the E-OBS v17.0 dataset. To this aim we should know the variables included in the dataset

```{r, echo=TRUE, eval=FALSE, message=FALSE}
di <- dataInventory(eobsNcML)
## Variables:
names(di)
```

and load the corresponding data:

```{r,eval=TRUE, echo = TRUE, message=FALSE}
tg.eobs <- loadGridData(eobsNcML, "tg", dictionary = FALSE, lonLim = lonLim, latLim = latLim, 
                           season = c(1:12), years = years, time = "DD", aggr.d = "mean")
tg.eobs <- climatology(tg.eobs, clim.fun = list(FUN = "mean", na.rm = TRUE))
spatialPlot(tg.eobs, backdrop.theme = "countries", at = seq(0,25,1), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Mean Temperature (1971-2010)")
```

## Iberia01:
Now, we consider the Iberia01 dataset:
```{r,eval=TRUE, echo = FALSE, message=FALSE}
loginUDG(username = "SixtoHerrera", password = "carlosv") # Replace "---" by the personal user/password
```

```{r,echo=TRUE, eval=FALSE, message=FALSE}
di <- dataInventory(ib01NcML)
## Variables:
names(di)
```

```{r,echo=FALSE, eval=TRUE, message=FALSE}
loginUDG(username = "SixtoHerrera", password = "carlosv") # Replace "---" by the personal user/password
tg.Ib01 <- loadGridData(ib01NcML, "tas", dictionary = FALSE, lonLim = lonLim, latLim = latLim, 
                           season = c(1:12), years = years, time = "DD", aggr.d = "mean", aggr.m = "mean")
tg.Ib01 <- climatology(tg.Ib01, clim.fun = list(FUN = "mean", na.rm = TRUE))
spatialPlot(tg.Ib01, backdrop.theme = "countries", at = seq(0,25,1), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Mean Temperature (1971-2010)")
```

```{r,echo=TRUE, eval=FALSE, message=FALSE}
tg.Ib01 <- loadGridData(ib01NcML, "tas", dictionary = FALSE, lonLim = lonLim, latLim = latLim, 
                           season = c(1:12), years = years, time = "DD", aggr.d = "mean")
tg.Ib01 <- climatology(tg.Ib01, clim.fun = list(FUN = "mean", na.rm = TRUE))
spatialPlot(tg.Ib01, backdrop.theme = "countries", at = seq(0,25,1), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Mean Temperature (1971-2010)")
```

In order to obtain the difference between both datasets, we interpolate E-OBS v17.0 to the grid of the Iberian01 dataset:

```{r,eval=TRUE, echo = TRUE, message=FALSE}
## E-OBS climatology is regridded to the Iberian01 grid:
tg.eobs <- interpGrid(tg.eobs, new.coordinates = getGrid(tg.Ib01), method = "nearest")
```

## Differences between E-OBS v17.0 and Iberia01:
- Bias:
```{r,eval=TRUE, echo = TRUE, message=FALSE}
bias <- gridArithmetics(tg.eobs, tg.Ib01, operator = "-")
spatialPlot(bias, backdrop.theme = "countries", at = seq(-5,5,0.5), 
            col.regions = veri.colors(51), colorkey = list(space = "bottom"), 
            main = "Bias (1971-2010)")
```

```{r,eval=TRUE, echo = FALSE, message=FALSE}
rm(list=c("tg.Ib01", "tg.eobs", "bias"))
```

# Additional packages

## Functions for Extreme Value Distributions (evd):

In order to illustrate the analysis included in Herrera et al. 2019 the [evd](https://cran.r-project.org/web/packages/evd/index.html) package should be installed to obtain the Generalized Extreme Value (GEV) distribution.

```{r, message=FALSE, eval=TRUE}
install.packages("evd")
library(evd, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
```

In order to extend the analysis shown in this document to other indices, the reader can consider the following R packages to easily estimate other climate and drought indices.

## R package for Climate Indices calculation: 

`climate4R.indices` is the package to compute several indices within the climate4R framework, therefore is seamlessly integrated with the climate4R data structures, and provides support for parallel computing.

```{r, message=FALSE, eval=TRUE}
devtools::install_github(c("SantanderMetGroup/climate4R.indices"))
library(climate4R.indices, lib.loc = "~/R/x86_64-pc-linux-gnu-library/3.5/")
indexShow()
?indexGrid   # see the examples 
```

-- 

# SESSION INFORMATION AND PACKAGE VERSIONS

```{r,eval=TRUE}
print(sessionInfo(package = c("loadeR", "convertR", "transformeR",
                        "visualizeR", "RColorBrewer")))
```