Merge pull request #90 from EcoJulia/tp/rasterdl

Big Chonky Boi
PoisotLab · Apr 7, 2021 · 42144e1 · 42144e1 · tpoisot · Apr 7, 2021
2 parents 3198306 + e7111bb
commit 42144e1
Show file tree

Hide file tree

Showing 41 changed files with 822 additions and 350 deletions.
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
@@ -14,8 +14,6 @@ jobs:
     strategy:
       matrix:
         version:
-          - '1.3'
-          - '1.4'
           - '1.5'
         os:
           - ubuntu-latest

diff --git a/.github/workflows/CompatHelper.yml b/.github/workflows/CompatHelper.yml
@@ -9,7 +9,7 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        julia-version: [1.4.0]
+        julia-version: [1.5.0]
         julia-arch: [x86]
         os: [ubuntu-latest]
     steps:

diff --git a/Project.toml b/Project.toml
@@ -1,20 +1,19 @@
 name = "SimpleSDMLayers"
 uuid = "2c645270-77db-11e9-22c3-0f302a89c64c"
 authors = ["Timothée Poisot <[email protected]>", "Gabriel Dansereau <[email protected]>"]
-version = "0.4.10"
+version = "0.5.0"
 
 [deps]
 ArchGDAL = "c9ce4bd3-c3d5-55b8-8973-c0e20141b8c3"
-HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3"
+Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
 RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
 Requires = "ae029012-a4dd-5104-9daa-d747884805df"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 ZipFile = "a5390f91-8eb1-5f08-bee0-b1d1ffed6cea"
 
 [compat]
 ArchGDAL = "0.4, 0.5, 0.6"
-HTTP = "0.8, 0.9"
 RecipesBase = "0.7, 0.8, 1.0"
 Requires = "1.0"
 ZipFile = "0.8, 0.9"
-julia = "1.3"
+julia = "1.5"
diff --git a/README.md b/README.md
@@ -1,7 +1,7 @@
 ## Simple Layers for Species Distributions Modelling
 
-This package offers very simple types and functions to interact with
-bioclimatic data and the output of species distribution models.
+This package offers very simple types and functions to interact with bioclimatic
+data and the output of species distribution models.
 
 [![d_stable](https://img.shields.io/badge/Doc-stable-green?style=flat-square)](https://ecojulia.github.io/SimpleSDMLayers.jl/stable/)
 [![d_latest](https://img.shields.io/badge/Doc-latest-blue?style=flat-square)](https://ecojulia.github.io/SimpleSDMLayers.jl/latest/)
@@ -19,8 +19,9 @@ bioclimatic data and the output of species distribution models.
        title = "SimpleSDMLayers.jl & GBIF.jl example">
 </p>
 
-Curious to know more? Have a look at our [paper in Journal of Open Source Software](https://doi.org/10.21105/joss.02872), our [JuliaCon poster](https://github.com/gabrieldansereau/juliacon-2020-poster/blob/master/juliacon-poster.pdf), our [NextJournal demo notebook](https://nextjournal.com/gabrieldansereau/SimpleSDMLayers-JuliaCon2020-demo/), and our [extended documentation](https://ecojulia.github.io/SimpleSDMLayers.jl/stable/), or keep reading for a quick overview.
+Curious to know more? Have a look at our [paper in Journal of Open Source Software][joss], our [JuliaCon poster](https://github.com/gabrieldansereau/juliacon-2020-poster/blob/master/juliacon-poster.pdf), our [NextJournal demo notebook](https://nextjournal.com/gabrieldansereau/SimpleSDMLayers-JuliaCon2020-demo/), and our [extended documentation](https://ecojulia.github.io/SimpleSDMLayers.jl/stable/), or keep reading for a quick overview.
 
+[joss]: https://doi.org/10.21105/joss.02872
 ### Installation
 
 The currently released version of the package can be installed with:
@@ -68,19 +69,22 @@ are immutable.
 
 ### Bioclimatic data
 
-#### WorldClim 2.1
-
-The `worldclim` function will get a range, or an array of indices, and return
-the corresponding bioclim 2.1 layers at the specified `resolution`. For
-example, to get the annual temperature, and annual precipitation:
-
-~~~ julia
-temperature, precipitation = worldclim([1,12])
-~~~
-
-By default, the function will return the layers for the entire globe, and they
-can be cropped later. The layers are returned as `SimpleSDMPredictor` objects.
-
+| Data provider                    | Dataset                | Layers | Future models | Future scenarios                     |
+| -------------------------------- | ---------------------- | ------ | ------------- | ------------------------------------ |
+| `EarthEnv`                       | `Landcover`            | 12     |               |                                      |
+| `EarthEnv`                       | `HabitatHeterogeneity` | 14     |               |                                      |
+| [`WorldClim`][worldclim-current] | `BioClim`              | 19     | `CMIP6`       | `SharedSocioeconomicPathway`         |
+| [`CHELSA`][chelsa-bioclim]       | `BioClim`              | 12     | `CMIP5`       | `RepresentativeConcentrationPathway` |
+
+[earthenv-landcover]: http://www.earthenv.org/landcover
+[earthenv-texture]: http://www.earthenv.org/texture
+[worldclim-current]: https://www.worldclim.org/data/worldclim21.html
+[chelsa-bioclim]: http://chelsa-climate.org/
+
+When downloaded (using `SimpleSDMPredictor`), the layers are stored either in an
+`assets` subfolder of the current project (strongly advised against), or at the
+location determined by the `SDMLAYERS_PATH` environment variable. The datasets/providers
+with future models and scenarios also accept years.
 
 ### Plotting
 
@@ -89,6 +93,7 @@ Using the `Plots` package, one can call the `heatmap`, `contour`, `density`
 `heatmap`.
 
 ~~~ julia
+temperature = SimpleSDMPredictor(WorldClim, BioClim, 1)
 plot(temperature)
 ~~~
 

diff --git a/docs/make.jl b/docs/make.jl
@@ -17,15 +17,15 @@ makedocs(
         ],
         "Examples" => [
             "Temperature data" => "examples/temperature.md",
-            "GBIF integration" => "examples/gbif.md",
             "DataFrames integration" => "examples/dataframes.md",
-            "Importing raster data" => "examples/import.md",
             "Sliding window analysis" => "examples/slidingwindow.md",
             "Landcover data" => "examples/landcover.md",
             "Landcover consensus" => "examples/consensus.md"
         ],
         "Building SDMs" => [
-            "BIOCLIM from scratch" => "sdm/bioclim.md"
+            "GBIF integration" => "sdm/gbif.md",
+            "BIOCLIM from scratch" => "sdm/bioclim.md",
+            "Future data" => "sdm/future.md"
         ]
     ]
 )

diff --git a/docs/src/examples/consensus.md b/docs/src/examples/consensus.md
@@ -12,72 +12,34 @@ using Plots
 bbox = (left=8.25, right=10.0, bottom=41.2, top=43.2)
 ```
 
-We will then do two things. First, get the first layer of landcover (see the
-help of `landcover` for a list of the layers), and then create a datacube,
-organized around dimensions of latitude, longitude, and layer value - we will
-only focus on the 11 first variables, since we do not want the information on
-open water (layer 12):
+First, we will download all values for our layers:
 
 ```@example cons
-lc = landcover(1; full=false, bbox...)
-use = fill(NaN32, size(lc)..., 11)
+lc = convert.(Float32, SimpleSDMPredictor(EarthEnv, LandCover, 1:12; full=false, bbox...))
 ```
 
-At this point, we will simply fill in the first "slice" of our datacube with
-values from the layer:
+To perform the actual analysis, we will define a `shannon` function, which will
+return the entropy of the land use categories:
 
 ```@example cons
-for (i,e) in enumerate(lc.grid)
-    coord = (CartesianIndices(size(lc.grid))[i].I..., 1)
-    if !isnothing(e)
-        use[coord...] = e
-    end
-end
-```
-
-The next step is to repeat this process for all other layers, filling the
-appropriate data cube slice:
-
-```@example cons
-for layer in 2:11
-    lc = landcover(layer; full=false, bbox...)
-    for (i,e) in enumerate(lc.grid)
-        coord = (CartesianIndices(size(lc.grid))[i].I..., layer)
-        if !isnothing(e)
-            use[coord...] = e
-        end
-    end
-end
-```
-
-To perform the actual analysis, we will define a `get_most_common_landuse` function, which will return the index of the layer with the highest score:
-
-```@example cons
-function get_most_common_landuse(f)
-    f[isnan.(f)] .= 0.0
-    sum(f) == 0 && return NaN
-    return last(findmax(f))
-end
-
 function shannon(x)
-    v = filter(!isnan, x)
+    v = filter(n -> n>zero(eltype(x)), x)
     length(v) == 0 && return NaN
     v = v ./ sum(v)
     return -sum(v.*log2.(v))
 end
 ```
 
-```@example cons
-consensus = mapslices(get_most_common_landuse, use; dims=3)[:,:,1]
-entropy = mapslices(shannon, use; dims=3)[:,:,1]
+We can then apply these functions using the `mosaic` method:
 
-consensus = SimpleSDMResponse(consensus, lc)
-entropy = SimpleSDMResponse(entropy, lc)
+```@example cons
+consensus = mosaic(x -> last(findmax(x)), lc)
+entropy = mosaic(shannon, lc)
 ```
 
 ```@example cons
-p1 = plot(consensus, c=cgrad(:Set3_11, categorical=true), frame=:none)
+p1 = plot(consensus, c=:terrain, frame=:none)
 p2 = plot(entropy, c=:bamako, frame=:none)
 
-plot(p1, p2, size=(900, 400), dpi=600)
+plot(p1, p2)
 ```
diff --git a/docs/src/examples/landcover.md b/docs/src/examples/landcover.md
@@ -8,34 +8,26 @@ limits of a bounding box.
 
 ```@example urban
 using SimpleSDMLayers
-urban = landcover(9; left=-11.0, right=31.1, bottom=29.0, top=71.1)
+urban = SimpleSDMPredictor(EarthEnv, LandCover, 9; left=-11.0, right=31.1, bottom=29.0, top=71.1)
 ```
 
 This dataset is returning data as `UInt8` (as it represents a proportion of the
 pixel occupied by the type), but this is not something that can be plotted
 efficiently. So in the next step, we will manipulate this object a little bit to
 have something more workable.
 
-Let's start by preparing a new grid, with the same dimensions, but a friendlier
-type, and then we can then fill these values using a simple rule of using either
-`NaN` or the converted value:
-
 ```@example urban
-n_urban_grid = zeros(Float32, size(urban));
-for (i,e) in enumerate(urban.grid)
-  n_urban_grid[i] = isnothing(e) ? NaN32 : Float32(e)
-end
+urban = convert(Float32, urban)
 ```
 
-We can now overwrite our `urban` object as a layer:
+We will replace the values of 0 by `nothing`, to only see the pixels with some
+urban cover:
 
 ```@example urban
-urban = SimpleSDMPredictor(n_urban_grid, urban)
+replace!(urban, zero(eltype(urban)) => nothing)
 ```
 
-Note that the previous instruction uses a shortcut where the bounding box from a
-new `SimpleSDMLayer` is drawn from the bounding box for an existing layer. With
-this done, we can show the results:
+With this done, we can plot the results:
 
 ```@example urban
 using Plots

diff --git a/docs/src/examples/slidingwindow.md b/docs/src/examples/slidingwindow.md
@@ -9,7 +9,7 @@ using SimpleSDMLayers
 using Plots
 using Statistics
 
-precipitation = worldclim(12; left=-80.0, right=-56.0, bottom=44.0, top=62.0)
+precipitation = SimpleSDMPredictor(WorldClim, BioClim, 12; left=-80.0, right=-56.0, bottom=44.0, top=62.0)
 ```
 
 The sliding window works by taking all pixels *within a given radius* (expressed

diff --git a/docs/src/examples/temperature.md b/docs/src/examples/temperature.md
@@ -7,7 +7,7 @@ for each layers are in the function documentation):
 
 ```@example temp
 using SimpleSDMLayers
-temperature = worldclim(1)
+temperature = SimpleSDMPredictor(WorldClim, BioClim, 1)
 ```
 
 Thanks to the integration with Plots and StatsPlots, we can very rapidly

diff --git a/docs/src/man/data.md b/docs/src/man/data.md
@@ -4,26 +4,60 @@ The package offers access to bioclimatic and other datasets - they are
 downloaded, saved to the disk, and then read locally. Please note that some of
 them require a lot of memory, so make sure your machine can handle them.
 
-## Worldclim 2.1
+By default, the layers are stored in the `assets` subfolder of the current
+project. This being said, the prefered solution is to define a `SDMLAYERS_PATH`
+environment variable pointing to a specific path, where the layers will live.
+This will ensure that they are re-used between projects.
+
+## General interface
+
+All layers are returned as `SimpleSDMPredictor`, and therefore constructed by
+calling the `SimpleSDMPredictor` function on a `LayerProvider` and a
+`LayerDataset`, possibly with a future climate model and scenario. In all cases,
+the method accepts either a single layer, or an array of layers.
+
+| Data provider                    | Dataset                | Layers | Future models | Future scenarios                     |
+| -------------------------------- | ---------------------- | ------ | ------------- | ------------------------------------ |
+| `EarthEnv`                       | `Landcover`            | 12     |               |                                      |
+| `EarthEnv`                       | `HabitatHeterogeneity` | 14     |               |                                      |
+| [`WorldClim`][worldclim-current] | `BioClim`              | 19     | `CMIP6`       | `SharedSocioeconomicPathway`         |
+| [`CHELSA`][chelsa-bioclim]       | `BioClim`              | 12     | `CMIP5`       | `RepresentativeConcentrationPathway` |
+
+[earthenv-landcover]: http://www.earthenv.org/landcover
+[earthenv-texture]: http://www.earthenv.org/texture
+[worldclim-current]: https://www.worldclim.org/data/worldclim21.html
+[chelsa-bioclim]: http://chelsa-climate.org/
+
+## Later providers
 
 ```@docs
-worldclim
+SimpleSDMLayers.LayerProvider
+WorldClim
+CHELSA
+EarthEnv
 ```
 
-## CHELSA V1
+## Layer datasets
 
 ```@docs
-bioclim
+SimpleSDMLayers.LayerDataset
+BioClim
+LandCover
+HabitatHeterogeneity
 ```
 
-## EarthEnv landcover
+## Future climate models
 
 ```@docs
-landcover
+SharedSocioeconomicPathway
+RepresentativeConcentrationPathway
+CMIP5
+CMIP6
 ```
 
-## ASCII files
+## File reading and writing
 
 ```@docs
 SimpleSDMLayers.ascii
+geotiff
 ```
diff --git a/docs/src/sdm/bioclim.md b/docs/src/sdm/bioclim.md
@@ -40,7 +40,7 @@ example, we will take all worldclim data, at the default 10 arc minute
 resolution:
 
 ```@example bioclim
-predictors = worldclim(1:19; left=left, right=right, bottom=bottom, top=top);
+predictors = SimpleSDMPredictor(WorldClim, BioClim, 1:19; left=left, right=right, bottom=bottom, top=top);
 first(predictors)
 ```