-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.Rmd
255 lines (182 loc) · 10.8 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
---
output: github_document
editor_options:
chunk_output_type: console
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
<!-- badges: start -->
[](https://github.com/hypertidy/lazyraster/actions)
[](https://www.tidyverse.org/lifecycle/#maturing) [](https://cran.r-project.org/package=lazyraster)
[](https://cran.r-project.org/package=lazyraster)
<!-- badges: end -->
```{r setup, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
library(raadtools)
```
# lazyraster
The goal of lazyraster is to get raster data on demand at the right resolution. This means that you can
define a graphics device and then stream just the right amount of pixels to fill it from a GDAL data source.
## Installation
The package can be installed from Github.
```R
## install.packages("remotes")
remotes::install_github("hypertidy/lazyraster")
```
See [vapour](https://github.com/hypertidy/vapour) for more on the installation prerequisites.
## Details
There are functions `lazyraster()` to act like the `raster::raster()` function and provide information but no data, and `crop()` to act like `raster::crop()` and then `as_raster` to break the lazy chain and build an actual raster object. The size of the currently open (or latent-default) device is used as a reasonable size for the output grid, but can be controlled by argument `dim`.
When the data is read `lazyraster` can specify the exact dimensions of the output raster, and by default a reasonable guess at the number of pixels required to fill the current device is used.
A variety of resampling algorithms are available (nearest neighbour is the default, [see this list for more](https://github.com/hypertidy/vapour/blob/master/R/raster-input.R#L9)) and will be applied to reduce or increase the resolution.
## Limitations
We can't utilize the RasterIO level-of-detail functionality for non-GDAL sources.
We can't control the details of the data type. (This is WIP)
## GDAL
This uses a standard internal functionality of GDAL, the [RasterIO function of the GDALRasterBand](https://gdal.org/doxygen/classGDALRasterBand.html). This is used in a lot of different software, and is obviously pretty robust and well tested by the GDAL community, but I only really have experience with one product (commercial, now defunct) that used it extensively for live interactive visualization and data streaming. I haven't found any problems with it at all using it in R, but the support for it is very minimal. You can access it indirectly using `rgdal::readGDAL` for the underlying function, as the `raster` package does, and there is the obscure 'RasterIO' list in the `stars::read_stars()`, but so far I can't get this to work for tile servers.
## vapour
To make this work we use the GDAL package [vapour](https://github.com/hypertidy/vapour). All of the ease-of-use code is in this package, `vapour` is pointedly bare-bones and provides very little interpretation of a data source because it is designed for use in development.
## Examples
Connect lazily to a GeoTIFF, see details of what's there, crop to a section and then read it in and plot.
This is not a huge file, but is easily accessible and demonstrates the idea.
First we connect to a source and show two versions, the first is information about the data in its native form (286 rows and 143 columns), and then an actual RasterLayer but at a very small requested size (24 rows by 12 columns).
```{r GeoTIFF}
sstfile <- system.file("extdata/sst.tif", package = "vapour")
library(lazyraster)
lazy <- lazyraster(sstfile)
lazy ## stay lazy
## be only so lazy
as_raster(lazy, dim = c(12, 24))
```
The call to `as_raster` read actual data from the file, hence the difference between the range of data values reported first from the whole extent, and then reported by raster itself for the resample data read in.
Now let `lazyraster` make its own choice about the size of the output. This will be based on the return value of `dev.size("px")`.
```{r latent-device}
## note how we actually resample up because this data is not very large
as_raster(lazy)
```
More concretely, if we open a graphics device at a given size the raster data read in will match it. (This is not the best choice but works fine for demonstration and experimenting.)
```{r png-device}
## what do we get if we set up a bitmap device
tf <- tempfile(fileext = "png")
png(tf, height = 50, width = 40)
as_raster(lazy)
#plot(as_raster(lazy))
dev.off()
unlink(tf)
```
This will work on really big files or online tile servers. Here are some online examples.
```{r tile-serv}
## a global satellite image tile server
vearth <- '<GDAL_WMS> <Service name="VirtualEarth">
<ServerUrl>http://a${server_num}.ortho.tiles.virtualearth.net/tiles/a${quadkey}.jpeg?g=90</ServerUrl></Service>
<MaxConnections>4</MaxConnections> <Cache/> </GDAL_WMS>'
## a USA topographic map service
usgs <- paste0("WMTS:https://basemap.nationalmap.gov/arcgis/rest/services/USGSTopo/",
"MapServer/WMTS/1.0.0/WMTSCapabilities.xml,layer=USGSTopo,tilematrixset=default028mm")
## a global elevation service
awselev <- "<GDAL_WMS> <Service name=\"TMS\"> <ServerUrl>https://s3.amazonaws.com/elevation-tiles-prod/geotiff/${z}/${x}/${y}.tif</ServerUrl> </Service> <DataWindow> <UpperLeftX>-20037508.34</UpperLeftX> <UpperLeftY>20037508.34</UpperLeftY> <LowerRightX>20037508.34</LowerRightX> <LowerRightY>-20037508.34</LowerRightY> <TileLevel>14</TileLevel> <TileCountX>1</TileCountX> <TileCountY>1</TileCountY> <YOrigin>top</YOrigin> </DataWindow> <Projection>EPSG:3857</Projection> <BlockSizeX>512</BlockSizeX> <BlockSizeY>512</BlockSizeY> <BandsCount>1</BandsCount> <DataType>Int16</DataType> <ZeroBlockHttpCodes>403,404</ZeroBlockHttpCodes> <DataValues> <NoData>-32768</NoData> </DataValues> <Cache/> </GDAL_WMS>"
library(raster)
ve <- lazyraster(vearth)
#plotRGB(as_raster(ve, band = 1:3, c(1024, 1024)))
#(ve1 <- as_raster(ve, band = 1:3, c(1024, 1024)))
plotRGB(as_raster(ve, band = 1:3, c(1024, 1024)))
# raster::plotRGB(as_raster(crop(ve, merc(0, 51.5, wh = 1e4)), band = 1:3, c(1024, 1024)))
#
# us <- lazyraster(usgs)
# raster::plotRGB(as_raster(us, band = 1:3, c(1024, 1024)))
# raster::plotRGB(as_raster(crop(us, merc(-74, 41), wh = 150e3), dev.size("px"), band = 1:3, resample = "bilinear"))
#
# elev <- lazyraster(awselev)
# plot(elev, col = grey.colors(24))
# plot(crop(elev, merc(147, -41, wh = 150e3)), zlim = c(0, NA), col = grey.colors(24))
```
(This example can't work on your computer probably given use of local raadtools, but try it on your favourite big file).
This takes a fairly large grid and plots just enough detail by reading just enough detail for the plot space. That's all that happens.
```{r raadtools}
library(raadtools)
f <- raadtools::topofile("gebco_19")
lazyraster(f)
library(raster)
op <- par(mar = rep(0, 4))
system.time({
rworld <- lazyraster(f)
plot(rworld, col = grey(seq(0, 1, length = 100)), axes = FALSE, xlab = "", ylab = "", asp = "", legend = FALSE)
})
par(op)
```
Now, plot the same kind of image but zoom in on a region purposefully. The resolution provided has adapted to the context asked for.
```{r}
rtas <- crop(rworld, extent(143.4, 149, -44, -39.1))
plot(rtas, col = grey(seq(0, 1, length.out = 64)), zlim = c(0, 1550))
rbath <- as_raster(rtas)
rbath[rbath > 0] <- NA
contour(rbath, add = TRUE, levels = quantile(rbath, prob = seq(0, 1, length.out = 8)))
title("Tasmania topography + bathymetric contours, from Gebco 2019", cex.main = 0.85)
```
This is not just to plot big rasters, it's potentially useful for streaming gridded data to a device that is resizing the view port interactively. We also use it to explore a data set for useability and general coverage, and designing sensible resampling workflows for very large data models.
```{r mars}
## PLEASE take care with this url, it is a 11Gb GeoTIFF
## https://astrogeology.usgs.gov/search/map/Mars/Topography/HRSC_MOLA_Blend/Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2
dangerurl <- "https://planetarymaps.usgs.gov/mosaic/Mars/HRSC_MOLA_Blend/Mars_HRSC_MOLA_BlendDEM_Global_200mp_v2.tif"
## note prefix with GDAL's URL indirection
mars <- lazyraster(file.path("/vsicurl", dangerurl))
mars
plot(mars, col = grey(seq(0, 1, length = 256)))
as_raster(mars)
plot(crop(mars, raster::extent(-100, -30, -18, 5)),
col = grey(seq(0, 1, length = 256)))
```
We've successfully used it to plot a DEM of Australia from a 67 Gb ESRI binary grid (ADF) supplied by GeoScience Australia in *a few minutes* (the grid is more than 1e5 pixels each dimension, so I'm not having this document build do the job but here's a figure I prepared earlier).
```{r ga-srtm, eval=FALSE}
gafile <- raadtools::topofile("ga_srtm")
ga <- lazyraster(gafile)
ga
plot(ga)
```
![GeoScience SRTM][im]
```{r tms, eval=TRUE}
# Make a TMS source and read at the desired resolution.
# More here: http://rpubs.com/cyclemumner/358029
#
library(lazyraster)
gibs_xml <- function(date, level = 3) {
date <- format(date, "%Y-%m-%d")
sprintf('<GDAL_WMS>
<Service name="TMS">
<ServerUrl>
https://gibs.earthdata.nasa.gov/wmts/epsg3413/best/MODIS_Terra_CorrectedReflectance_TrueColor/default/%s/250m/${z}/${y}/${x}.jpg</ServerUrl>
</Service>
<DataWindow>
<UpperLeftX>-4194304</UpperLeftX>
<UpperLeftY>4194304</UpperLeftY>
<LowerRightX>4194304</LowerRightX>
<LowerRightY>-4194304</LowerRightY>
<TileLevel>%i</TileLevel>
<TileCountX>2</TileCountX>
<TileCountY>2</TileCountY>
<YOrigin>top</YOrigin>
</DataWindow>
<Projection>EPSG:3413</Projection>
<BlockSizeX>512</BlockSizeX>
<BlockSizeY>512</BlockSizeY>
<BandsCount>3</BandsCount>
</GDAL_WMS>
', date, level)
}
s <- gibs_xml(Sys.Date()-10)
gibs <- lazyraster(s)
r2 <- as_raster(gibs, dim = c(150, 150))
library(raster)
## run the same simplification but with a different resampling
## method
plot(as_raster(gibs, dim = c(150, 150), resample = "CubicSpline"), col = head(palr::sstPal(64), 45))
## run with a different extent
e <- extent(-806000, 1080000, -3200000, -500000)
plot(as_raster(crop(gibs, e), dim = c(150, 150), resample = "CubicSpline"), col = head(palr::sstPal(64), 45))
```
Please note that this project is released with a [Contributor Code of Conduct](https://github.com/hypertidy/lazyraster/blob/master/CODE_OF_CONDUCT.md).
By participating in this project you agree to abide by its terms.
[im]: inst/images/ga_srtm.png "GeoScience SRTM"