R Cloud hackdays – enhancing Cloud workflows & R support the Cookbook

[jules32]

Following a conversation between @BriannaLind and @cboettig at the ESA conference, we are going to have breakout rooms in Thursday coworking sessions focused on enhancing R support in the Cookbook. Carl Boettiger (@cboettig) is an amazing person in many ways; he's a co-founder of rOpenSci, faculty of Environmental Science, Policy & Management at Berkeley, and a contributor to so, so many open source efforts in at the interface of environment, R, cloud, data, python.

Some starting links Carl has shared:

• https://schmidtdse.github.io/biodiversity-catalog/ - At the UC Berkeley DSE center, we've just been using STAC to create our own little catalog of data products we are using in our efforts, along with some documentation on how to access (and add) data without downloading
  • see also it connected with Radiant Earth: https://radiantearth.github.io/stac-browser/#/external/raw.githubusercontent.com/schmidtDSE/biodiversity-catalog/main/stac/v1/catalog.json?.language=en
• https://source.coop/ - Radiant Earth just released this, like GitHub for data. Carl has started an entry for FishBase: https://beta.source.coop/repositories/cboettig/fishbase/description
• https://github.com/cboettig/duckdbfs - duckdb is powerful with parquet cloud formats (still tricky with netdcf and hdf5)
• https://planetarycomputer.microsoft.com/catalog
• https://schmidtdse.github.io/biodiversity-catalog/examples/cloud-basics.html

[BriannaLind]

August 31, 2023 - R co-working

@BriannaLind screenshared and tried to access NASA data (that has major credentialling associated with it) from R Studio in openscapes2i2c. @cboettig talked through. Underlying concept of using gdal to do heavy lifting and avoiding complications from authentication when possible. @jules32 @mjami00 @stefaniebutland @betolink had awesome contributions throughout.

Annotating Bri’s code

1. Make keys.json

   • Step 1.1: Search for a collection that is available in the cloud in Earthdata
     

   • Step 1.2: Choose a collection
     

   • Step 1.3: Scroll down and navigate to "Get AWS S3 Credentials". You will be prompted to login in at this time.
     
     *if you do not have an Earthdata Login you can make one here

   • Step 1.4: Click the link and copy the credentials.

     • Note the "expiration"; this token will only last 1 hour (the clock is ticking!) and only for LP_CLOUD (can't get data from other NASA sources)
       

   • Step 1.5: (in R Studio) Copy the credentials into a file called keys.json and save.

2. Set R Studio Environment to recognize information in keys.json

   • Step 2.1: assign a name to the file keys.json
     keys <- jsonlite::read_json("keys.json")

   • Step 2.2: use keys to set up the enviroment
     Sys.setenv("AWS_ACCESS_KEY_ID" = keys$accessKeyId) Sys.setenv("AWS_SECRET_ACCESS_KEY" = keys$secretAccessKey) Sys.setenv("AWS_SESSION_TOKEN" = keys$sessionToken)

3. Load the stars package
   library(stars)

4. Copy s3 url from Earthdata

   • Step 4.1: Click the download icon assocaited with a single granuale
     

   • Step 4.2 Click the .h5 file name at the bottom of the pop up window
     
     This will copy the file path to you clipboard
     

5. Paste the s3 url into your scipt into R.
   s3://lp-prod-protected/VNP09A1.002/VNP09A1.A2023185.h31v13.002.2023218221648/VNP09A1.A2023185.h31v13.002.2023218221648.h5

6. Replace "s3://" with "/vsis3/" so gdal can read it with a virtual file system (using the environment set up in step 2.2) and assign to an object called "url"
   url <- '/vsis3/lp-prod-protected/VNP09A1.002/VNP09A1.A2023185.h31v13.002.2023218221648/VNP09A1.A2023185.h31v13.002.2023218221648.h5'

7. Read in the data
   read_stars(url)

8. Now you are ready to roll!

Take Aways

• this is possible!...but there IS a need for earthaccess analog in R!
• When introducing new learners, authentification issues are a massive bummer and burden -> we need to obscure this as much as possible.

[jules32]

Sept 14 brief notes

Carl has created: https://boettiger-lab.github.io/earthdatalogin 😍

• Auth can get trickier soon - we spent an hour troubleshooting but ultimately got it in using one of Carl’s RStudio Cloud instances
• The user still needs to type in a password
• Carl updated the package with an example for now; next steps are error handling
• To update GDAL: see the sf website documentation for all installations
• Can update the rocker one

[jules32]

Sept 28 updates

TL;DR and next steps

TL; DR means Too Long, Didn't Read

@cboettig has been working on https://boettiger-lab.github.io/earthdatalogin 😍 - check out the Articles (screenshots below). Today, he screenshared workin in GitHub Codespaces to launch small instance in cloud. This is where he's been shaking out some bugs with the R environment for working in the cloud and so we can then develop R workflows for Cookbook that parallel the experience for Python workflows.

Carl: "the goal is for my students not to be stressing/thinking about data formats, I want them to think about raster data, which then we can use tools/workflows like rast in R and xarray in python”. 😍

Next steps: Calling all testers!
Please test this page: https://boettiger-lab.github.io/earthdatalogin/index.html
Carl has updated the authentication default so and gdal issues that we ran into last time. Does it work for you, what challenges remain? Please respond here or in https://github.com/boettiger-lab/earthdatalogin/issues

earthdatalogin - more details

https://boettiger-lab.github.io/earthdatalogin/index.html - install this R package by reading this home page and following along.

Check out the Articles

These are small narrated examples!

One small peek at data formats article

https://boettiger-lab.github.io/earthdatalogin/articles/data-formats.html

This vignette illustrates cloud-native access of across a range of common formats, including netcdf, Zarr, hdf5, hdf4, and COGs.

Small peek at legacy formats

https://boettiger-lab.github.io/earthdatalogin/articles/legacy-formats.html

Reusable code - this is a gif showing 12 months of data; the gif is much more powerful that my screenshot here.

Example notebooks - more details

This is the code that we walked through in Codespaces, developed by Carl:

https://github.com/espm-157/nasa-topst-env-justice

Abstract:

1968: In the same year as NASA’s first manned mission to the moon, racially segregated housing became illegal with the Fair Housing Act. The law would now ban practices known as redlining – in which the federal government’s Home Owners Loan Corporation dividing cities into areas graded ‘minimal risk’ to ‘hazardous’ for home loans based largely on racial and ethnic make-up (Nelson et al 2022). But the consequences of such practices are not easily reversed. More than 50 years later, scientists can still see the pattern of the inequalities etched onto those maps even from space (Schell et al, 2020). This educational module will seek to introduce students to the open source platforms and tools used to manipulate and analyze NASA’s open earth observation imagery through the lens of examining the environmental legacy of redlining practices in major urban areas. We will introduce key concepts of cloud-native geospatial workflows including STAC, COG, and GDAL Virtual Filesystem to allow students to leverage analysis against terrabytes of NASA data. The module design is also deeply rooted in pedagogical practices established by education research literature to increase engagement and reach an inclusive audience (Handelsman et al. 2004; Allen & Tanner 2005). The module will be provided in both R and Python and be made available in both English and Spanish translations.

Codespaces - more details

We focused on the Codespaces setup and how it's a friendly way for new learners (Carl uses to teach Berkeley students; NASA Mentors could use for workshops). >>> From GitHub, you should be able to set this up and click “open in Openscapes 2i2c”

GH talks to Azure, it give a secure instance that’s controlled through your GH authentication. We all have our own virtual machine (VM) if we each click on "Codespaces" beneath the green Code button

Can run ipynb code with an extension!
This is a good setup to teach; no GitHub setup, no PAT tokens on morning 1. Learners-first, early wins

At bottom here in Ports > RStudio, you can launch RSTudio in the Cloud if you prefer (but would need to figure out GitHub tab and project). Things are ready to rock-and-roll in VSCode, needs more setup for RStudio

This is where you could set up to be able to click on "Open in 2i2c Openscapes"

[yuvipanda]

BTW, https://github.com/microsoft/vscode-jupyter-hub seems to be an extension microsoft is working on to allow people to use vscode from github codespaces to connect to a hub so you can get the environment and home directories in a hub while still running from inside vscode. Not fully production ready yet but getting there.

[cboettig]

Yes, connecting a notebook to an existing jupyter server totally works! Not sure how the auth step would work with openscapes (we need to know the token), but I've done this in a classroom setting to let students access my little GPU workstation from their Codespaces.

One thing that is a little weird though is how paths work in this setup! Say you open a ipynb notebook in VSCode and select the local python kernel. If you're reading in any extra custom modules or data from the posix file path, it works as expected. But then you switch the 'kernel' to use the remote Jupyter -- now 'local' paths in fact refer to paths on that jupyter server, not the paths you see in your local vscode. In a way this feels very like back-to-google-colab, where the notebook needs to be self-contained and not operate with relative paths in a repo.

Separate aside -- but this set-up adds to the pedagogical challenge of students having any mental model of "where" code execution occurs or where data lives 😂

[jules32]

Nov 9, 2023 update from R/Codespaces breakout room

From @eeholmes

Today @cboettig and I started by discussing this blog https://openscapes.org/blog/2023-11-07-coiled-openscapes/ that compares cost and speed for downloading data to a laptop versus working in the cloud. Carl had written up a variant of the example in the blog in R and walked me through that code. A key difference is that his code does not download all the data but instead only downloads the needed cropped area. Pre-cropping before downloading makes the laptop speed 20 minutes instead of 383 minutes: https://github.com/boettiger-lab/earthdatalogin/blob/edl_search/inst/examples/sst.R
Then he showed me how to extract values at points in a cube with gdalcube:

library(sf)
pts <- sf::st_as_sf(data.frame(long = -93, lat = 41),
                    coords=c("long", "lat"),
                    crs = st_crs(4326))

tmp <- cube |> gdalcubes::extract_geom(pts)

Then he showed me the Python files at https://planetarycomputer.microsoft.com/dataset/modis-15A3H-061#Example-Notebook and we waxed about how it would be a huge boon to the R community if there were R examples of the Python examples. We ended up here because one of Carl's examples involves some legacy file formats that does not allow subsetting (you have to download the whole file). When Planetary Computer re-hosted the data, they converted to a modern file format that is a) faster and b) allows subsetting. Note 'subsetting' is not the right word, but I can't remember the word Carl used.

Then he quickly showed me an example of duckdb with gbif: https://github.com/cboettig/example-notebooks/blob/main/gbif-notebooks/gbif-duckdb-aws.ipynb

We joked about how this formatting is so very R like. R programmers have a hard time living without their "pipes" and here Carl has re-created something that feels pipelike in Python. This is very amusing if you are an R programmer.

tbl = (gbif
       .filter(gbif.phylum == "Chordata")
       .mutate(latitude = gbif.decimallatitude.round(0),
               longitude = gbif.decimallongitude.round(0))
       .select(["class", "genus", "longitude", "latitude"])
       .distinct()
       .group_by(["class","longitude", "latitude"])
       .size()
      )

Last time we had talked about the devcontainer files that Carl created here https://github.com/cboettig/nasa-topst-env-justice. Since then he made some tweaks that improved the user experience and I showed off the codespaces at the NMFS R User Group meeting yesterday. Today we ran into one of the issues that I worried about which is 'what to do if the devcontainer file is updated?'. I could have updated my fork of nasa-topst-env-justice, but I had run into a new problem. For the first time, I used up all my free Codespace minutes. Meh 😞 so Carl instead ran the code today.

From @cboettig

Credit to @stefaniebutland who shared the cool blog post yesterday on https://openscapes.org/blog/2023-11-07-coiled-openscapes/ . We were looking at how to do that in R, on a local machine using range requests so that it would still be fast but also cost-free to user.

[yuvipanda]

THIS IS AMAZING WORK, @cboettig! I'm always interested in looking at ways to provide alternatives to proprietary services like coiled that also are easier for the end users. Thank you for working on this.

Separately, I'd love for us to have more information / documentation about what 'data in the cloud' actually means, as it can cover a wide array of things, all with different properties. In this case, I think the important factor is being able to access just the parts of data you want, covered by standard HTTP Range requests in one way or another. But I think it's too muddled right now for regular users to have an easy grasp, making it hard to decide when what tools are useful. How do we go about building this suite of educational materials?

[jules32]

From @cboettig 🎉 👍

Just wanted to share that a first release of the R package, earthdatalogin , is now on CRAN: https://cran.r-project.org/web/packages/earthdatalogin/ . Includes 5 vignettes covering various use cases.
It doesn't do all the fancy stuff that earthaccess can do in python, but should streamline access to EarthData over HTTPS with no changes to the R user's workflow (in most cases you just set_edl_token() at the top of your code and can continue to use terra, stars, sf on https addresses as you would normally). Big thanks to @betolink, @BriannaLind , and many others here for help in initial ideas and testing; and bug reports etc welcome (suggestions/improvements to vignette docs definitely welcome).

[jules32]

In Coworking today - Carl and Eli:
Create docker image to help Cookbook align/test with 2i2c Hub
#301