feat: New example transferring data to ORNL DAAC

access_testing/edl-token-example.ipynb

@@ -0,0 +1,224 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "018b6eca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "from datetime import datetime\n",
+    "from datetime import timedelta\n",
+    "import getpass\n",
+    "import json\n",
+    "import os\n",
+    "import requests\n",
+    "from requests.auth import HTTPBasicAuth\n",
+    "import shutil"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8b50a274",
+   "metadata": {},
+   "source": [
+    "# Programmatic Access to NASA Data via Earthdata Login Tokens\n",
+    "\n",
+    "This notebook details how to create and use EDL tokens to request data from NASA.\n",
+    "\n",
+    "There are a few ways to handle access from a user perspective.\n",
+    "\n",
+    "1. Users could have their own EDL username and password, and use those credentials to generate tokens or use them directly in requests for data.\n",
+    "2. The system stores a system username and password, such as username `esa_maap_user` along with an associated password. This username and password are stored in a secret store which only the algorithm development and async worker systems have access too. For example, the system that launches new Jupyter Lab instances, such as a Kubernetes cluster. These credentials are used to fetch or generate a token which is then set as an environment variable in Jupyter Lab or worker instances for making requests for data.\n",
+    "\n",
+    "We are assuming option 2 is more desirable for the ESA MAAP system, so ESA MAAP users and workers don't have to create a new username and login with Earthdata Login in addition to their ESA MAAP login. Below is detailed an approach for generating fresh-enough tokens for users as the system launches their environments.\n",
+    "\n",
+    "# Creating new tokens\n",
+    "\n",
+    "Given a lifetime of an active Jupyter Lab system or worker of `active_session_time`, we want to ensure the token deployed to an active session will remain valid for the extent of the session. EDL tokens are valid for 90 days and there can be 2 active tokens, (see https://urs.earthdata.nasa.gov/documentation/for_users/user_token), when a system (development environment or worker) boots up:\n",
+    "\n",
+    "1. If there are no active tokens, generate a token.\n",
+    "2. If there are active tokens and one of those active tokens has an expiration that is greater than the current datetime + active session time, that token can be used since it won't expire in the lifetime of the active session.\n",
+    "3. If there is only 1 active token but it will expire within the time of an active session, generate a new token. This token should be used by any new development environments or workers that boot up afterwards (until that token is also about to expire).\n",
+    "\n",
+    "# Considerations:\n",
+    "\n",
+    "* The below are taken from https://urs.earthdata.nasa.gov/users/aimeeb/user_tokens\n",
+    "    * You can generate a bearer token for federated token access sharing. Not all EDL applications support Federated token access sharing.\n",
+    "    * You may have up to 2 active tokens at a time.\n",
+    "    * The token can then be passed into an application by using an Authorization: Bearer header\n",
+    "    * The token will only authorize for applications that are EDL compliant and do not have unapproved EULAs\n",
+    "* So, not all DAACs support token-based access - For example, this access was tested and working for GEDI02_A.002 but not SENTINEL-1A_SLC"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b4f839d6-ee75-4d07-8ac0-e4527b0326b9",
+   "metadata": {},
+   "source": [
+    "# Code to generate a token when the user logs in\n",
+    "\n",
+    "The following steps should happen server-side when a new Jupyter Lab session is starting.\n",
+    "\n",
+    "These would be stored in a secret store, having inputs here is just for demonstration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c71be34a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "EDL Username aimeeb\n",
+      "EDL Password ··················\n"
+     ]
+    }
+   ],
+   "source": [
+    "# These would be stored in a secret store, having inputs here is just for demonstration.\n",
+    "username = input('EDL Username')\n",
+    "password = getpass.getpass('EDL Password')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "313f56fe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "active_session_days = 1 # for example\n",
+    "\n",
+    "def find_or_get_token():\n",
+    "    urs_endoint = 'https://urs.earthdata.nasa.gov/api/users/token'\n",
+    "    # first try and retrieve an active token\n",
+    "    token_response = requests.get(f\"{urs_endoint}s\", auth=HTTPBasicAuth(username, password))\n",
+    "    error_message = \"Attempting to get a existing tokens returned an error: {status_code} - {content}\"\n",
+    "    if token_response.status_code != 200:\n",
+    "        return error_message.format(status_code=token_response.status_code, content=token_response.content)\n",
+    "    else:\n",
+    "        # Tokens were found\n",
+    "        tokens = json.loads(token_response.content)\n",
+    "        for token in tokens:\n",
+    "            x_date = datetime.now()\n",
+    "            expiration_datetime = datetime.strptime(token['expiration_date'], '%m/%d/%Y')\n",
+    "            # Token expiration is greater than now + active session time, so it shouldn't expire during the lifetime of an active session\n",
+    "            if expiration_datetime > (x_date + timedelta(days=active_session_days)):\n",
+    "                return token['access_token']\n",
+    "\n",
+    "    # no active tokens were found or active tokens will soon expire, generate a new token      \n",
+    "    error_message = \"Attempting to create a new token returned an error: {status_code} - {content}\"    \n",
+    "    token_response = requests.post(urs_endoint, auth=HTTPBasicAuth(username, password))\n",
+    "    if token_response.status_code != 200:\n",
+    "        return error_message.format(status_code=token_response.status_code, content=token_response.content)\n",
+    "    return json.loads(token_response.content)['access_token']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f6f8524f-3121-4aa6-91ef-292a874f7677",
+   "metadata": {},
+   "source": [
+    "# Set the environment for new Jupyter Lab sessions\n",
+    "\n",
+    "The following `TOKEN` variable should be set in the users environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "d62360f2-c055-4209-be3b-1ccb8299e4e7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"TOKEN\"] = find_or_get_token()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f87893a7-4a1b-49f8-a061-b6bd970f340b",
+   "metadata": {},
+   "source": [
+    "# Provide a function for accessing data using the token\n",
+    "\n",
+    "For the NASA MAAP, this type of function is part of the `maap-py` library."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "8f53f672",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Simplified version of https://github.com/MAAP-Project/maap-py/blob/master/maap/Result.py#L89-L111\n",
+    "def get_data(url, destfile, destpath = '.'):\n",
+    "    api_header = {\n",
+    "        'Authorization': f\"Bearer {os.environ['TOKEN']}\"\n",
+    "    }\n",
+    "    r = requests.get(\n",
+    "        url=url,\n",
+    "        headers=api_header,\n",
+    "        stream=True\n",
+    "    )\n",
+    "\n",
+    "    if r.status_code != 200:\n",
+    "        raise ValueError('Bad search response for url {}: {}'.format(url, r.text))\n",
+    "    print(r.status_code)\n",
+    "\n",
+    "    r.raw.decode_content = True\n",
+    "\n",
+    "    with open(destpath + \"/\" + destfile, 'wb') as f:\n",
+    "        shutil.copyfileobj(r.raw, f)\n",
+    "\n",
+    "    return destpath + '/' + destfile"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "91bb09c4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "200\n"
+     ]
+    }
+   ],
+   "source": [
+    "url = 'https://e4ftl01.cr.usgs.gov//GEDI_L1_L2/GEDI/GEDI02_A.002/2022.08.31/GEDI02_A_2022243234134_O21062_01_T10906_02_003_02_V002.h5'\n",
+    "destfile = url.split('/')[-1]\n",
+    "get_data(url, destfile)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}