cloud_lesson.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c0df9c8-6056-4f1d-b909-b4abc40193fa",
   "metadata": {},
   "source": [
    "# Science on the Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "704ad45e-82e5-4110-8a40-6bec5497acfa",
   "metadata": {},
   "source": [
    "## Context\n",
    "\n",
    "Today we are going to be exploring the cloud platform and digging into some of the details of what it means to do science on the cloud. We aren't actually going too write much code today, but we are going to look at a few pieces of how computers work and explore some of the ways that the cloud facilitates collaboration. It's going to be a bit of a hodge-podge of skills, but together they give you some of the computer science building blocks to navigate the cloud interface."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "078beca5-da3f-441d-8c0c-e0ea7437cd85",
   "metadata": {},
   "source": [
    "## Computer Images\n",
    "\n",
    "When you first started the hub you had a lot of options for what setup to choose. Choosing your setup options is a bit of a \"build your own computer\" moment every time your start the hub. You can choose what type of computer you want each time you start the hub! It's a bit like building a custom burrito at chipotle.\n",
    "\n",
    "Every time you login to the cloud a computer gets turned on somewhere for you to use. In cloud lingo this computer is referred to as a **node**.\n",
    "\n",
    "> When we talk about accessing a \"computer\" in the cloud we usually call it a **node** or a **compute node**\n",
    "\n",
    "The two options you are choosing between on that first page are **image** and **node** size. The **image** defines the types of softwares that are availble when you start the hub. The options you'll see are Python, R, and Matlab. The **node** size refers to how powerful of a computer you are going to start. You see options ranging from about 2 GB to 32 GB. \n",
    "\n",
    "Please only ever used the smallest node size that you need. It helps saves energy and money!\n",
    "\n",
    "![chiptole](https://www.foodandwine.com/thmb/oCl8jGErMHGWoXWBUC-jZhvKTSA=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc()/chipotle-food-hacks-overhead-FT-BLOG0717-4cb0442dcddf4bc9b1d774374d00be8f.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f00a948-d508-4990-a54d-6fefbf7224da",
   "metadata": {},
   "source": [
    ":::{admonition} Vocabulary\n",
    "\n",
    "**Image**: An image contains the software setup of a computer. It knows the operating system, the installed softwares, and the programming packages or libraries availble.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f32356d-576c-4de8-a02d-52312b258ac1",
   "metadata": {},
   "source": [
    ":::{admonition} Cloud benefit: Scalability\n",
    "\n",
    "The ability to choose the amount of computing power you need on demand and grow what you are using as you need more. This is **scalability**.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974d45d7-1ce6-40de-8b7e-e39e600505cf",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Tour of the interface\n",
    "\n",
    "First, let's tour the interface of JupyterHub! There are lots of exciting tools. We will look at the left and right side tab tools. On the left side you'll find:\n",
    "- file explorer\n",
    "- kernel manager\n",
    "- dask interface\n",
    "- git interface\n",
    "- markdown table of contents\n",
    "- extension manager\n",
    "- jupyter AI interface\n",
    "\n",
    "On the rigth side you'll find:\n",
    "\n",
    "- property inspector\n",
    "- kernal usage\n",
    "- debugger"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "421ad11d-2945-4a60-a95a-50a02a3e9a6f",
   "metadata": {},
   "source": [
    "## Private and Shared Files\n",
    "\n",
    "One of the major benefits of the cloud is the way you can access shared spaces with your collaborators. In cryocloud there are both public and private spaces. The three public spaces on Cryocloud are:\n",
    "- `shared`\n",
    "- `shared-public`\n",
    "- `shared-readwrite`\n",
    "\n",
    "Anything in these folders is visible to everyone on Cryocloud. Anything any files you create outside those folders are visible only to you.\n",
    "\n",
    "If you would like to distribute data to your group, feel free to use the `NASA-SARP` folder of `shared-public`. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e5d436-3269-4afb-a805-6c4f5958bfd1",
   "metadata": {},
   "source": [
    ":::{warning}\n",
    "\n",
    "Please do not delete anything you did not create in this folder, and don't add or remove anything from any of the other folders in `shared-public`. We want to be good stewards of the space!\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e721837c-7146-4472-8e67-d2a7ced7df3e",
   "metadata": {},
   "source": [
    ":::{admonition} Cloud benefit #2 -- Collaboration\n",
    "\n",
    "Accessing shared resources (folders, data, environment) is a huge leap for collaboration. One way we take advantage of this is through access to shared folders.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7549aa0-8bda-4b9b-8e89-aff4f738a900",
   "metadata": {},
   "source": [
    ":::{tip} Checkin\n",
    "\n",
    "Within your table pick one person to make a folder in the shared NASA-SARP folder. Have each person in the group create a text file in that folder. Type something in your file, close it, then open someone else's to see what they wrote. Delete the file when you are done (but keep the folder)\n",
    "\n",
    "Be careful -- when possible try not to access the same file at the same time. The computer can get confused about which version to save if different edits are being made at the same time.\n",
    "\n",
    ":::{dropdown} Make it harder\n",
    "\n",
    "Instead of a text file, create a csv file. Open the csv file in the CSV\" Viewer. Then right click on the file > open with > Editor and add the following text:\n",
    "```\n",
    "temperature, co2, ch4\n",
    "C, ppm, ppb\n",
    "80, 420, 1852\n",
    "82, 411, 1903\n",
    "```\n",
    "Save the file, close it, then open it again in the CSV Viewer.\n",
    "\n",
    ":::\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d267a15a-38b9-4365-8bd7-8dcbd68891d0",
   "metadata": {},
   "source": [
    "## Terminal\n",
    "\n",
    "Working in the command line is like working on the inside of your computer. There is no visual screen, or graphical user interface (GUI), for you to use, but you are accessing the same files and programs as you can access in your File Explorer.\n",
    "\n",
    "In general, command line commands can have multiple parts:\n",
    "1. Command_name (Ex. `cd`, `ncdump`)\n",
    "2. Flags (Ex. `-r`, `-h`, `--help`)\n",
    "3. Arguments (Ex. `file1.txt`, `outputfile.nc`)\n",
    "\n",
    " \n",
    "### Navigating your File System\n",
    "* `cd` - “change directory” go into a folder or directory\n",
    "* `cd ..` goes back up a level in the folder hierarchy\n",
    "* `ls` - shows the contents of the current directory\n",
    "* `pwd` - shows the absolute path of the current directory\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325d111a-4606-4e32-8795-fea1f4d1bf32",
   "metadata": {},
   "source": [
    "Doing these commands is all fine and good, but it's easy to get lost inside of a computer if you don't have a general idea of how it is setup. Let's start building one of those by drawing a diagram for movement between folders and files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b225c8b4-62ab-4c2f-99ec-53b388b27637",
   "metadata": {},
   "source": [
    "As a note, you can also run these terminal commands from your jupyter notebook by using a code cell that starts with `!`. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "337a71ee-ae07-4a2f-a626-55e65e568cd8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jovyan\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "77011821-3287-4e87-9e97-ca23838d9833",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "allmydata\t    envs\t\t  projects\t\t  Sync\n",
      "cloud_lesson.ipynb  folder_structure.jpg  shared\t\t  test\n",
      "data\t\t    hierarchical_data\t  shared-public\t\t  trainings\n",
      "Desktop\t\t    Lessons\t\t  shared-readwrite\n",
      "emit-utils\t    myfile.csv\t\t  SpectralUnmixing-0.2.3\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72bc111b-d1d1-4a24-81b2-0df541916f9c",
   "metadata": {},
   "source": [
    "Checkin: starting from `/shared-public/NASA-SARP` draw a tree diagram representing the folders / files present."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d46feb19-2d50-43ff-aa4d-d6acccd46e9b",
   "metadata": {},
   "source": [
    ":::{tip} Checkin\n",
    "\n",
    "Starting with the folder `/shared-public/NASA-SARP/` draw a diagram the shows the structure of this folder. Start with the NASA-SARP folder at the top in a box and draw arrows for each of the folders in each sub-folder.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986a50b2-525f-4997-8d4b-522248cd6cee",
   "metadata": {},
   "source": [
    "You likely made a diagram that has a similar shape to this:\n",
    "\n",
    "![tree](https://upload.wikimedia.org/wikipedia/commons/thumb/5/5f/Tree_%28computer_science%29.svg/800px-Tree_%28computer_science%29.svg.png)\n",
    "\n",
    "Notice that each of the folders or files can be traced back to the single starting point. If you draw a diagram like this for your whole file systme this starting point is called the **root** folder and this type of organization in computer science is called a **tree** structure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acb90f07-0eb8-488f-8ba8-e4571deb6fb9",
   "metadata": {},
   "source": [
    ":::{admonition} Vocabulary\n",
    "\n",
    "**Root folder**: The root folder is the folder at the very top of the tree. It is the folder in which every single folder on your computer lives.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a92d4f77-ece4-4b08-87fc-6e88e00f1564",
   "metadata": {},
   "source": [
    "Why go into all this detailed discussion? It's easy to get mentally lost in terminal, but it helps to remember that we are always scooting around on this tree. When we are in terminal we are always _somewhere_ on this tree, and every time we change directories we are just moving to another place on the tree. That's how the inside of a computer is organized! 🌳💻"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fae02cb4-778b-4c26-bc8e-4ad03bf22c1e",
   "metadata": {},
   "source": [
    "Note about how Cryocloud works: When we start up a terminal from Cryocloud, the terminal is always going to initialize itself to whatever folder you have open in your file explorer window. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e6cc7f-6650-4825-8e46-00d373b8b302",
   "metadata": {},
   "source": [
    ":::{tip} Checkin\n",
    "\n",
    "Create a new folder with a name of your choice (ex. lesson notebooks). `wget` a file into the folder.\n",
    "\n",
    "\n",
    ":::{dropdown} Make it harder\n",
    "The command for making a new folder in terminal is `mkdir`. Use `mkdir` to create a new folder instead of using the clickable interface.\n",
    ":::\n",
    "\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393e7bca-a736-4bc8-9cf1-420732941923",
   "metadata": {},
   "source": [
    "### Moving and Copying Files\n",
    "* `mv file1.txt file2.txt` - `mv` -  moves a file from one location to another. Also used for renaming files\n",
    "* `rm file1.txt` - `rm` - permanently deletes a file.\n",
    "* - asterisk is a wildcard so it can be used to remove all the files that match a certain pattern\n",
    "\tEx. rm -Confirm *.txt - removes all text files in a directory\n",
    "\tEx. rm -Confirm m704_2022*.nc - removes all files that start with m704_2022 and end with .nc\n",
    "* `cp file1.txt file1_copy.txt` - `cp` - copy a file to a new location"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b76c512-d892-46bd-a1c5-252af67b86f6",
   "metadata": {},
   "source": [
    "## Virtual Desktop\n",
    "\n",
    "Not all tasks are done best with code - sometimes an interface is better. For that purpose we have the Virtual Desktop. This tool allows us to access _the same files_ from a graphical user interface (GUI) that is much more similar to a laptop.\n",
    "\n",
    "> Demo: viewing WAS data in QGIS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212ea44f-5024-4e61-a46c-2969ac545a35",
   "metadata": {},
   "source": [
    "## Uploading and Downloading Files\n",
    "\n",
    "Something you may want to do very commonly is upload or download data. Let's demo doing that in the cryocloud interface."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5985edd6-d8fe-473f-8e71-197d5bbc277d",
   "metadata": {},
   "source": [
    ":::{tip} Final Checkin\n",
    "\n",
    "Go to earth data search and find one granule of any data product. You can choose any data product (maybe one you're considering using?!). If you are not sure which one to use try MODIS chlorophyll-a. Download it and upload to your home directory.\n",
    "\n",
    ":::{dropdown} Make it harder\n",
    "The code below shows how you would download data directly using a library called `earthaccess`.\n",
    "\n",
    "```\n",
    "import earthaccess\n",
    "\n",
    "earthaccess.login()\n",
    "\n",
    "results = earthaccess.search_data(\n",
    "    short_name='ECO4ESIPTJPL',\n",
    "    bounding_box=(-77, 37, -76, 38),\n",
    "    temporal=(\"2020-02-01\", \"2020-02-10\"),\n",
    "    count=8,\n",
    ")\n",
    "\n",
    "earthaccess.download(\n",
    "    results,\n",
    "    local_path=data_dir,\n",
    ")\n",
    "```\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b40ed23-9504-4b0e-b66a-4aa2d108788e",
   "metadata": {},
   "source": [
    ":::{tip} Final Checkin\n",
    "\n",
    "Take out a piece of paper and pen. Draw a diagram that represents what happens when you logon to cryocloud. pieces you might include are:\n",
    "- 1+ nodes\n",
    "- your computer\n",
    "- the type of image\n",
    "- the size of the node\n",
    "\n",
    ":::{dropdown} Make it harder\n",
    "\n",
    "Add to your diagram the process of uploading a data file to the cloud from your computer.\n",
    "\n",
    "\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "529565b1-c43c-4dcf-bb8a-007ff5e63d45",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}