Netrc

README.md

@@ -9,8 +9,10 @@ Content include in this repository are listed below.
 |----|-----|----|
 | **[Data_Discovery_CMR_API_Request.ipynb](/python/tutorials/Data_Discovery_CMR_API_Request.ipynb)** | Jupyter Notebook | Demonstrates how to search for Earthdata data collections and granules using CMR API and Request Python package|
 | **[Data_Discovery_CMR_API_Bulk_Query.ipynb](/python/tutorials/Data_Discovery_CMR_API_Bulk_Query.ipynb)** | Jupyter Notebook | Demonstrates how to search and extract data URLs for an entire collection using Python's `asyncio` package|
+| **[create_netrc_file.md](/guides/create_netrc_file.md)** | Markdown | Demonstrates how to create a .netrc file in your home directory |
 | **[bulk_download_using_curl.md](/guides/bulk_download_using_curl.md)** | Markdown | Demonstrates how to bulk download LP DAAC data using Curl from command line |
 | **[bulk_download_using_wget.md](/guides/bulk_download_using_wget.md)** | Markdown | Demonstrates how to bulk download LP DAAC data using Wget from command line |
+| **[DAACDataDownload.py](/python/scripts/daac_data_download_python)** | Python script | Demonstrates how to download LP DAAC data using a command line executable script |
 
 The other guides, tutorials, how-tos and scripts can be accessed in our mission specific repositories. 
 

guides/bulk_download_using_curl.md

@@ -11,40 +11,7 @@ This guide shows how to bulk download [LP DAAC](https://lpdaac.usgs.gov/) data u
 Save download links for your data as a text file using [Nasa Earthdata Search](https://search.earthdata.nasa.gov/search) or [Common Metadata Repository (CMR)](https://www.earthdata.nasa.gov/eosdis/science-system-description/eosdis-components/cmr) API. Follow the steps in the [Earthdata Search guide](https://github.com/nasa/EMIT-Data-Resources/blob/main/guides/Getting_EMIT_Data_using_EarthData_Search.md) to find your data and save the download links. If you prefer to use an API to find your data and save the download links, a tutorial on how to use the CMR API can be found [here](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/python/tutorials/Data_Discovery_CMR_API_Request.ipynb). 
 
 ## Step 2: Set up a .netrc file for Authentication
-Set up a .netrc file in your home directory.
-
-
-- ### Manual set up
-  - Download the [.netrc template file](https://github.com/nasa/LPDAAC-Data-Resources/tree/main/data/.netrc) and save it in your home directory.
-    - Open the .netrc file in a text editor and replace `<USERNAME>` with your NASA Earthdata Login username and `<PASSWORD>` with your NASA Earthdata Login password.
-
-- ### Create .netrc file from the Command Line
-  - Enter the following in Terminal:
-    - #### Windows
-      To Create a .netrc file, enter the following in the command line. 
-      ```
-      NUL >> %userprofile%\.netrc | echo machine urs.earthdata.nasa.gov >> %userprofile%\.netrc
-      ```
-      To insert your NASA Earthdata login username and password into the file, enter the following in the Command Prompt and replace your username and password.
-
-      ```
-      echo login Insert_Your_Username >> %userprofile%\.netrc | echo password Insert_Your_Password >> %userprofile%\.netrc
-      ```
-    - #### MacOS:
-
-      To Create a .netrc file, enter the following in the command line. 
-      ```
-      touch ~/.netrc | chmod og-rw ~/.netrc | echo machine urs.earthdata.nasa.gov >> ~/.netrc
-      ```
-      To insert your NASA Earthdata login username and password into the file, enter the following in the Command Prompt and replace your username and password.
-
-      ```
-      echo login Insert_Your_Username >> ~/.netrc | echo password Insert_Your_Password >> ~/.netrc
-      ```
-
-- ### Programmatically:
-  - Run [Authentication for NASA Earthdata notebook](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/python/how-tos/Earthdata_Authentication__Create_netrc_file.ipynb) to create _.netrc_ file. 
-  - Alternatively, you can run the [EarthdataLoginSetup script](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/python/scripts/EarthdataLoginSetup.py) in a Python interpreter or from the command line.
+Follow the instruction on how to create a `.netrc` file [here](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/guides/create_netrc_file.md) to set up the file using your Earthdata Login Credentials. 
 
 ## Step 3: Download LP DAAC Data
 You should now be able to run the command to download data directly from the LP DAAC. 

guides/create_netrc_file.md

@@ -0,0 +1,54 @@
+# How to Set up a `.netrc` file for Authentication
+There are several ways to set up a `.netrc` file in your home directory.
+
+- ### Manual set up
+    Download the [.netrc template file](https://github.com/nasa/LPDAAC-Data-Resources/tree/main/data/.netrc) and save it in your *home/user/* directory, where *user* is your personal user directory. For example: `C:\Users\user\.netrc` or `home/user/.netrc.`  
+  - Open the `.netrc` file in a text editor and replace <USERNAME> with your NASA Earthdata Login username and <PASSWORD> with your NASA Earthdata Login password.
+
+  After editing, the file should look something like this:
+
+  ![Example .netrc 1](../img/example_netrc1.png)
+
+  or you can also have everything on a single line separated by spaces, like:
+
+  ![example .netrc 2](../img/example_netrc2.png)
+
+
+- ### Create .netrc file from the Command Line
+
+  **For Linux/MacOS:**
+
+  To Create a .netrc file, enter the following in the command line, replacing <USERNAME> and <PASSWORD> with your NASA Earthdata username and password. This will create a file in your home directory or append your NASA credentials to an existing file.
+
+  ```bash
+  echo "machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD>" >>~/.netrc
+  ```
+
+  **For Windows:**
+
+  To Create a .netrc file, enter the following in the command line, replacing <USERNAME> and <PASSWORD> with your NASA Earthdata username and password. This will create a file in your home directory or append your NASA credentials to an existing file.
+
+  ```cmd
+  echo machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD> >> %userprofile%\.netrc
+  ```
+
+  You can verify that the file is correct by opening with a text editor. It should look like an example in one of the figures above.
+
+- ### Programmatically:
+    - #### Python
+        The [`earthaccess` Python library](https://earthaccess.readthedocs.io/en/latest/) provides a convenient way to authenticate, search, and access NASA Earth science data using Python. It can be used to manage Earthdata Login and generate access tokens.
+        Run the code below to create a `.netrc` file in your home directory. You will be prompted to enter your Earthdata Login credentials.
+        ```python
+        import earthaccess
+        earthaccess.login(persist=True)
+        ``` 
+        Instruction on how to install the `earthaccess` library is provided [here](https://earthaccess.readthedocs.io/en/latest/quick-start/).
+
+    - #### R 
+        The [`earthdatalogin` R Package](https://cran.r-project.org/web/packages/earthdatalogin/index.html) provides convenient authentication and access to NASA 'EarthData' products using R. `edl_netrc` function will create a `.netrc` file using your Earthdata Login (EDL) credentials. 
+        ```r
+        library(earthdatalogin)
+        edl_netrc(username = Insert_Your_Username, password = Insert_Your_Password, netrc_path = '~/.netrc')
+        ```
+        More details can be found [here](https://github.com/boettiger-lab/earthdatalogin/blob/main/R/edl_netrc.R).
+

python/scripts/daac_data_download_python/DAACDataDownload.py

@@ -0,0 +1,80 @@
+"""
+---------------------------------------------------------------------------------------------------
+ How to Access the LP DAAC Data with Python
+ The following Python code example demonstrates how to configure a connection to download LP DAAC data 
+ from Data Pool or NASA Earthdata Cloud. 
+ 'earthaccess' package handles the NASA EarthData Login (EDL). 
+ Last Updated: 09/06/2024
+---------------------------------------------------------------------------------------------------
+"""
+# Load necessary packages into Python
+from subprocess import Popen
+from colorama import Fore, Back, Style
+import earthaccess 
+import argparse
+import os
+
+# ----------------------------------USER-DEFINED VARIABLES--------------------------------------- #
+# Set up command line arguments
+parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+parser.add_argument('-dir', '--directory', required=True, help='Specify directory to save files to')
+parser.add_argument('-f', '--files', required=True, help='A single granule URL, or the location of csv or textfile containing granule URLs')
+args = parser.parse_args()
+
+saveDir = args.directory  # Set local directory to download to
+files = args.files        # Define file(s) to download from the LP DAAC Data Pool
+
+# ---------------------------------SET UP WORKSPACE---------------------------------------------- #
+# Create a list of files to download based on input type of files above
+if files.endswith('.txt') or files.endswith('.csv'):
+    with open(files, 'r') as f:
+        fileList = f.read().splitlines() # If input is text/csv file with file URLs
+
+    # fileList = open(files, 'r').readlines().splitlines()  
+elif isinstance(files, str):
+    fileList = [files]                       # If input is a single file
+# Check if the directory exists
+if not os.path.isdir(saveDir): 
+    os.makedirs(saveDir)
+
+# --------------------------------AUTHENTICATION CONFIGURATION----------------------------------- #
+# AUthenticate using earthaccess.login function. 
+
+earthaccess.login(strategy = 'netrc', persist = True) 
+
+print(Fore.RED + Back.GREEN + 'Please note: if you just entered your Earthdata Login info, your username and password are now stored in a .netrc file located at the Home directory on this system.')
+print(Style.RESET_ALL)
+# -----------------------------------------DOWNLOAD FILE(S)-------------------------------------- #
+# Loop through and download all files to the directory 8 files at a time, and keeping same filenames
+num = int(len(fileList)/8)
+
+for n in list(range(num+1)):
+    try:
+        subList = fileList[8*(n):8*(n+1)]
+    except: 
+        try: 
+            subList = fileList[8*(n):len(fileList)]
+        except: 
+            subList = fileList[8*(n)]
+    # Remove the files that are downloaded already
+    subList_filter = []
+    for f in subList:
+        if not os.path.isfile(os.path.join(saveDir, f.rsplit('/')[-1])):
+            subList_filter.append(f)
+        else:
+            print(f'{f.rsplit("/")[-1]} already exists in the {saveDir}.')
+
+    if len(subList_filter) != 0:        
+        attempts = 0
+        success = False
+        while (attempts < 3 and success == False):
+            try:
+                # Download files 
+                earthaccess.download(subList_filter, saveDir, threads=8)
+                success = True
+            except:
+                attempts +=1
+                print(Fore.RED + Back.GREEN + f'There is an issue with downloading files in {subList_filter}')
+                print(Style.RESET_ALL)
+
+