uavsar_boise_convert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "standard-worse",
   "metadata": {},
   "source": [
    "# UAVSAR Data Access and Conversion\n",
    "\n",
    "```{admonition} Learning Objectives\n",
    "- overview of UAVSAR data\n",
    "- demonstrate how to access and transform data to Geotiffs\n",
    "```\n",
    "\n",
    "There are multiple ways to access UAVSAR data. Also the SQL database.\n",
    "\n",
    "* [Alaska Satellite Facility Vertex Portal](https://search.asf.alaska.edu/#/?dataset=UAVSAR)\n",
    "* [NASA Earthdata Suborbital Search](https://search.earthdata.nasa.gov/portal/suborbital/search?fi=UAVSAR&as[instrument][0]=UAVSAR)\n",
    "* [JPL UAVSAR Data Search](https://uavsar.jpl.nasa.gov/cgi-bin/data.pl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "above-diabetes",
   "metadata": {},
   "source": [
    "```{admonition} InSAR Data Types\n",
    ":class: InSAR Data Types\n",
    "- ANN file (.ann): a text annotation file with metadata\n",
    "- AMP files (.amp1 and .amp2): calibrated multi-looked amplitude products\n",
    "- INT files (.int): interferogram product, complex number format (we won't be using these here)\n",
    "- COR files (.cor): interferometric correlation product, a measure of the noise level of the phase\n",
    "- GRD files (.grd): interferometric products projected to the ground in simple geographic coordinates (latitude, longitude)\n",
    "- HGT file  (.hgt): the DEM that was used in the InSAR processing\n",
    "- KML and KMZ files (.kml or .kmz): format for viewing files in Google Earth (can't be used for analysis)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "prospective-score",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import re\n",
    "import zipfile\n",
    "import getpass\n",
    "from osgeo import gdal \n",
    "import os  # for chdir, getcwd, path.basename, path.exists\n",
    "import pandas as pd # for DatetimeIndex\n",
    "import codecs # for text parsing code\n",
    "import netrc\n",
    "import rasterio as rio\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blocked-sandwich",
   "metadata": {},
   "source": [
    "### Data Download\n",
    "\n",
    "We will use our NASA EarthData credentials and ASF Vertex to download an InSAR pair data into our notebook directly. For this tutorial, we will be working with UAVSAR data from February of 2020. If you want to use different data in the future, change the links in the files variable. The screengrab below shows how I generated these download links from the ASF site."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "introductory-liabilities",
   "metadata": {},
   "source": [
    ":::{figure-md} vertex\n",
    "<img src=\"../../img/asf_vertex.png\" alt=\"asf vertex\" width=\"800px\">\n",
    "\n",
    "Screenshot of ASF Vertex interface\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chemical-comparison",
   "metadata": {},
   "source": [
    "Now we only have the HH polarization, the annotation file, and the 6 .grd files!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "middle-chile",
   "metadata": {},
   "source": [
    "## Converting Data to GeoTiffs\n",
    "\n",
    "The downloadable UAVSAR data comes in a flat binary format (.grd), which is not readable by GDAL (Geospatial Data Abstraction Library). Therefore it needs to be transformed for use in standard spatial analysis software (ArcGIS, QGIS, Python, R, MATLAB, etc.). To do this, we will use the uavsar_tiff_convert function, which takes information (latitude, longitude, number of lines and samples, data type, pixel size) from the annotation file to create an ENVI header (.hdr). Once the ENVI header is created, the files can be read into Python and converted to GeoTiffs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extended-lecture",
   "metadata": {},
   "source": [
    "This function pulls out information from the annotation file, builds and ENVI header, and then converts the data to GeoTIFFS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "centered-developer",
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder is path to a folder with an .ann (or .txt) and .grd files (.amp1, .amp2, .cor, .unw, .int)\n",
    "\n",
    "def uavsar_tiff_convert(folder):\n",
    "    \"\"\"\n",
    "    Builds a header file for the input UAVSAR .grd file,\n",
    "    allowing the data to be read as a raster dataset.\n",
    "    :param folder:   the folder containing the UAVSAR .grd and .ann files\n",
    "    \"\"\"\n",
    "\n",
    "    os.chdir(folder)\n",
    "    int_file = glob.glob(os.path.join(folder, 'int.grd'))\n",
    "\n",
    "    # Empty lists to put information that will be recalled later.\n",
    "    Lines_list = []\n",
    "    Samples_list = []\n",
    "    Latitude_list = []\n",
    "    Longitude_list = []\n",
    "    Files_list = []\n",
    "\n",
    "    # Step 1: Look through folder and determine how many different flights there are\n",
    "    # by looking at the HDR files.\n",
    "    for files in os.listdir(folder):\n",
    "        if files [-4:] == \".grd\":\n",
    "            newfile = open(files[0:-4] + \".hdr\", 'w')\n",
    "            newfile.write(\"\"\"ENVI\n",
    "description = {DESCFIELD}\n",
    "samples = NSAMP\n",
    "lines = NLINE\n",
    "bands = 1\n",
    "header offset = 0\n",
    "data type = DATTYPE\n",
    "interleave = bsq\n",
    "sensor type = UAVSAR L-Band\n",
    "byte order = 0\n",
    "map info = {Geographic Lat/Lon, \n",
    "            1.000, \n",
    "            1.000, \n",
    "            LON, \n",
    "            LAT,  \n",
    "            0.0000555600000000, \n",
    "            0.0000555600000000, \n",
    "            WGS-84, units=Degrees}\n",
    "wavelength units = Unknown\n",
    "                \"\"\"\n",
    "                          )\n",
    "            newfile.close()\n",
    "            if files[0:18] not in Files_list:\n",
    "                Files_list.append(files[0:18])\n",
    "\n",
    "    #Variables used to recall indexed values.\n",
    "    var1 = 0\n",
    "\n",
    "    #Step 2: Look through the folder and locate the annotation file(s).\n",
    "    # These can be in either .txt or .ann file types.\n",
    "    for files in os.listdir(folder):\n",
    "        if Files_list[var1] and files[-4:] == \".txt\" or files[-4:] == \".ann\":\n",
    "            #Step 3: Once located, find the info we are interested in and append it to\n",
    "            # the appropriate list. We limit the variables to <=1 so that they only\n",
    "            # return two values (one for each polarization of\n",
    "            searchfile = codecs.open(files, encoding = 'windows-1252', errors='ignore')\n",
    "            for line in searchfile:\n",
    "                if \"Ground Range Data Latitude Lines\" in line:\n",
    "                    Lines = line[65:70]\n",
    "                    print(f\"Number of Lines: {Lines}\")\n",
    "                    if Lines not in Lines_list:\n",
    "                        Lines_list.append(Lines)\n",
    "\n",
    "                elif \"Ground Range Data Longitude Samples\" in line:\n",
    "                    Samples = line[65:70]\n",
    "                    print(f\"Number of Samples: {Samples}\")\n",
    "                    if Samples not in Samples_list:\n",
    "                        Samples_list.append(Samples)\n",
    "\n",
    "                elif \"Ground Range Data Starting Latitude\" in line:\n",
    "                    Latitude = line[65:85]\n",
    "                    print(f\"Top left lat: {Latitude}\")\n",
    "                    if Latitude not in Latitude_list:\n",
    "                        Latitude_list.append(Latitude)\n",
    "\n",
    "                elif \"Ground Range Data Starting Longitude\" in line:\n",
    "                    Longitude = line[65:85]\n",
    "                    print(f\"Top left Lon: {Longitude}\")\n",
    "                    if Longitude not in Longitude_list:\n",
    "                        Longitude_list.append(Longitude)\n",
    "    \n",
    "                        \n",
    "                 \n",
    "            #Reset the variables to zero for each different flight date.\n",
    "            var1 = 0\n",
    "            searchfile.close()\n",
    "\n",
    "\n",
    "    # Step 3: Open .hdr file and replace data for all type 4 (real numbers) data\n",
    "    # this all the .grd files expect for .int\n",
    "    for files in os.listdir(folder):\n",
    "        if files[-4:] == \".hdr\":\n",
    "            with open(files, \"r\") as sources:\n",
    "                lines = sources.readlines()\n",
    "            with open(files, \"w\") as sources:\n",
    "                for line in lines:\n",
    "                    if \"data type = DATTYPE\" in line:\n",
    "                        sources.write(re.sub(line[12:19], \"4\", line))\n",
    "                    elif \"DESCFIELD\" in line:\n",
    "                        sources.write(re.sub(line[15:24], folder, line))\n",
    "                    elif \"lines\" in line:\n",
    "                        sources.write(re.sub(line[8:13], Lines, line))\n",
    "                    elif \"samples\" in line:\n",
    "                        sources.write(re.sub(line[10:15], Samples, line))\n",
    "                    elif \"LAT\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Latitude, line))\n",
    "                    elif \"LON\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Longitude, line))\n",
    "                    else:\n",
    "                        sources.write(re.sub(line, line, line))\n",
    "    \n",
    "    # Step 3: Open .hdr file and replace data for .int file date type 6 (complex)                 \n",
    "    for files in os.listdir(folder):\n",
    "        if files[-8:] == \".int.hdr\":\n",
    "            with open(files, \"r\") as sources:\n",
    "                lines = sources.readlines()\n",
    "            with open(files, \"w\") as sources:\n",
    "                for line in lines:\n",
    "                    if \"data type = 4\" in line:\n",
    "                        sources.write(re.sub(line[12:13], \"6\", line))\n",
    "                    elif \"DESCFIELD\" in line:\n",
    "                        sources.write(re.sub(line[15:24], folder, line))\n",
    "                    elif \"lines\" in line:\n",
    "                        sources.write(re.sub(line[8:13], Lines, line))\n",
    "                    elif \"samples\" in line:\n",
    "                        sources.write(re.sub(line[10:15], Samples, line))\n",
    "                    elif \"LAT\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Latitude, line))\n",
    "                    elif \"LON\" in line:\n",
    "                        sources.write(re.sub(line[12:15], Longitude, line))\n",
    "                    else:\n",
    "                        sources.write(re.sub(line, line, line))\n",
    "                        \n",
    "    \n",
    "    # Step 4: Now we have an .hdr file, the data is geocoded and can be loaded into python with rasterio\n",
    "    # once loaded in we use gdal.Translate to convert and save as a .tiff\n",
    "    \n",
    "    data_to_process = glob.glob(os.path.join(folder, '*.grd')) # list all .grd files\n",
    "    for data_path in data_to_process: # loop to open and translate .grd to .tiff, and save .tiffs using gdal\n",
    "        raster_dataset = gdal.Open(data_path, gdal.GA_ReadOnly)\n",
    "        raster = gdal.Translate(os.path.join(folder, os.path.basename(data_path) + '.tiff'), raster_dataset, format = 'Gtiff', outputType = gdal.GDT_Float32)\n",
    "    \n",
    "    # Step 5: Save the .int raster, needs separate save because of the complex format\n",
    "    data_to_process = glob.glob(os.path.join(folder, '*.int.grd')) # list all .int.grd files (only 1)\n",
    "    for data_path in data_to_process:\n",
    "        raster_dataset = gdal.Open(data_path, gdal.GA_ReadOnly)\n",
    "        raster = gdal.Translate(os.path.join(folder, os.path.basename(data_path) + '.tiff'), raster_dataset, format = 'Gtiff', outputType = gdal.GDT_CFloat32)\n",
    "\n",
    "    print(\".tiffs have been created\")\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "interior-samba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/Users/jacktarricone/boise_uavsar/HH/lowman_23205_21002-004_21004-003_0005d_s01_L090HH_01.hgt.grd', '/Users/jacktarricone/boise_uavsar/HH/lowman_23205_21002-004_21004-003_0005d_s01_L090HH_01.cor.grd', '/Users/jacktarricone/boise_uavsar/HH/lowman_23205_21002-004_21004-003_0005d_s01_L090HH_01.int.grd', '/Users/jacktarricone/boise_uavsar/HH/lowman_23205_21002-004_21004-003_0005d_s01_L090HH_01.unw.grd', '/Users/jacktarricone/boise_uavsar/HH/lowman_23205_21002-004_21004-003_0005d_s01_L090HH_01.ann']\n"
     ]
    }
   ],
   "source": [
    "data_folder = '/Users/jacktarricone/boise_uavsar/HH/' # define folder where the .grd and .ann files are\n",
    "print(glob.glob(\"/Users/jacktarricone/boise_uavsar/HH/*.*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "infrared-industry",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Lines: 16064\n",
      "Number of Samples: 24968\n",
      "Top left lat: 44.485669680000001  \n",
      "Top left Lon: -116.374142520000007\n",
      ".tiffs have been created\n"
     ]
    }
   ],
   "source": [
    "uavsar_tiff_convert(data_folder) # call the tiff convert function, and it will print the information it extracted from the .ann file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "immune-ribbon",
   "metadata": {},
   "source": [
    "Now we'll delete the unneeded .grd and .hdr files that our tiffs have been created. If you're using this code on your local machine, this probably isn't absolutely necessary. The JupyterHub cloud we're working in has limited space, so deletion is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sublime-disease",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(data_folder)\n",
    "grd = glob.glob('*.grd') #define .grd\n",
    "hdr = glob.glob('*.hdr*') #define .hdr\n",
    "\n",
    "# remove both\n",
    "for f in grd:\n",
    "    os.remove(f)\n",
    "    \n",
    "for f in hdr:\n",
    "    os.remove(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "sticky-dynamics",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check what's in the directory, only .tiffs and our annotation file!\n",
    "print(glob.glob(\"*.*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "portuguese-cornell",
   "metadata": {},
   "outputs": [],
   "source": [
    "### inspect our newly created .tiffs, and create named objects for each data type. We'll use these new obects in the next step\n",
    "\n",
    "# amplitude from the first acquisition\n",
    "for amp1 in glob.glob(\"*amp1.grd.tiff\"):\n",
    "    print(amp1)\n",
    "    \n",
    "# amplitude from the second acquisition\n",
    "for amp2 in glob.glob(\"*amp2.grd.tiff\"):\n",
    "    print(amp2)\n",
    "\n",
    "# coherence\n",
    "for cor in glob.glob(\"*cor.grd.tiff\"):\n",
    "    print(cor)\n",
    "\n",
    "# unwrapped phase\n",
    "for unw in glob.glob(\"*unw.grd.tiff\"):\n",
    "    print(unw)\n",
    "\n",
    "# dem used in processing\n",
    "for dem in glob.glob(\"*hgt.grd.tiff\"):\n",
    "    print(dem)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "multiple-heather",
   "metadata": {},
   "source": [
    "Inspect the meta data the rasters using the rio (shorthand for rasterio) ```profile``` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "metric-poverty",
   "metadata": {},
   "outputs": [],
   "source": [
    "unw_rast = rio.open(unw)\n",
    "meta_data = unw_rast.profile\n",
    "print(meta_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wireless-feeling",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:test_env]",
   "language": "python",
   "name": "conda-env-test_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}