lidar_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "797f557e-435f-49cd-8002-2135758b8c67",
   "metadata": {},
   "source": [
    "# Lidar Processing Workflow\n",
    "**Cara Piske, Graduate Program of Hydrologic Sciences, 2022; Advisor: Dr. Adrian Harpold**<br>\n",
    "This code processes raw lidar point clouds in order to calculate snow depth using PDAL. <br>\n",
    "Lidar data were provided by the Airborne Snow Observatory (ASO), the National Center for Airborne Laser Mapping (NCALM), and Watershed Sciences Inc. (WSI). <br>\n",
    "\n",
    "The goal of this project is to process snow depth to the one-meter spatial scale while maintaining conservative under-canopy snow depth estimates. Therefore, little interpolation occurs under-canopy. NCALM and WSI flights were obtained through OpenTopography."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e7346b-ed91-48fe-912a-4dc9d7b20be4",
   "metadata": {},
   "source": [
    "Start by importing necessary files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ef19ecac-1150-469a-9a3f-3eb6f8b1c531",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary files\n",
    "import numpy as np \n",
    "import matplotlib.pyplot as plt\n",
    "import json # where we will save the json files to run a pipeline\n",
    "import os # for file management\n",
    "import subprocess # allows us to run command line commands\n",
    "import pdal # lidar processing package\n",
    "import rasterio\n",
    "import pyproj\n",
    "from osgeo import gdal\n",
    "\n",
    "import time\n",
    "# packages to copy files\n",
    "from pathlib import Path\n",
    "import shutil\n",
    "import glob\n",
    "\n",
    "# packages to extract wkt from polygon\n",
    "import shapefile\n",
    "import pygeoif\n",
    "\n",
    "# for parallel processing\n",
    "import concurrent.futures\n",
    "from multiprocessing import Pool\n",
    "import csv\n",
    "\n",
    "# See lidar_functions.py\n",
    "import lidar_functions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afec494a-5f52-4fc5-8ced-a5886f938892",
   "metadata": {},
   "source": [
    "Note that many functions are dependent on specific directory structures. See README"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dfd00aac-03ba-4d6d-a022-bbfad350854f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Set current working directory\n",
    "# path = 'path/to/current/working/directory/'\n",
    "# os.chdir(path)\n",
    "# os.getcwd() # print to ensure we're in the right directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "11eb2762-4c68-4a8e-a2e0-3c4434606bbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_base_path = 'path/to/JSON/files/' # set so that we can redefine json across operating systems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1933dbf5-603d-4eac-a4df-8ec90e6278d4",
   "metadata": {},
   "source": [
    "Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e7a6c326-633d-4550-b011-0f012e2b2825",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'G:\\\\'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set current working directory\n",
    "path = '/'\n",
    "os.chdir(path)\n",
    "os.getcwd() # print to ensure we're in the right directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "87f30faa-fe09-4f19-b736-214b2aad11c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_base_path = 'piske_processing/PDAL_workflow/JSON/' # set so that we can redefine json across operating systems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8ca020-c0f4-40e5-92c0-f102dad6cb9e",
   "metadata": {},
   "source": [
    "# Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86795e51-90af-4190-9561-cbcb5498f04f",
   "metadata": {},
   "source": [
    "There are two ways to process LAS files. An efficient way, depending on computing power, is through parallelization. This involves running processes on tiles by spreading out the computations to multiple cores. Ultimately, the rasterized product will combine all tiles. For some of our processing, we combined all tiles prior to rasterization within a pipeline. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6436eea6-1aef-403f-a7be-0d4195cbdd8f",
   "metadata": {},
   "source": [
    "## Info\n",
    "Get to know lidar file... <br>\n",
    "In general, running PDAL python bindings can be difficult, so an easy workaround is to initiate a terminal command and run via subprocess. This is what we'll do for most of the processing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28fce8d2-28ae-466b-a779-d3640b9cb37b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # define input file\n",
    "# input_lid = 'full/path/to/las/lidar.las'\n",
    "# pdal_info_cmd = ['pdal','info',input_lid] # general info\n",
    "# pdal_metadata_cmd = ['pdal','info',input_lid,'--metadata'] # full file metadata, including details crs\n",
    "# subprocess.run(pdal_metadata_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32ee9d55-d48f-4228-a49d-76b7f83e8370",
   "metadata": {},
   "source": [
    "Save results to a dictionary <br>\n",
    "This will be useful below when renaming files based on specific metadata... sometimes results aren't printed above so this is also helpful to isolate outputs and print. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "41ce21a8-7035-4261-a81e-83b58a9e26cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pdal_info_results = subprocess.run(pdal_metadata_cmd, stdout = subprocess.PIPE) # stout (standard out), PIPE indicates that a new pipe to the child should be created\n",
    "# pdal_info_dict = json.loads(pdal_info_results.stdout.decode()) # create dict with metadata info\n",
    "# pdal_info_dict # print results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e4913e1-d156-44ce-b3af-9c458d3c1e3d",
   "metadata": {},
   "source": [
    "## Writers Bounds\n",
    "Set the extents of the final output raster file. Setting this prior to raster operations is beneficial because it presets pixel delineation, prevents major cropping/resampling, and allows raster operations to occur directly from the final product. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479cb008-1e56-44fd-9c60-62af0cf950ec",
   "metadata": {},
   "source": [
    "Check extents of all files in a directory (in order to create bounds for rasterization below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24bae246-98de-4fd3-bc62-61fbf7609012",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_folder = 'path/to/directory/'\n",
    "onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "full_paths = [lidar_folder +f for f in onlyfiles]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "731b95b6-8837-498c-937d-97e69981d04e",
   "metadata": {},
   "source": [
    "Use lidar_functions to extract the min and max extents of each file in a flight <br>\n",
    "For our purposes, we compared extents of all flights and used the values from the flight with the most limited spatial extent. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35805d87-9081-4043-91f7-70e1bf5984a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Use lidar_functions to extract the min and max extents of each file in a flight\n",
    "# extents = lidar_functions.check_flight_extent(full_paths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c7d1967-1c1a-49e4-be56-7c34bc7f6246",
   "metadata": {},
   "outputs": [],
   "source": [
    "# writers_bounds = '([llx, urx], [lly, ury])'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3011c82-ff21-4e0a-8194-7b38b6553d66",
   "metadata": {},
   "source": [
    "Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd33abaf-570b-4473-8a3f-ca795945c8a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# writers_bounds = '([730235.96, 738826.45], [4364741.66, 4372273.16])'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adaa2852-9e99-4e2f-ab6e-d1ee311ec38c",
   "metadata": {},
   "source": [
    "## Retile\n",
    "Retiling lidar files allows for parallelization, speeding up processing. Standard tiles come in 500-1500m extents. Choose depending on point density and spatial extent of files. Many NCALM files come in 1000x1000m already. Buffer zones are dependent on point density etc. and allow for redundant classifications to avoid the edge effect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f53392-26c0-4904-88e2-5f1a1749e6c5",
   "metadata": {},
   "source": [
    "**Single Folder**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "84755a16-6c56-4a92-839f-5f6d09e1bf44",
   "metadata": {},
   "outputs": [],
   "source": [
    "# retile_input_path = 'path/to/input/lidar/folder/' # define directory to original LAS files\n",
    "# retile_output_path = 'path/to/output/lidar/folder/' # define directory to output LAS tiles\n",
    "# onlyfiles = [f for f in os.listdir(retile_input_path) if os.path.isfile(os.path.join(retile_input_path, f))] # create var list of every file in input directory\n",
    "# for files in onlyfiles: # for each of these files\n",
    "#     full_path = os.path.join(retile_input_path, files) # define the full file path\n",
    "#     output_path = retile_output_path+'retile_#'+files # create new file path\n",
    "#     retile_command = ['pdal', 'tile', full_path, output_path, '--length=1000','--buffer=50'] # define retile command to be run in the termine\n",
    "#     subprocess.run(retile_command) # run the command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d29947-d28b-4f15-a15f-ba45ace15a23",
   "metadata": {},
   "source": [
    "## Rename\n",
    "Many files come with inconsistent naming (including unsupported characters...) <br>\n",
    "Rename all files to maintain consistency. Here, our file structure is flight_watershed_date_llx_lly <br>\n",
    "Note that some functions are written based on "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fcd5305-ef7d-4586-889f-7c36d07b0cd5",
   "metadata": {},
   "source": [
    "Start by renaming with llx and lly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a08ac24-eb5f-4486-8da8-0c4f47e6ce95",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Many files come with inconsistent naming (including unsupported characters...)\n",
    "# # Rename all files to maintain consistency\n",
    "\n",
    "# # Rename a file with the lower left x and y defined as the corner point of the bounding box (add resolution/2 to get the center point of the box) \n",
    "# # input - full lidar file path (i.e. 'folder1/folder2/SCB/flight1/lid_files/filename.laz'), [str]\n",
    "\n",
    "# def rename_llx_lly(full_path):\n",
    "    \n",
    "#     pdal_info_command = ['pdal', 'info', full_path, '--metadata'] # set up pdal info command\n",
    "#     pdal_info_results = subprocess.run(pdal_info_command, stdout = subprocess.PIPE) # stout (standard out), PIPE indicates that a new pipe to the child should be created, execute command\n",
    "#     pdal_info_dict = json.loads(pdal_info_results.stdout.decode()) # save metadata to dictionary\n",
    "    \n",
    "#     pathname = os.path.dirname(os.path.realpath(full_path)) # isolate only pathname (i.e. 'folder1/folder2/SCB/flight1/lid_files/')\n",
    "#     new_name = os.path.join(pathname, str(round(pdal_info_dict['metadata']['minx'])) +\"_\"+ str(round(pdal_info_dict['metadata']['miny']))+full_path[-4:])    \n",
    "#     os.rename(full_path, new_name) # rename file\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cbc3829-54be-4f56-a0df-e6a197c49ef2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # run similar function except add in file tag. This is helpful for retiled files where there may be redundant llx_lly values \n",
    "# def rename_llx_lly_repeats(full_path):\n",
    "    \n",
    "#     pdal_info_command = ['pdal', 'info', full_path, '--metadata'] # set up pdal info command\n",
    "#     pdal_info_results = subprocess.run(pdal_info_command, stdout = subprocess.PIPE) # stout (standard out), PIPE indicates that a new pipe to the child should be created, execute command\n",
    "#     pdal_info_dict = json.loads(pdal_info_results.stdout.decode()) # save metadata to dictionary\n",
    "#     pathname = os.path.dirname(os.path.realpath(full_path)) # isolate only pathname (i.e. 'folder1/folder2/SCB/flight1/lid_files/')\n",
    "#     new_name = os.path.join(pathname, str(round(pdal_info_dict['metadata']['minx'])) +\"_\"+ str(round(pdal_info_dict['metadata']['miny']))+full_path[-4:])    \n",
    "#     if os.path.exists(new_name):\n",
    "#         lidar_folder = pathname\n",
    "#         onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "#         full_paths = [os.path.join(lidar_folder, s) for s in onlyfiles] \n",
    "#         full_str = ','.join(full_paths)\n",
    "#         num_occurences = full_str.count(new_name[:-4])\n",
    "#         new_name_b = new_name[:-4]+'_'+str(num_occurences)+new_name[-4:]\n",
    "#         os.rename(full_path, new_name_b)\n",
    "#     else:\n",
    "#         os.rename(full_path, new_name) # rename file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e18d07-0d9f-439a-b71d-f38b9db65c5e",
   "metadata": {},
   "source": [
    "Add consistent string (FlightSource_Watershed_date) to the beginning of the filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "761a2f81-6bcf-4388-ad5f-de698c216f50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # See lidar_functions.py\n",
    "# # rename a file with additional metadata at the beginning of the filename\n",
    "# # flight organization, watershed, date of flight (i.e. ASO_ICB_20140423)\n",
    "# # additional text will be taken from the flight directory name, could hardcode additional string instead of using folder name\n",
    "# # input - full lidar file path (i.e. 'lidar/lidar_files/filename.laz'), str\n",
    "\n",
    "# def add_str_to_filename(full_path):\n",
    "#     filename = os.path.basename(full_path) # isolate only filename (i.e 'filename.laz')\n",
    "#     pathname = os.path.dirname(os.path.realpath(full_path)) # isolate only path name (i.e. 'lidar/lidar_files')\n",
    "#     add_str = os.path.normpath(pathname) # split up the path name (i.e. full path)\n",
    "#     add_str = [i for i in add_str.split(os.sep) if (i.startswith('ASO_') or i.startswith('NCALM_') or i.startswith('WSI_'))] # (i.e. 'ASO_SCB_2016')\n",
    "#     rename = os.path.join(pathname, add_str[0] + '_'+ filename) # add string to full pathname \n",
    "#     os.rename(full_path, rename) # rename file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99f22c8c-db32-4cb5-ba75-593e3004bdb7",
   "metadata": {},
   "source": [
    "**Single file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8a6166-4d4d-4841-be9b-e7b1bb5715eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# rename_llx_lly('filepath.laz')\n",
    "# add_str_to_filename('new_filepath.laz')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cff4837-64c2-4eac-bb08-50ab9339c178",
   "metadata": {},
   "source": [
    "**Parallel Processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e4cc2f-6a00-4fde-9a32-c4b5e3605ee8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lidar_folder = 'path/to/lid/folder/'\n",
    "# tic = time.perf_counter()\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "#         full_path = [os.path.join(lidar_folder, s) for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.rename_llx_lly, full_path) #running 10 times\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0fc5fd4-3cb3-4304-acd3-fb2d3462c854",
   "metadata": {},
   "source": [
    "*Repeat with lidar_functions.add_str_to_filename...*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f32942d-f717-4927-b44a-53e966ae1cf2",
   "metadata": {},
   "source": [
    "use if there are - in the output filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ff3226e-cbc3-4afe-9dd7-574b297c6f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # use if there are - in the output filename\n",
    "# pathname = 'Piske_lidar/MRB/Merced_lidar/ASO/ASO_MRB_20210429/tindex/tiles/'\n",
    "# for files in [f for f in os.listdir(pathname) if os.path.isfile(os.path.join(pathname, f))]:\n",
    "#     target = files.replace(\"-\",\"\")\n",
    "#     target = pathname + target\n",
    "#     full_path = pathname+files\n",
    "#     os.rename(full_path,target)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4192a006-96c8-4f98-be10-3c3c7b3b5200",
   "metadata": {},
   "source": [
    "## Save tile boundaries\n",
    "Save sqlite (shp) of tile boundaries of retiles <br>\n",
    "Alter filepaths "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8700fa7-5eb7-4315-ae71-8a8f8e1e7c2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # save the tile index of a file to a new folder\n",
    "\n",
    "# # input - full lidar/sqlite file patsh (i.e. 'lidar/lidar_files/filename.laz'), [str, str]\n",
    "# def create_tindex(input_path, output_path):\n",
    "#     boundary_cmd = ['pdal', 'tindex', 'create', '--tindex', output_path, '--filespec', input_path, '-f', 'SQLite']\n",
    "#     subprocess.run(boundary_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae36ff13-c4ea-4b39-93d6-437501ca1a43",
   "metadata": {},
   "source": [
    "**Parallel Processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc5ae699-3088-4040-8387-5a9c5a4c6537",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # one folder\n",
    "# tic = time.perf_counter()\n",
    "# lidar_folder = 'path/to/lidar/files/'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "#         full_path = [os.path.join(lidar_folder, s) for s in onlyfiles]\n",
    "#         # change depending on directory formats\n",
    "#         output_path = [os.path.join(os.path.basename(lidar_folder),'tindex' + s[:-3] + 'sqlite') for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.create_tindex, full_path, output_path)\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c71a2070-2c2d-4f4d-acc7-2982914f2b81",
   "metadata": {},
   "source": [
    "## Copy files\n",
    "Save a copy of files that meet criteria into a new folder <br>\n",
    "The goal is to decrease processing time for datum conversions in the MRB <br>\n",
    "Change file extents depending on watershed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f073a7f7-f2cb-4bfc-a220-5261c4a03627",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # copy file based on llx and lly into output filename. Note that this function relies on the assumption that files follow the structure \n",
    "\n",
    "# # input_lid 'folder1/folder2/folder3a/filename.laz' and output_lid 'folder1/folder2/folder3b/filename.laz'\n",
    "# # if files are held in a different file structure, code must be changed to accomodate change\n",
    "# # input - full path to a lid file [str]\n",
    "# # output_path - path to folder with ICB tiles (e.g. 'path/to/folder/') [str]\n",
    "# def copy_lid_by_ext_ICB(full_path, output_path):\n",
    "#     pdal_info_command = ['pdal', 'info', full_path, '--metadata'] # set up pdal command\n",
    "#     pdal_info_results = subprocess.run(pdal_info_command, stdout = subprocess.PIPE) # stout (standard out), PIPE indicates that a new pipe to the child should be created, execute command\n",
    "#     pdal_info_dict = json.loads(pdal_info_results.stdout.decode()) # save metadata to dict\n",
    "#     # extract llx,lly of tile\n",
    "#     minx = pdal_info_dict['metadata']['minx']\n",
    "#     miny = pdal_info_dict['metadata']['miny']\n",
    "#     maxx = pdal_info_dict['metadata']['maxx']\n",
    "#     maxy = pdal_info_dict['metadata']['maxy']\n",
    "#     # if file origin is within bounds of ICB extents, copy the file\n",
    "#     if (minx <= 288000 and maxx >= 265000):\n",
    "#         if (maxy >= 4165000 and miny<= 4180000):\n",
    "#             input_lid = full_path\n",
    "#             output_lid = output_path + os.path.basename(full_path)\n",
    "#             pdal_copy_cmd = ['pdal','translate', input_lid, output_lid]\n",
    "#             subprocess.run(pdal_copy_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa00a1a8-28bd-4752-9f6b-6d97899287f3",
   "metadata": {},
   "source": [
    "**Parallel Processing**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72a7a6f8-70c5-43cc-bc6f-f7220b2bbffe",
   "metadata": {},
   "source": [
    "All folders called \"retile\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f78ecbd-88af-42d9-9ad0-fa217cfb4cbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# all_folders = [x[0] for x in os.walk('path/to/directory/')]\n",
    "# # list indices of all folders that are called laz\n",
    "# index_pos_list = [ i for i in range(len(all_folders)) if all_folders[i][-6:] == 'retile' ]\n",
    "# # save only those files \n",
    "# retile_list = [all_folders[i] for i in index_pos_list]\n",
    "# for lidar_folder in retile_list:\n",
    "#     if __name__ == '__main__':\n",
    "#         with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#             onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "#             full_path = [os.path.join(lidar_folder, s) for s in onlyfiles]\n",
    "#             executor.map(copy_by_ext_ICB, full_path)\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc08c8f-e7f2-430a-b8d3-7649064e1287",
   "metadata": {
    "tags": [],
    "toc-hr-collapsed": true
   },
   "source": [
    "## Datum Conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913244f4-51c7-4c8a-ae9c-9f1ed0ccdaab",
   "metadata": {},
   "source": [
    "### Horizontal Datum Conversion\n",
    "From PDAL library: https://pdal.io/tutorial/grid-shift.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b77dbfe-5199-4a43-9231-2b022fe93509",
   "metadata": {},
   "outputs": [],
   "source": [
    "horizontal_datum_conversion = json_base_path+'horizontal_datum_conversion.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cb3d599-0898-4f70-9035-6a8d5bea7657",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_las = 'path/to/filename/filename.las'\n",
    "# output_las = \"path/to/filename/filename.las\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "820da99a-637f-438d-bb6d-cfbd5a8bde28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reader_dict = {\"type\":\"readers.las\",\n",
    "#               \"filename\":input_reproj_hor}\n",
    "\n",
    "# reproject_dict = {\"type\":\"filters.reprojection\",\n",
    "#                   \"in_srs\":\"+proj=utm +zone=10 +datum=WGS84 +units=m +no_defs\",\n",
    "#                   \"out_srs\":\"EPSG:26910\"} #\"in_srs\":\"EPSG:8999\"\n",
    "# writer_dict = {\"type\":\"writers.las\",\n",
    "#                \"a_srs\":\"EPSG:26910\",\n",
    "#               #  \"scale_x\":0.00000001,\n",
    "#               #  \"scale_y\":0.00000001,\n",
    "#               # \"offset_x\":\"auto\",\n",
    "#               # \"offset_y\":\"auto\",\n",
    "#               \"filename\":output_reproj_hor}\n",
    "\n",
    "# pipeline_list = [reader_dict, reproject_dict, writer_dict]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(horizontal_datum_conversion, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d367333-36ed-4375-be85-292e1cb5504c",
   "metadata": {},
   "source": [
    "### Vertical Datum Conversion\n",
    "using output of code above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "084ed65b-202b-432b-a24f-d916ef6c8cff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # best option for now \n",
    "# input_las = 'path/to/filename.las'\n",
    "# output_las = 'path/to/filename.las'\n",
    "# gtx_path = 'path/to/fielname.gtx' # /cpiske/lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/reproject/gtx_links/core/geoid12b/g2012bu5.gtx\n",
    "# reproj_fil_in_srs = '--filters.reprojection.in_srs=EPSG:26910+5703'\n",
    "# reproj_fil_out_srs = '--filters.reprojection.out_srs=+init=EPSG:26910' + ' +geoidgrids='+gtx_path + ' +t_epoch=2010.0'\n",
    "# writers_compression = '--writers.las.compression=true'\n",
    "# writers_a_srs = '--writers.las.a_srs=+proj=utm +zone=10 +ellps=GRS80 +datum=NAD83 +units=m +no_defs' "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9597edf1-6d46-4850-ac9c-fe5d127ccaa9",
   "metadata": {},
   "source": [
    "# Rasterize\n",
    "Set up the rasterization pipeline which we will use throughout the workflow. <br>\n",
    "We use two pipelines here, one which takes the mean of all points in a 1m pixel, using a 0.7m radius. <br>\n",
    "*Note that this step is now incorporated into many of the below pipelines*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a1b7e4d8-b390-460f-b479-23c7a453b4a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name JSON file\n",
    "rasterize_json_mean = json_base_path+'rasterize_mean.json'\n",
    "rasterize_json_count = json_base_path+'rasterize_count.json'\n",
    "rasterize_json_max = json_base_path+'rasterize_max.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "299fe1e3-959c-44aa-a2d9-e5eb9efb5c42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a pipeline and save to a json file \n",
    "reader_dict = {'type':'readers.las'}\n",
    "writers_gdal= {\"type\": \"writers.gdal\",\n",
    "              'output_type': 'max',\n",
    "              'resolution': '1.0',\n",
    "              'radius': '0.7'}#,\n",
    "             #'window_size':3}\n",
    "\n",
    "pipeline_list = [reader_dict,writers_gdal]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "with open(rasterize_json_max, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e6a74d90-1f76-47fd-a439-0ddb86a1b5f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.perf_counter()\n",
    "input_path = 'MRB/Merced_lidar/NCALM/NCALM_MRB_20180921/HAG/'\n",
    "output_path = 'MRB/Merced_lidar/NCALM/NCALM_MRB_20180921/CHM/'\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "        full_input_las = [input_path + s for s in onlyfiles]\n",
    "        full_output_tif = [output_path + s[:-4] + '.tif' for s in onlyfiles]\n",
    "        executor.map(lidar_functions.rasterize_max, full_input_las, full_output_tif) \n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adeead3-aada-4acc-9f3a-9125d76bab43",
   "metadata": {},
   "source": [
    "**Single Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e07c5e51-9641-4e7b-aacf-2826f3db4c58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Single Files\n",
    "# reader = '--readers.las.filename='+'path/to/file/filename.las'\n",
    "# writer = '--writers.gdal.filename='+'path/to/file/filename.tif'\n",
    "# rasterize_command = ['pdal', 'pipeline', rasterize_json_mean, writer, reader]\n",
    "# subprocess.run(rasterize_command)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6515d31e-20fd-455d-9c57-ad2c3b5d4935",
   "metadata": {},
   "source": [
    "**parallel processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eeb31ce-5fc0-43c6-9135-d2ed202c6996",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/lid/'\n",
    "# output_path = 'path/to/tif/'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_input_las = [input_path + s for s in onlyfiles]\n",
    "#         full_output_tif = [output_path + s[:-4] + '.tif' for s in onlyfiles]\n",
    "#         executor.map(lidar_functionsrasterize_count, full_input_las, full_output_tif) \n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3608299f-146c-46d5-84bf-2e4d9425eb40",
   "metadata": {},
   "source": [
    "# Ground Filter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f5d8d12-ab95-4eae-82ea-4c9cc49ad718",
   "metadata": {
    "tags": []
   },
   "source": [
    "Filter classified las files based on [Las 1.4 Specifications](http://www.asprs.org/wp-content/uploads/2019/03/LAS_1_4_r14.pdf)<br>\n",
    "Where Classification 2 = ground <br>\n",
    "Classification 7 = noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b188f6d2-0f8d-4aab-90c4-e86f024ef402",
   "metadata": {},
   "outputs": [],
   "source": [
    "gf_json = json_base_path+'ground_filter_preClassified.json' # define path to json files\n",
    "\n",
    "filter_range = {\"type\":\"filters.range\", \n",
    "                \"limits\":\"Classification[2:2]\"}\n",
    "pipeline_list = [filter_range]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# save to json\n",
    "with open(gf_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e43df2ce-870c-4cda-8bae-99551a154645",
   "metadata": {},
   "source": [
    "**Single File**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0aea2950-4175-4d41-976a-4cebe29a9709",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_las = 'path/to/lid/file/filenam.las'\n",
    "# output_las = 'path/to/lid/file/filename.las'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a611e7d2-02f1-497c-b16a-bad4ad7c2e31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# range_cmd = ['pdal', 'translate', input_las,  output_las, '--json',gf_json]\n",
    "# subprocess.run(range_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df014e65-d9b4-4b11-ab60-d1d8cfb37818",
   "metadata": {},
   "source": [
    "**All Files in Directory**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da82915d-9afe-4011-a28c-c718d9385d4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# #329.1668573822826s\n",
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/input/directory'\n",
    "# output_path = 'path/to/output/directory'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_path = [input_path + '/' + s for s in onlyfiles]\n",
    "#         output_path_full = [output_path + '/' + s for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.ground_filter_preClassified, full_path, output_path_full) #running 10 times\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53ee030-7399-488f-81d2-a7affd29afe6",
   "metadata": {},
   "source": [
    "### Ground Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f17cb58-397b-42bc-84ba-48bb341858c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "filters_assign = {\"type\":\"filters.assign\",\n",
    "                  \"assignment\":\"Classification[:]=0\"}\n",
    "filters_elm = {'type':'filters.elm'}\n",
    "filters_outlier = {'type':'filters.outlier'}\n",
    "filters_smrf = {\"type\":\"filters.smrf\",\n",
    "                \"last\":true,\n",
    "                \"ignore\":\"Classification[7:7]\",\n",
    "                \"slope\":0.2,\n",
    "                \"window\":16,\n",
    "                \"threshold\":0.45,\n",
    "                \"scalar\":1.2}\n",
    "{\n",
    "      \"type\":\"filters.elm\"\n",
    "    },\n",
    "    {\n",
    "      \"type\":\"filters.outlier\"\n",
    "    },\n",
    "    {\n",
    "      \"type\":\"filters.smrf\",\n",
    "      \"last\":true,\n",
    "      \"ignore\":\"Classification[7:7]\",\n",
    "      \"slope\":0.2,\n",
    "      \"window\":16,\n",
    "      \"threshold\":0.45,\n",
    "      \"scalar\":1.2\n",
    "    },\n",
    "    {\n",
    "      \"type\":\"filters.range\",\n",
    "      \"limits\":\"Classification[2:2]\"\n",
    "    }\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63db602d-9fcc-434c-9ba0-bb1b8c1f106a",
   "metadata": {},
   "source": [
    "## Create DEM\n",
    "This pipeline combines aboves steps to avoid issues with merging rasters. We were seeing tile signatures when we merged all tif files after ground-filtering/rasterization, so here we combine these steps through merging the las and then rasterizing (importantly, we are not writing the merged las files to a new merged file, which defeatst the purpose of tiles). A weakness of this pipeline is that it doesn't allow for tile-level parallelization <br> see: https://pdal.io/pipeline.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e6da2558-582a-4f89-b6d0-a38e2b1c707e",
   "metadata": {},
   "outputs": [],
   "source": [
    "DEM_json = json_base_path+'DEM_from_las.json' # define path to json files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae9b4f2-b703-4abc-a640-d282c933be43",
   "metadata": {},
   "source": [
    "Define the las files we want to create a DEM from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "de06f83c-ddda-4906-b2d3-68e81f9cac79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_path = 'path/to/gf/las/'\n",
    "# output_tif = 'path/to/dem/filenamem.tif'\n",
    "# onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))] # make a list of all filenames in directory\n",
    "# input_list = [input_path + s for s in onlyfiles] # make a list of full filename paths in directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b49074ab-8990-4c85-8daf-d074f144a23a",
   "metadata": {},
   "source": [
    "Create the reader stages of the pipeline. Each file is read as an individual reader stage here, and we cheat here by copying the formatting of json files and creating a dictionary with the values as the correctly formatted inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "04bf4967-3b22-41e9-a61a-d548d791995d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filename_dict = {} # initiate an empty dict to hold the readers\n",
    "# tags = ['']*len(input_list) # initiate an empty list, size = number of files\n",
    "# filenames = ['']*len(input_list) # repeat\n",
    "# for i in range(len(input_list)): # for each file, create a dictionary element with the values matching json formatting for file merging\n",
    "#     filename_dict['filename_'+str(i)] = {'filename':input_list[i], 'tag':'A_'+str(i)}\n",
    "#     tags[i] = 'A_'+str(i) # add a tag to the reader stage\n",
    "#     filenames[i] = filename_dict[list(filename_dict)[i]] # Add all values to a list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad52c80-eb6d-4b7f-bff7-98daa477a0e7",
   "metadata": {},
   "source": [
    "Define the filter and writer stages of the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b18dffb-0523-440d-8e12-d13db245a745",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# merge all las files or stages\n",
    "filter_merge = {\"type\":\"filters.merge\",\n",
    "               \"tag\": \"merged\",\n",
    "               \"inputs\": tags}\n",
    "# filter out the ground points of the tiles\n",
    "filter_range = {\"type\":\"filters.range\",\n",
    "                \"limits\":\"Classification[2:2]\"}\n",
    "# write merged las to raster\n",
    "writers_gdal= {\"type\": \"writers.gdal\",\n",
    "               'output_type': 'mean',\n",
    "              'resolution': '1.0',\n",
    "              'radius': '0.7',\n",
    "               'window_size':3, # we want more of a wall-to-wall product here so we use a secondary algorithm to increase calculation distance\n",
    "               'filename': output_tif}\n",
    "# Append each stage to a list prior to saving to json \n",
    "pipeline_list = filenames.copy()\n",
    "pipeline_list.append(filter_merge)\n",
    "#pipeline_list.append(filter_range)\n",
    "pipeline_list.append(writers_gdal)\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# save to json\n",
    "with open(DEM_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "977a2727-5d65-414a-b13a-52209a70d5ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pdal_cmd = ['pdal','pipeline', DEM_json]\n",
    "# subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ceede00-eb91-4562-9cdc-fb2034abdc4f",
   "metadata": {},
   "source": [
    "# HAG and Noise Filter\n",
    "Height above ground DEM (raster format). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3146179f-5906-4f9e-a81e-a0f939b1cc35",
   "metadata": {},
   "source": [
    "## Replace Z with HAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fab65f9a-5992-49ae-8593-a29fdd88646c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define json path \n",
    "HAG_json = json_base_path + 'HAG_dem.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10ca73ed-543d-4024-a9fe-8dfa8331795d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert all z values to the height above ground \n",
    "target_dem = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/DEM/NCALM_2014_BE.tif'\n",
    "filter_hag = {\"type\":\"filters.hag_dem\",\n",
    "              \"raster\":target_dem, # full file path of target DEM (.tif)\n",
    "              \"zero_ground\":\"false\"} # Do not assign 0 to ground classified points\n",
    "filter_ferry = {\"type\":\"filters.ferry\",\n",
    "                \"dimensions\":\"HeightAboveGround=>Z\"} # replace all Z dimensions with HAG instead of elevation\n",
    "filter_range = {\"type\":\"filters.range\",\n",
    "                \"limits\":\"Z[-0.2:70]\"} # apply a noise filter\n",
    "pipeline_list = [filter_hag, filter_ferry,filter_range]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "with open(HAG_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58c59b90-ae2f-4f10-aea1-d998229902e6",
   "metadata": {},
   "source": [
    "**Single Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0950d373-9976-4959-89b2-823496a7b01c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# input_las = \"path/to/full/filename/filename.laz\"\n",
    "# output_las = \"path/to/full/filename/filename.laz\"\n",
    "# HAG_cmd = ['pdal', 'translate',input_las, output_las, '--json', HAG_json]\n",
    "# subprocess.run(HAG_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff387277-2d63-48ea-99c6-c065c70a076c",
   "metadata": {},
   "source": [
    "**Multiple Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e915847b-3725-4eaa-9e54-31eafcbd0b8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # multiple files\n",
    "# pathname = \"path/to/lid/folder/\"\n",
    "# output_pathname = \"path/to/lid/folder/\"\n",
    "# onlyfiles = [f for f in os.listdir(pathname) if os.path.isfile(os.path.join(pathname, f))]\n",
    "# for file in onlyfiles:\n",
    "#     input_las = pathname + file\n",
    "#     output_las = output_pathname + file\n",
    "#     HAG_cmd = ['pdal', 'translate',input_las, output_las, '--json', HAG_json]\n",
    "#     subprocess.run(HAG_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab32ad77-3a2c-4777-9bf0-f88da02214fb",
   "metadata": {},
   "source": [
    "**Parallel Processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d3c68e1-b2ef-4e2d-b9bd-3c82f3f444a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/lid/folder/'\n",
    "# output_path = 'path/to/lid/folder/'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_path = [input_path + s for s in onlyfiles]\n",
    "#         output_path_full = [output_path + s for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.HAG_dem, full_path, output_path_full) #running 10 times\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4252154-a780-4aba-8f9a-3fefccf8edf2",
   "metadata": {},
   "source": [
    "## Add a new dimension\n",
    "Add a new dimension to the file but maintain Z values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52bdee97-69bd-4634-86f6-368120456623",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # this works to add a new dimension\n",
    "# output_las_hag = 'input/lid/files/lidar.laz'\n",
    "# filter_dem = '--filters.hag_dem.raster=' + target_DEM\n",
    "# hag_addDim = ['pdal', 'translate', input_las_hag, output_las_hag, 'hag_dem' ,filter_dem, '--writers.las.extra_dims=HeightAboveGround=float32']\n",
    "# subprocess.run(hag_addDim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a96d8471-c3d2-4d8c-8399-80123ec762f3",
   "metadata": {},
   "source": [
    "# Vertical Bias Correction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df470a66-ad72-4c4d-9dac-8561ef50dbca",
   "metadata": {},
   "source": [
    "## Move-Merge_Clip\n",
    "1. all vbc values <br>\n",
    "2. vbc values <=5 <br>\n",
    "3. vbc values <=3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e9c65aa-b334-4894-8d1d-1897eb04b52c",
   "metadata": {},
   "source": [
    "### Move files\n",
    "Make a copy of road covered tiles. <br>\n",
    "The spatial extents of the road polygon are 738382,4371452 : 738686,4372063 <br>\n",
    "chose only files with a min x > 737000 and a min y > 4370000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85ed5feb-c166-4a78-9b2a-1c9e8a8446d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # defining source and destination\n",
    "# # paths\n",
    "# src = 'path/to/source/folder/*.la*'\n",
    "# trg = 'path/to/target/folder/target'\n",
    "# file_paths = glob.glob(src)\n",
    "\n",
    "# for files in file_paths:\n",
    "#     filename = os.path.basename(files)\n",
    "#     if int(files[-18:-12]) > 737000 and int(files[-11:-4]) > 4370000:\n",
    "#     # copying the files to the\n",
    "#     # destination directory\n",
    "#         shutil.copy2(files, trg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86d7eee-ca20-4316-8770-9caf004b2983",
   "metadata": {},
   "source": [
    "### Merge\n",
    "Merge files over the road"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ddac58e-07e1-48b2-910e-1771140fd18a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_path = 'path/to/target_lid/*.la*' # define path of input files\n",
    "# output_fname = 'path/to/target/filename.laz'# set output filename\n",
    "# input_fname = glob.glob(input_path) # save to list\n",
    "# pdal_merge_command = input_fname\n",
    "# pdal_merge_command.insert(len(pdal_merge_command),output_fname) # insert output file to list\n",
    "# #pdal_merge_command.insert(0,'-f')\n",
    "# pdal_merge_command.insert(0,'merge')\n",
    "# pdal_merge_command.insert(0,'pdal')\n",
    "# subprocess.run(pdal_merge_command)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9bd53f-151f-4372-a350-b0508ebdd129",
   "metadata": {},
   "source": [
    "### Clip\n",
    "Crop file to outline of the road shapefile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "e4dff30f-58d1-4fde-a8f7-fbc43e6d52c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name JSON file\n",
    "clip_json = json_base_path +'clip_to_polygon.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "5b4ea3b1-6f9b-48a4-8d72-f8aa1c920f83",
   "metadata": {},
   "outputs": [],
   "source": [
    "# path_to_shapefile = 'SCB/supporting_files/bounding_box/hwy89_poly.shp'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "4fa11d72-f70c-41b5-9fde-1e6b9d78ebf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # extract wkt from hwy 89 polygon\n",
    "# hwy89 = shapefile.Reader(path_to_shapefile)\n",
    "# geom=[]\n",
    "# for s in hwy89.shapes():\n",
    "#     geom.append(pygeoif.geometry.as_shape(s)) \n",
    "# poly_base = pygeoif.MultiPolygon(geom)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "2c812914-fe1f-4e42-ac6b-9d536fcb52c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # create a pipeline and save to a json file \n",
    "# filter_crop = {'type':'filters.crop',\n",
    "#                'polygon':poly_base.wkt}\n",
    "# # write merged las to raster\n",
    "# writers_gdal = {\"type\":\"writers.text\",\n",
    "#                 \"format\":\"csv\",\n",
    "#                 \"order\":\"Z\",\n",
    "#                 'keep_unspecified':False}\n",
    "# pipeline_list = [filter_crop]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(clip_json, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a918a49-39f5-4e69-a70c-02644904a3bc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "debc6066-e045-4733-893a-77aef7147dc5",
   "metadata": {},
   "source": [
    "**Single Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cfa7245-70b0-455c-9436-eeeb1d8a8a70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# writer_cmd = '--writers.text.filename='+output_txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d297c8c-9339-4935-8866-79cc0179f9a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pdal_commands = ['pdal', 'translate', input_las, '--json', clip_json,writer_cmd]\n",
    "# subprocess.run(pdal_commands)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eee2a61-16b3-495c-910d-7a473cdf91dd",
   "metadata": {},
   "source": [
    "### Combined Pipeline Method\n",
    "Note that this example uses points instead of polygons. Alter accordingly. <br>\n",
    "target_point should look something like \"POINT(easting northing)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48bedb2e-f8ac-430c-8f3b-ceb0dc365b2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "clip_json = json_base_path +'extract_las_atPoint.json'\n",
    "filenames, tags = lidar_functions.create_command_template(input_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "762d6fc0-dac4-4cd0-b988-ca6a1a2a0f1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_las_atPoint(filenames, tags, target_point, output_txt,distance):\n",
    "    # merge all las files or stages\n",
    "    filter_merge = {\"type\":\"filters.merge\",\n",
    "                    \"tag\": \"merged\",\n",
    "                    \"inputs\": tags}\n",
    "    # crop\n",
    "    filter_crop = {'type':'filters.crop',\n",
    "                   'point':target_point,\n",
    "                   'distance':distance,\n",
    "                   'inputs':'merged',\n",
    "                   'tag': 'cropped'}\n",
    "    # write merged las to raster\n",
    "    writers_gdal = {\"type\":\"writers.text\",\n",
    "                    \"format\":\"csv\",\n",
    "                    \"order\":\"Z\",\n",
    "                    \"write_header\":False,\n",
    "                    'keep_unspecified':False,\n",
    "                    'filename':output_txt}\n",
    "    # Append each stage to a list prior to saving to json \n",
    "    pipeline_list = filenames.copy()\n",
    "    pipeline_list.append(filter_merge)\n",
    "    pipeline_list.append(filter_crop)\n",
    "    pipeline_list.append(writers_gdal)\n",
    "\n",
    "    pipeline_dict = {'pipeline' : pipeline_list}\n",
    "    # save to json\n",
    "    with open(extract_las_atPoint_json, 'w') as out:\n",
    "        json.dump(pipeline_dict, out, indent=4)\n",
    "    pdal_cmd = ['pdal','pipeline', extract_las_atPoint_json]\n",
    "    subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0624558b-84f7-41c6-90ad-420c4354ec39",
   "metadata": {},
   "source": [
    "## Calculate Stats Over Control Area\n",
    "In Kostadinov et al., 2019 the vertical bias was corrected using the lowest 10th percentile value between the snow on and snow off flights above the road. We will calculate a number of statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff93bcd6-1f96-4a80-8d7a-ba68c5f79c54",
   "metadata": {},
   "source": [
    "### Calculate Stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "6dfe977b-b130-447f-a776-6a97a1aa9836",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # input_las - HAG, las file clipped to the road\n",
    "# # output_path - path to output files\n",
    "# # base_name - string, typically flight name\n",
    "# def calculate_vertical_bias(input_las):\n",
    "#     # convert height only to txt file\n",
    "#     output_las_txt = input_las[:-3]+'csv'\n",
    "#     txt_cmd = ['pdal', 'translate', input_las, output_las_txt, '-w', 'writers.text', '--writers.text.format=csv','--writers.text.order=Z', '--writers.text.keep_unspecified=false']\n",
    "#     subprocess.run(txt_cmd)\n",
    "#     # calculate stats\n",
    "#     hag_arr = np.loadtxt(output_las_txt,skiprows=1)\n",
    "#     lowest_10th_per = np.nanpercentile(hag_arr, 10)\n",
    "#     mean_hag = np.nanmean(hag_arr)\n",
    "#     median_hag = np.nanmedian(hag_arr)\n",
    "#     stats = [\"lowest_10th\",lowest_10th_per, \"mean\",mean_hag, \"median\", median_hag]\n",
    "#     return(stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8d4de4-992b-45d8-8595-85fa88beaa84",
   "metadata": {},
   "source": [
    "## Correct Z Values\n",
    "Using PDAL filters.assign we can add a value to each lidar point\n",
    "See lidar_functions.py - correct_by-target_val <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1828ebe0-561f-4d93-a709-e1299cf41897",
   "metadata": {},
   "source": [
    "### Correct LAS Only\n",
    "This involves simple command line function operations instead of a json derived pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b50f9689-0fe8-4948-a617-a99e1e15bf4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_lid = 'path/to/input/filename/filename.laz'\n",
    "# output_lid = 'path/to/output/file/filename.laz'\n",
    "# filters_assign = '--filters.assign.value=Z=Z'+target_val\n",
    "# assign_cmd = ['pdal', 'translate', input_lid, output_lid, 'assign' ,filters_assign]\n",
    "# subprocess.run(assign_cmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23069994-84d1-4775-901d-d5de2df658e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # or use function\n",
    "# # assign target val\n",
    "# correct_by_targetVal_pipeline(target_val)\n",
    "# full_input_path = 'path/to/input/filename/filename.laz'\n",
    "# full_output_path = 'path/to/output/file/filename.laz'\n",
    "# target_val = 0\n",
    "# lidar_functions.correct_by_targetVal(full_input_path, full_output_path, target_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5863e1-f95b-42f5-8c5a-b23f924c3fc1",
   "metadata": {},
   "source": [
    "**Parallelize**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d40b353a-ffdd-4e67-bd9a-a54e5574ce16",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/lid/folder/'\n",
    "# output_path = 'path/to/lid/folder/'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_path = [input_path + s for s in onlyfiles]\n",
    "#         output_path_full = [output_path + s for s in onlyfiles]\n",
    "#         target_vals = np.repeat(target_val, len(full_path))\n",
    "#         executor.map(lidar_functions.correct_by_targetVal, full_path, output_path_full, target_vals) \n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c950fb1-51e9-437a-8d3d-059af9b1b270",
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.perf_counter()\n",
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/HAG/'\n",
    "output_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/corrected_las/'\n",
    "target_val = 'Z+'+str(abs(NCALM_2014_hwy89_stats[1])) # lowest 10th percentile \n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "        full_path = [input_path + s for s in onlyfiles]\n",
    "        output_path_full = [output_path + s for s in onlyfiles]\n",
    "        target_vals = np.repeat(target_val, len(full_path))\n",
    "        executor.map(lidar_functions.correct_by_targetVal, full_path, output_path_full, target_vals)\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af023f2f-1e3d-455e-a4c3-9f562731dcb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_functions.correct_by_targetVal(full_path[0], output_path_full[0], target_vals[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb12df9-d815-4e2c-88ee-fbcdb20c380f",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_functions.correct_by_targetVal(full_path[0], output_path_full[0], target_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec00e0b9-a256-42f4-9089-58392a8d83b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "filters_assign = '--filters.assign.value=Z=Z+0.08'\n",
    "assign_cmd = ['pdal', 'translate', full_path[0], output_path_full[0], 'assign' ,filters_assign]\n",
    "subprocess.run(assign_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "547b13e5-3b84-454b-8ba7-e42bf62bac03",
   "metadata": {},
   "source": [
    "### Correct LAS and Rasterize\n",
    "For this section, we overwrite the json for each flight depending on the target value. This is beneficial for saving memory (no redundant corrected las files created)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "b52dfa85-c720-4415-9614-59a111300465",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_rasterize_json = json_base_path + 'correct_by_targetVal_rasterize.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "4591310b-9cc2-4c4b-86d3-60f8c5b4b9b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_merge_rasterize_json = json_base_path + 'correct_merge_rasterize.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdd4957a-4782-453d-8aea-8ac91e44b808",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note that the corrected Z is Z - target val, because we are using the height above ground\n",
    "# input: target_assign - str e.g 'Z+0.08'\n",
    "def correct_by_targetVal_pipeline(target_assign):\n",
    "    readers_las = {'type':'readers.las'}\n",
    "    filters_assign = {'type': 'filters.assign',\n",
    "                      'value':\"Z=\"+target_assign}\n",
    "    writers_gdal= {\"type\": \"writers.gdal\",\n",
    "                  'output_type': 'mean',\n",
    "                  'resolution': '1.0',\n",
    "                  'radius': '0.7'}\n",
    "    pipeline_list = [readers_las,filters_assign,writers_gdal]\n",
    "    pipeline_dict = {'pipeline' : pipeline_list}\n",
    "    with open(correct_rasterize_json, 'w') as out:\n",
    "        json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52921924-7680-49af-afe2-98dcaeb453db",
   "metadata": {},
   "source": [
    "**Single Files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c61f447e-8a36-4ae0-b619-6af3a3c3a1ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# full_input_path = 'path/to/input/lid/filename.las'\n",
    "# full_output_path = 'path/to/output/lid/filename.las'\n",
    "# target_val = -9999\n",
    "# correct_by_targetVal_pipeline(target_val)\n",
    "# lidar_functions.correct_by_targetVal_rasterize(full_input_path, full_output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84f9983-f55b-4bfa-84f0-481e3d134207",
   "metadata": {},
   "source": [
    "**Parallel Processing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b18cc446-73fd-4dc9-a461-0cd82f76b23a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/HAG/'\n",
    "# output_path = 'path/to/output/folder/'\n",
    "# target_val = ASO_20160518_hwy89_stats[1] # lowest 10th percentile \n",
    "# correct_by_targetVal_pipeline(target_val)\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_path = [input_path + s for s in onlyfiles]\n",
    "#         output_path_full = [output_path + s[:-3] +'tif' for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.correct_by_targetVal_rasterize, full_path, output_path_full)\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce6ad7c3-e05e-4519-8eb7-b85c6e875494",
   "metadata": {},
   "source": [
    "#### Combined Pipeline Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9215f711-9e6d-47eb-aac6-238009f4f64e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_path = 'path/to/lidar/files/'\n",
    "# output_tif = 'path/to/output/tif/filename.tif'\n",
    "# onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))] # make a list of all filenames in directory\n",
    "# input_list = [input_path + s for s in onlyfiles] # make a list of full filename paths in directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98604b17-612a-48f3-8639-fa9adb192ffd",
   "metadata": {},
   "source": [
    "Create the reader stages of the pipeline. Each file is read as an individual reader stage here, and we cheat here by copying the formatting of json files and creating a dictionary with the values as the correctly formatted inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aced1c92-e72c-4a87-8012-f9851216ebd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filename_dict = {} # initiate an empty dict to hold the readers\n",
    "# tags = ['']*len(input_list) # initiate an empty list, size = number of files\n",
    "# filenames = ['']*len(input_list) # repeat\n",
    "# for i in range(len(input_list)): # for each file, create a dictionary element with the values matching json formatting for file merging\n",
    "#     filename_dict['filename_'+str(i)] = {'filename':input_list[i], 'tag':'A_'+str(i)}\n",
    "#     tags[i] = 'A_'+str(i) # add a tag to the reader stage\n",
    "#     filenames[i] = filename_dict[list(filename_dict)[i]] # Add all values to a list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae96d29-171d-4f74-8b2a-01139e417f56",
   "metadata": {},
   "source": [
    "Define the filter and writer stages of the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "9941bad6-5093-43aa-a09a-e28352921470",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # merge all las files or stages\n",
    "# filter_merge = {\"type\":\"filters.merge\",\n",
    "#                \"tag\": \"merged\",\n",
    "#                \"inputs\": tags}\n",
    "# # filter out the ground points of the tiles\n",
    "# filter_assign = {'type': 'filters.assign',\n",
    "#                  'value':\"Z=\"+target_assign,\n",
    "#                  'inputs':'merged',\n",
    "#                  'tag':'corrected'}\n",
    "# # write merged las to raster\n",
    "# writers_gdal= {\"type\": \"writers.gdal\",\n",
    "#                'output_type': 'mean',\n",
    "#               'resolution': 1.0,\n",
    "#               'radius': '0.7',\n",
    "#                'bounds': writers_bounds,\n",
    "#                'inputs': 'corrected',\n",
    "#                'filename':output_tif}\n",
    "# # Append each stage to a list prior to saving to json \n",
    "# pipeline_list = filenames.copy()\n",
    "# pipeline_list.append(filter_merge)\n",
    "# pipeline_list.append(filter_assign)\n",
    "# pipeline_list.append(writers_gdal)\n",
    "\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# # save to json\n",
    "# with open(correct_merge_rasterize_json, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "95e96e70-2a76-470e-9368-31f03d583517",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# pdal_cmd = ['pdal','pipeline', correct_merge_rasterize_json]\n",
    "# subprocess.run(pdal_cmd)\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59be1457-619a-4d53-91f8-ebc5c07873ed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc2a1b9a-07b9-4010-b9a7-3b6c283f472d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0d1051c8-91d9-4f5a-ac1f-264802882b41",
   "metadata": {},
   "source": [
    "# Calculate Canopy Metrics\n",
    "\n",
    "\n",
    "Vegetation Height Strata:<br>\n",
    "[-0.15:0.15), used to determine open sites<br>\n",
    "[0.15:1.5), if there are returns in this region, we can look into weather there is short vegetation or understory here<br>\n",
    "[1.5:3), we remove all pixels with returns here, classified as \"low branches or significant shrub/grass/ground-veg\"<br>\n",
    "[3:), this is our threshold for \"tall\" vegetation<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6275079d-a4c5-4b7d-a485-886951cc76d2",
   "metadata": {},
   "source": [
    "## Vegetation Height Strata (counts)\n",
    "Determine the number of returns (counts) of each height strata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "816196e4-8094-427d-a40e-48e1ef5e742e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function allows us to change the json pipeline depending on the files\n",
    "# the final output of a subprocess that uses this pipeline will be one raster with 1x1m pixels, each containing the number of returns (counts) of the specified lidar height strata\n",
    "# e.g. if the range_limit is Z[-0.15:0.15), each pixel will give us the number of returns between -0.15 and 0.15 in elevation (HAG in this case)\n",
    "# The goal of this function is to return a json pipeline with specific Z limits\n",
    "# input: filenames - see: lidar_functions.create_command_template\n",
    "# input: target_assign - if you need to perform a vertical bias correction on the snow-off flight, insert a string with the correction function (e.g. 'Z+0.05' if you need to add 0.05 to each lidar point), if no correction is necessary\n",
    "# insert \"Z+0\" [str]\n",
    "# input: writers_bounds - '([minx, maxx], [miny, maxy])';  e.g.'([730235.96, 738826.45], [4364741.66, 4372273.16])' [str]\n",
    "# input: range_limits: Z limits for output file\n",
    "def create_vegetation_heigh_strata(filenames, tags, target_assign, writers_bounds, range_limits, output_tif): \n",
    "    # merge all las files or stages\n",
    "    filter_merge = {\"type\":\"filters.merge\",\n",
    "                   \"tag\": \"merged\",\n",
    "                   \"inputs\": tags}\n",
    "    # bias correct files\n",
    "    filter_assign = {'type': 'filters.assign',\n",
    "                     'value':\"Z=\"+target_assign,\n",
    "                     'inputs':'merged',\n",
    "                     'tag':'corrected'}\n",
    "    # filter out the ground points of the tiles\n",
    "    filter_range = {\"type\":\"filters.range\",\n",
    "                    \"limits\":range_limits,\n",
    "                   'inputs':'merged',\n",
    "                   'tag':'filtered'}\n",
    "    # write merged las to raster\n",
    "    writers_gdal= {\"type\": \"writers.gdal\",\n",
    "                   'output_type': 'count',\n",
    "                  'resolution': '1.0',\n",
    "                  'radius': '0.7',\n",
    "                   'bounds': writers_bounds,\n",
    "                   'inputs': 'filtered',\n",
    "                   'filename':output_tif}\n",
    "    # Append each stage to a list prior to saving to json \n",
    "    pipeline_list = filenames.copy()\n",
    "    pipeline_list.append(filter_merge)\n",
    "    pipeline_list.append(filter_assign)\n",
    "    pipeline_list.append(filter_range)\n",
    "    pipeline_list.append(writers_gdal)\n",
    "\n",
    "    pipeline_dict = {'pipeline' : pipeline_list}\n",
    "    return(pipeline_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "205d6fb9-63f6-43f6-94cc-938caf1d2ed9",
   "metadata": {},
   "source": [
    "Constant inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de1f01f9-2bb4-41fb-8a60-3343b5f2c453",
   "metadata": {},
   "outputs": [],
   "source": [
    "# these inputs remains the same for all heigh strata\n",
    "input_path = 'path/to/HAG/files/'\n",
    "filenames, tags = lidar_functions.create_command_template(input_path)\n",
    "target_assign = \"Z+0\" # default\n",
    "writers_bounds = '([minx,maxx],[miny,maxy])'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2762db89-3788-4f9f-94bf-f42dc09fd896",
   "metadata": {},
   "source": [
    "Variable inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7af08b-e94e-42e7-8490-1eba6f992f0e",
   "metadata": {},
   "source": [
    "[-0.15:0.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fab9be2-575c-4afc-9d50-a678f7d2414d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify name of the json pipeline file\n",
    "filterMergeRasterize_json = json_base_path+'filterMergeRasterize_neg0pt15_0pt15.json'\n",
    "# specify output tif file path\n",
    "output_tif = 'pat/to/canopy_metric/folder/veg_height_strata/vegStrata_neg0pt15_0pt1.tif'\n",
    "# heigh strata range limits\n",
    "range_limits = 'Z[-0.15:0.15)'\n",
    "# create pipline\n",
    "pipeline_dict = create_vegetation_heigh_strata(filenames, tags, target_assign, writers_bounds, range_limits, output_tif)\n",
    "# save to json\n",
    "with open(filterMergeRasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n",
    "pdal_cmd = ['pdal','pipeline', filterMergeRasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4fd62e3-81eb-419e-aeb9-5e9167bdc213",
   "metadata": {},
   "source": [
    "[0.15:1.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "782f78e3-cd86-4d9c-8a4f-b67e36dc1da3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify name of the json pipeline file\n",
    "filterMergeRasterize_json = json_base_path+'filterMergeRasterize_0pt15_1pt5.json'\n",
    "# specify output tif file path\n",
    "output_tif = 'pat/to/canopy_metric/folder/veg_height_strata/vegStrata_0pt15_1pt5.tif'\n",
    "# heigh strata range limits\n",
    "range_limits = 'Z[0.15:1.5)'\n",
    "# create pipline\n",
    "pipeline_dict = create_vegetation_heigh_strata(filenames, tags, target_assign, writers_bounds, range_limits, output_tif)\n",
    "# save to json\n",
    "with open(filterMergeRasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n",
    "pdal_cmd = ['pdal','pipeline', filterMergeRasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c153cb3a-2aca-4d7c-aa9a-4741b03a199d",
   "metadata": {},
   "source": [
    "[1.5:3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da4cddad-8b23-45c1-97d7-6129a93730fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify name of the json pipeline file\n",
    "filterMergeRasterize_json = json_base_path+'filterMergeRasterize_1pt5_3.json'\n",
    "# specify output tif file path\n",
    "output_tif = 'pat/to/canopy_metric/folder/veg_height_strata/vegStrata_1pt5_3.tif'\n",
    "# heigh strata range limits\n",
    "range_limits = 'Z[1.5:3)'\n",
    "# create pipline\n",
    "pipeline_dict = create_vegetation_heigh_strata(filenames, tags, target_assign, writers_bounds, range_limits, output_tif)\n",
    "# save to json\n",
    "with open(filterMergeRasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n",
    "pdal_cmd = ['pdal','pipeline', filterMergeRasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5bb904-affa-4205-bec2-0ae2c0a695d8",
   "metadata": {},
   "source": [
    "[3:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8df286f-94e5-4c7a-8a41-f861b20dd7c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify name of the json pipeline file\n",
    "filterMergeRasterize_json = json_base_path+'filterMergeRasterize_3.json'\n",
    "# specify output tif file path\n",
    "output_tif = 'pat/to/canopy_metric/folder/veg_height_strata/vegStrata_3.tif'\n",
    "# heigh strata range limits\n",
    "range_limits = 'Z[1.5:3)'\n",
    "# create pipline\n",
    "pipeline_dict = create_vegetation_heigh_strata(filenames, tags, target_assign, writers_bounds, range_limits, output_tif)\n",
    "# save to json\n",
    "with open(filterMergeRasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n",
    "pdal_cmd = ['pdal','pipeline', filterMergeRasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a258e327-f92a-4f57-9d24-c216f1f89230",
   "metadata": {},
   "source": [
    "## Old Code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30dc3664-957a-4333-a940-c332d1df0427",
   "metadata": {},
   "source": [
    "[-0.15,0.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e64d4c4d-c510-45e1-9dc6-affc6e0a7bc6",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# range_json_neg0pt15_0pt15 = json_base_path+'filter_pts_neg0pt15_0pt15.json'\n",
    "# reader_dict = {'type':'readers.las'}\n",
    "# filter_range_neg0pt15_0pt15 = {\"type\":\"filters.range\",\n",
    "#                 \"limits\":\"Z[-0.15:0.15)\"}\n",
    "# writers_gdal_count= {\"type\": \"writers.gdal\",\n",
    "#               'output_type': 'count',\n",
    "#               'resolution': '1.0',\n",
    "#               'radius': '0.7',\n",
    "#               'window_size':2}\n",
    "# pipeline_list = [reader_dict,filter_range_neg0pt15_0pt15,writers_gdal_count]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(range_json_neg0pt15_0pt15, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfa12690-4ad8-43a2-a46e-046ccfac7adf",
   "metadata": {
    "tags": []
   },
   "source": [
    "[0.15,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9560754-966d-4adf-8976-d26d5a222a7a",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# range_json_0pt15_2 = json_base_path+'filter_pts_0pt15_2.json'\n",
    "# reader_dict = {'type':'readers.las'}\n",
    "# filter_range_0pt15_2 = {\"type\":\"filters.range\",\n",
    "#                 \"limits\":\"Z[0.15:2)\"}\n",
    "# writers_gdal_count= {\"type\": \"writers.gdal\",\n",
    "#               'output_type': 'count',\n",
    "#               'resolution': '1.0',\n",
    "#               'radius': '0.7'}\n",
    "# pipeline_list = [reader_dict,filter_range_0pt15_2,writers_gdal_count]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(range_json_0pt15_2, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e3a1eb2-2e16-4f3e-8cdc-fc41e6b897f2",
   "metadata": {},
   "source": [
    "[2,inf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d83f7a01-7547-4cf2-b995-f9f014ddb531",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# range_json_2 = json_base_path+'filter_pts_2.json'\n",
    "# reader_dict = {'type':'readers.las'}\n",
    "# filter_range_2 = {\"type\":\"filters.range\",\n",
    "#                 \"limits\":\"Z[2:)\"}\n",
    "# writers_gdal_count= {\"type\": \"writers.gdal\",\n",
    "#               'output_type': 'count',\n",
    "#               'resolution': '1.0',\n",
    "#               'radius': '0.7'}\n",
    "# pipeline_list = [reader_dict,filter_range_2,writers_gdal_count]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(range_json_2, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d82e9cdf-52e7-400a-b1cd-83f6545a1431",
   "metadata": {},
   "source": [
    "[2,inf) Ground"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba45046c-3404-4f71-9a5f-62254414138c",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# range_json_2_ground = json_base_path+'filter_pts_2_ground.json'\n",
    "# reader_dict = {'type':'readers.las'}\n",
    "# filter_range_2_ground = {\"type\":\"filters.range\",\n",
    "#                 \"limits\":\"Z[2:), Classification[2:2]\"}\n",
    "# writers_gdal= {\"type\": \"writers.gdal\",\n",
    "#               'output_type': 'count',\n",
    "#               'resolution': '1.0',\n",
    "#               'radius': '0.7'}\n",
    "# pipeline_list = [reader_dict,filter_range_2_ground,writers_gdal]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(range_json_2_ground, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75dadba-154f-4ba3-b09f-4b0395779208",
   "metadata": {},
   "source": [
    "[2,inf) Nonground"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5edae626-5fa5-48ac-98dd-87f69e8ac9eb",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# range_json_2_nonground = json_base_path+'filter_pts_2_nonground.json'\n",
    "# reader_dict = {'type':'readers.las'}\n",
    "# filter_range_2_nonground = {\"type\":\"filters.range\",\n",
    "#                 \"limits\":\"Z[2:), Classification![2:2]\"}\n",
    "# writers_gdal= {\"type\": \"writers.gdal\",\n",
    "#               'output_type': 'count',\n",
    "#               'resolution': '1.0',\n",
    "#               'radius': '0.7'}\n",
    "# pipeline_list = [reader_dict,filter_range_2_nonground,writers_gdal]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(range_json_2_nonground, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac9a6cab-1854-4a51-8215-5658dea5f656",
   "metadata": {},
   "source": [
    "**Single File**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52f21ed5-5c0c-4ff1-8efd-1b9e872b6951",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# input_las = 'path/to/input/lid/filename.las'\n",
    "# output_las = 'path/to/output/lid/filename.las'\n",
    "# strata_cmd = ['pdal', 'translate', input_las,  output_las, '--json',range_json]\n",
    "# subprocess.run(strata_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd4ea08f-915f-472e-b622-14923d5435ea",
   "metadata": {},
   "source": [
    "**All Files in Directory**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5edde846-02f4-4983-821a-c76e1db03cf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# pathname = \"path/to/lid/folder/\"\n",
    "# output_pathname = 'path/to/lid/folder/'\n",
    "# onlyfiles = [f for f in os.listdir(pathname) if os.path.isfile(os.path.join(pathname, f))]\n",
    "# for file in onlyfiles:\n",
    "#     input_las = pathname + file\n",
    "#     output_las = output_pathname + file\n",
    "#     strata_cmd = ['pdal', 'translate', input_las,  output_las, '--json',range_json]\n",
    "#     subprocess.run(strata_cmd)\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c5e266d-d3e1-460b-81af-27aeeef498c3",
   "metadata": {},
   "source": [
    "**Parallelization**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7157e690-5c90-4650-bc4a-4ea8f201a56c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tic = time.perf_counter()\n",
    "# input_path = 'path/to/lid/folder/'\n",
    "# output_path = 'path/to/lid/folder/'\n",
    "# if __name__ == '__main__':\n",
    "#     with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "#         onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "#         full_path = [input_path + s for s in onlyfiles]\n",
    "#         output_path_full = [output_path + s for s in onlyfiles]\n",
    "#         executor.map(lidar_functions.range_json, full_path, output_path_full) #running 10 times\n",
    "# toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb1fb84-4b06-4418-acf2-f871e4c3999f",
   "metadata": {},
   "source": [
    "### Clipping Geometries\n",
    "The goal of this portion of the code is to clip the raster based on the control areas (in the case of SCB, hwy 89)<br>\n",
    "See: https://pdal.io/tutorial/clipping/index.html#clipping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2abb94d-8dc7-43b6-9d5e-371d5cbbc903",
   "metadata": {},
   "outputs": [],
   "source": [
    "[\n",
    "    \"autzen.laz\",\n",
    "    {\n",
    "      \"type\":\"filters.overlay\",\n",
    "      \"dimension\":\"Classification\",\n",
    "      \"datasource\":\"attributes.vrt\",\n",
    "      \"layer\":\"OGRGeoJSON\",\n",
    "      \"column\":\"CLS\"\n",
    "    },\n",
    "    {\n",
    "      \"type\":\"filters.range\",\n",
    "      \"limits\":\"Classification[5:5]\"\n",
    "    },\n",
    "    \"output.las\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49b57bfb-5914-4deb-8a50-ef9d334f2d7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name JSON file\n",
    "clipping_json = 'lidar_processing/python_scripts/PDAL_workflow/JSON/clip_las_to_shp.json'\n",
    "\n",
    "filter_overlay_dict = {\"type\":\"filters.overlay\",\n",
    "                       \"dimension\":\"Classification\",\n",
    "                       \"datasource\":\"SCB/bounding_box/hwy89_poly.shp\",\n",
    "                       \"column\":\"OBJECTID\"}\n",
    "filter_range_dict = {\"type\":\"filters.range\",\n",
    "                     \"limits\":\"Classification[4193:4193]\"}\n",
    "\n",
    "pipeline_list = [filter_overlay_dict,filter_range_dict]\n",
    "with open(clipping_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b802830c-14f8-4a7c-be1d-194a0978bcc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "pathname = 'lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/reproject/20160518_toNAD83/toNAVD88/'\n",
    "output_pathname = \"lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/ground_filtered/ASO_20160518/reproj_NAD83_NAVD88/\"\n",
    "onlyfiles = [f for f in os.listdir(pathname) if os.path.isfile(os.path.join(pathname, f))]\n",
    "pdal_clip = ['pdal', 'translate', input_las, output_las, '--json', output_json]\n",
    "subprocess.run(pdal_clip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fab18f5-64b2-4b50-9196-d71283014eb3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32043e49-150b-45e4-a858-a0e8673d6e2a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa1f5130-70e8-42ce-bdd5-14dada2c3926",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad9ef7e0-3aa6-4957-a9f9-c5e87ffb963f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "095da6a0-6985-4ca5-ab31-04d988240e38",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "073fdbf1-8dce-44e1-aadc-dc77c8d83e7b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1f880c3-1dc3-4518-ae1f-9e66cb2db0a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bdf0361-228b-45e0-8ea9-2bf68931089d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96106e87-c47e-429a-aaf8-854ad90e8399",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "24c96368-34b1-4455-944b-32d48920503f",
   "metadata": {},
   "source": [
    "# ----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b1734df-2865-4acd-a92a-9613f8867044",
   "metadata": {},
   "source": [
    "# Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "600a92ec-7986-41ff-99d1-28daea0535a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define all json files\n",
    "filterMergeRasterize_json = json_base_path + 'filterMergeRasterize.json'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c371e470-c770-4b4d-93a4-52d882661c02",
   "metadata": {},
   "source": [
    "## Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019afb1c-f0f0-4e99-8072-cdabfaee5be6",
   "metadata": {},
   "source": [
    "### Info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "caff1251-3a50-4fec-9e91-4837b39954f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"file_size\": 33332101,\n",
      "  \"filename\": \"MRB/Merced_lidar/ASO/ASO_MRB_20210429/laz/CA_20210429_pN94S_f1_tiledlas_fl195341_ch1_ti-6720_tj-4500_ts60_v2_UTMZ11.laz\",\n",
      "  \"metadata\":\n",
      "  {\n",
      "    \"comp_spatialreference\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",GEOGCS[\\\"WGS 84\\\",DATUM[\\\"WGS_1984\\\",SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],PRIMEM[\\\"Greenwich\\\",0,AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],UNIT[\\\"degree\\\",0.0174532925199433,AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"latitude_of_origin\\\",0],PARAMETER[\\\"central_meridian\\\",-117],PARAMETER[\\\"scale_factor\\\",0.9996],PARAMETER[\\\"false_easting\\\",500000],PARAMETER[\\\"false_northing\\\",0],UNIT[\\\"metre\\\",1,AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],AXIS[\\\"Easting\\\",EAST],AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "    \"compressed\": true,\n",
      "    \"count\": 5712854,\n",
      "    \"creation_doy\": 76,\n",
      "    \"creation_year\": 2022,\n",
      "    \"dataformat_id\": 6,\n",
      "    \"dataoffset\": 1725,\n",
      "    \"filesource_id\": 40371,\n",
      "    \"global_encoding\": 16,\n",
      "    \"global_encoding_base64\": \"EAA=\",\n",
      "    \"gtiff\": \"\",\n",
      "    \"header_size\": 375,\n",
      "    \"major_version\": 1,\n",
      "    \"maxx\": 271791.5348,\n",
      "    \"maxy\": 4179878.538,\n",
      "    \"maxz\": 2527.83,\n",
      "    \"minor_version\": 4,\n",
      "    \"minx\": 268477.6988,\n",
      "    \"miny\": 4173754.72,\n",
      "    \"minz\": 1053.16,\n",
      "    \"offset_x\": 0,\n",
      "    \"offset_y\": 0,\n",
      "    \"offset_z\": 0,\n",
      "    \"point_length\": 30,\n",
      "    \"project_id\": \"00000000-0000-0000-0000-000000000000\",\n",
      "    \"scale_x\": 0.01,\n",
      "    \"scale_y\": 0.01,\n",
      "    \"scale_z\": 0.01,\n",
      "    \"software_id\": \"PDAL 2.1.0 (2861c2)\",\n",
      "    \"spatialreference\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",GEOGCS[\\\"WGS 84\\\",DATUM[\\\"WGS_1984\\\",SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],PRIMEM[\\\"Greenwich\\\",0,AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],UNIT[\\\"degree\\\",0.0174532925199433,AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"latitude_of_origin\\\",0],PARAMETER[\\\"central_meridian\\\",-117],PARAMETER[\\\"scale_factor\\\",0.9996],PARAMETER[\\\"false_easting\\\",500000],PARAMETER[\\\"false_northing\\\",0],UNIT[\\\"metre\\\",1,AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],AXIS[\\\"Easting\\\",EAST],AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "    \"srs\":\n",
      "    {\n",
      "      \"compoundwkt\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",GEOGCS[\\\"WGS 84\\\",DATUM[\\\"WGS_1984\\\",SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],PRIMEM[\\\"Greenwich\\\",0,AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],UNIT[\\\"degree\\\",0.0174532925199433,AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"latitude_of_origin\\\",0],PARAMETER[\\\"central_meridian\\\",-117],PARAMETER[\\\"scale_factor\\\",0.9996],PARAMETER[\\\"false_easting\\\",500000],PARAMETER[\\\"false_northing\\\",0],UNIT[\\\"metre\\\",1,AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],AXIS[\\\"Easting\\\",EAST],AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "      \"horizontal\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",GEOGCS[\\\"WGS 84\\\",DATUM[\\\"WGS_1984\\\",SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],PRIMEM[\\\"Greenwich\\\",0,AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],UNIT[\\\"degree\\\",0.0174532925199433,AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"latitude_of_origin\\\",0],PARAMETER[\\\"central_meridian\\\",-117],PARAMETER[\\\"scale_factor\\\",0.9996],PARAMETER[\\\"false_easting\\\",500000],PARAMETER[\\\"false_northing\\\",0],UNIT[\\\"metre\\\",1,AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],AXIS[\\\"Easting\\\",EAST],AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "      \"isgeocentric\": false,\n",
      "      \"isgeographic\": false,\n",
      "      \"prettycompoundwkt\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",\\n    GEOGCS[\\\"WGS 84\\\",\\n        DATUM[\\\"WGS_1984\\\",\\n            SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,\\n                AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],\\n            AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],\\n        PRIMEM[\\\"Greenwich\\\",0,\\n            AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],\\n        UNIT[\\\"degree\\\",0.0174532925199433,\\n            AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],\\n        AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],\\n    PROJECTION[\\\"Transverse_Mercator\\\"],\\n    PARAMETER[\\\"latitude_of_origin\\\",0],\\n    PARAMETER[\\\"central_meridian\\\",-117],\\n    PARAMETER[\\\"scale_factor\\\",0.9996],\\n    PARAMETER[\\\"false_easting\\\",500000],\\n    PARAMETER[\\\"false_northing\\\",0],\\n    UNIT[\\\"metre\\\",1,\\n        AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],\\n    AXIS[\\\"Easting\\\",EAST],\\n    AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "      \"prettywkt\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",\\n    GEOGCS[\\\"WGS 84\\\",\\n        DATUM[\\\"WGS_1984\\\",\\n            SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,\\n                AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],\\n            AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],\\n        PRIMEM[\\\"Greenwich\\\",0,\\n            AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],\\n        UNIT[\\\"degree\\\",0.0174532925199433,\\n            AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],\\n        AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],\\n    PROJECTION[\\\"Transverse_Mercator\\\"],\\n    PARAMETER[\\\"latitude_of_origin\\\",0],\\n    PARAMETER[\\\"central_meridian\\\",-117],\\n    PARAMETER[\\\"scale_factor\\\",0.9996],\\n    PARAMETER[\\\"false_easting\\\",500000],\\n    PARAMETER[\\\"false_northing\\\",0],\\n    UNIT[\\\"metre\\\",1,\\n        AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],\\n    AXIS[\\\"Easting\\\",EAST],\\n    AXIS[\\\"Northing\\\",NORTH]]\",\n",
      "      \"proj4\": \"+proj=utm +zone=11 +datum=WGS84 +units=m +no_defs\",\n",
      "      \"units\":\n",
      "      {\n",
      "        \"horizontal\": \"metre\",\n",
      "        \"vertical\": \"\"\n",
      "      },\n",
      "      \"vertical\": \"\",\n",
      "      \"wkt\": \"PROJCS[\\\"WGS 84 / UTM zone 11N\\\",GEOGCS[\\\"WGS 84\\\",DATUM[\\\"WGS_1984\\\",SPHEROID[\\\"WGS 84\\\",6378137,298.257223563,AUTHORITY[\\\"EPSG\\\",\\\"7030\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"6326\\\"]],PRIMEM[\\\"Greenwich\\\",0,AUTHORITY[\\\"EPSG\\\",\\\"8901\\\"]],UNIT[\\\"degree\\\",0.0174532925199433,AUTHORITY[\\\"EPSG\\\",\\\"9122\\\"]],AUTHORITY[\\\"EPSG\\\",\\\"4326\\\"]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"latitude_of_origin\\\",0],PARAMETER[\\\"central_meridian\\\",-117],PARAMETER[\\\"scale_factor\\\",0.9996],PARAMETER[\\\"false_easting\\\",500000],PARAMETER[\\\"false_northing\\\",0],UNIT[\\\"metre\\\",1,AUTHORITY[\\\"EPSG\\\",\\\"9001\\\"]],AXIS[\\\"Easting\\\",EAST],AXIS[\\\"Northing\\\",NORTH]]\"\n",
      "    },\n",
      "    \"system_id\": \"PDAL\",\n",
      "    \"vlr_0\":\n",
      "    {\n",
      "      \"data\": \"UFJPSkNTWyJXR1MgODQgLyBVVE0gem9uZSAxMU4iLEdFT0dDU1siV0dTIDg0IixEQVRVTVsiV0dTXzE5ODQiLFNQSEVST0lEWyJXR1MgODQiLDYzNzgxMzcsMjk4LjI1NzIyMzU2MyxBVVRIT1JJVFlbIkVQU0ciLCI3MDMwIl1dLEFVVEhPUklUWVsiRVBTRyIsIjYzMjYiXV0sUFJJTUVNWyJHcmVlbndpY2giLDAsQVVUSE9SSVRZWyJFUFNHIiwiODkwMSJdXSxVTklUWyJkZWdyZWUiLDAuMDE3NDUzMjkyNTE5OTQzMyxBVVRIT1JJVFlbIkVQU0ciLCI5MTIyIl1dLEFVVEhPUklUWVsiRVBTRyIsIjQzMjYiXV0sUFJPSkVDVElPTlsiVHJhbnN2ZXJzZV9NZXJjYXRvciJdLFBBUkFNRVRFUlsibGF0aXR1ZGVfb2Zfb3JpZ2luIiwwXSxQQVJBTUVURVJbImNlbnRyYWxfbWVyaWRpYW4iLC0xMTddLFBBUkFNRVRFUlsic2NhbGVfZmFjdG9yIiwwLjk5OTZdLFBBUkFNRVRFUlsiZmFsc2VfZWFzdGluZyIsNTAwMDAwXSxQQVJBTUVURVJbImZhbHNlX25vcnRoaW5nIiwwXSxVTklUWyJtZXRyZSIsMSxBVVRIT1JJVFlbIkVQU0ciLCI5MDAxIl1dLEFYSVNbIkVhc3RpbmciLEVBU1RdLEFYSVNbIk5vcnRoaW5nIixOT1JUSF1dAA==\",\n",
      "      \"description\": \"OGC Transformation Record\",\n",
      "      \"record_id\": 2112,\n",
      "      \"user_id\": \"LASF_Projection\"\n",
      "    },\n",
      "    \"vlr_1\":\n",
      "    {\n",
      "      \"data\": \"UFJPSkNTWyJXR1MgODQgLyBVVE0gem9uZSAxMU4iLEdFT0dDU1siV0dTIDg0IixEQVRVTVsiV0dTXzE5ODQiLFNQSEVST0lEWyJXR1MgODQiLDYzNzgxMzcsMjk4LjI1NzIyMzU2MyxBVVRIT1JJVFlbIkVQU0ciLCI3MDMwIl1dLEFVVEhPUklUWVsiRVBTRyIsIjYzMjYiXV0sUFJJTUVNWyJHcmVlbndpY2giLDAsQVVUSE9SSVRZWyJFUFNHIiwiODkwMSJdXSxVTklUWyJkZWdyZWUiLDAuMDE3NDUzMjkyNTE5OTQzMyxBVVRIT1JJVFlbIkVQU0ciLCI5MTIyIl1dLEFVVEhPUklUWVsiRVBTRyIsIjQzMjYiXV0sUFJPSkVDVElPTlsiVHJhbnN2ZXJzZV9NZXJjYXRvciJdLFBBUkFNRVRFUlsibGF0aXR1ZGVfb2Zfb3JpZ2luIiwwXSxQQVJBTUVURVJbImNlbnRyYWxfbWVyaWRpYW4iLC0xMTddLFBBUkFNRVRFUlsic2NhbGVfZmFjdG9yIiwwLjk5OTZdLFBBUkFNRVRFUlsiZmFsc2VfZWFzdGluZyIsNTAwMDAwXSxQQVJBTUVURVJbImZhbHNlX25vcnRoaW5nIiwwXSxVTklUWyJtZXRyZSIsMSxBVVRIT1JJVFlbIkVQU0ciLCI5MDAxIl1dLEFYSVNbIkVhc3RpbmciLEVBU1RdLEFYSVNbIk5vcnRoaW5nIixOT1JUSF1dAA==\",\n",
      "      \"description\": \"OGR variant of OpenGIS WKT SRS\",\n",
      "      \"record_id\": 2112,\n",
      "      \"user_id\": \"liblas\"\n",
      "    },\n",
      "    \"vlr_2\":\n",
      "    {\n",
      "      \"data\": \"AwAAAAMEAwAAAAAAUMMAAP////////////////////8BAAoAHgADAA==\",\n",
      "      \"description\": \"by laszip of LAStools (200131)\",\n",
      "      \"record_id\": 22204,\n",
      "      \"user_id\": \"laszip encoded\"\n",
      "    }\n",
      "  },\n",
      "  \"now\": \"2022-04-11T14:11:12-0700\",\n",
      "  \"pdal_version\": \"2.3.0 (git-version: Release)\",\n",
      "  \"reader\": \"readers.las\"\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['pdal', 'info', 'MRB/Merced_lidar/ASO/ASO_MRB_20210429/laz/CA_20210429_pN94S_f1_tiledlas_fl195341_ch1_ti-6720_tj-4500_ts60_v2_UTMZ11.laz', '--metadata'], returncode=0)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define input file\n",
    "input_lid = r'MRB/Merced_lidar/ASO/ASO_MRB_20210429/laz/CA_20210429_pN94S_f1_tiledlas_fl195341_ch1_ti-6720_tj-4500_ts60_v2_UTMZ11.laz'\n",
    "pdal_info_cmd = ['pdal','info',input_lid] # general info\n",
    "pdal_metadata_cmd = ['pdal','info',input_lid,'--metadata'] # full file metadata, including details crs\n",
    "subprocess.run(pdal_metadata_cmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "47ecd2f6-067f-4c0a-bba7-86442f1d6c7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdal_info_results = subprocess.run(pdal_metadata_cmd, stdout = subprocess.PIPE) # stout (standard out), PIPE indicates that a new pipe to the child should be created\n",
    "pdal_info_dict = json.loads(pdal_info_results.stdout.decode()) # create dict with metadata info\n",
    "#pdal_info_dict # print results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "25fb2bc9-743a-480a-9ce6-7838c8ca8be6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pdal_info_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd7cd5d-0ac8-419c-9d36-fa0589d65e04",
   "metadata": {},
   "source": [
    "Check Extents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "fcfa978a-0e82-4d69-81f3-fcc1a59e258c",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_folder = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/NAD83_NAD83_epoch2010/'\n",
    "# lidar_folder = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/NAD83_NAD83_epoch2010/'\n",
    "onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "full_paths = [lidar_folder +f for f in onlyfiles]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "e0990454-2c2d-4b1b-906a-0561933b7954",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NCALM_2014_ext = lidar_functions.check_flight_extent(full_paths)\n",
    "# ASO_SCB_20160326_ext = lidar_functions.check_flight_extent(full_paths)\n",
    "# ASO_SCB_20160417_ext = lidar_functions.check_flight_extent(full_paths)\n",
    "ASO_SCB_20160518_ext = lidar_functions.check_flight_extent(full_paths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "8516d352-8076-4ecd-89bd-2ca235b8ba93",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[730235.96, 738826.45, 4364741.66, 4372273.16, 8590.48999999999, 7531.5]\n"
     ]
    }
   ],
   "source": [
    "# print(NCALM_2014_ext)\n",
    "# print(ASO_SCB_20160326_ext)\n",
    "# print(ASO_SCB_20160417_ext)\n",
    "print(ASO_SCB_20160518_ext)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4751d94d-d53d-40ce-a8d4-d62adad47b08",
   "metadata": {},
   "source": [
    "### Retile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea4032e7-3646-4f0e-9d5e-acc98c7ea589",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_folder = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/NAD83_NAD83_epoch2010/'\n",
    "retile_folder = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/retile_uo/'\n",
    "onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "for files in onlyfiles:\n",
    "    full_path = os.path.join(lidar_folder, files)\n",
    "    output_path = retile_folder+'#' + files\n",
    "    retile_command = ['pdal', 'tile', full_path, output_path, '--length=1000','--buffer=50']\n",
    "    subprocess.run(retile_command)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26292a6b-d7e1-4671-8a2c-81b27432469c",
   "metadata": {},
   "source": [
    "### Rename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd725789-60cb-4aa6-b79f-dfca7e1ff3f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "lidar_folder = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/NAD83_NAD83_epoch2010/'\n",
    "tic = time.perf_counter()\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "        full_path = [os.path.join(lidar_folder, s) for s in onlyfiles]\n",
    "        executor.map(lidar_functions.rename_llx_lly_b, full_path) #running 10 times\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a97d2d1-e28e-4bed-a4ba-b0363d4434a4",
   "metadata": {},
   "source": [
    "### Save Tile Boundaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbace8bd-7da0-4913-9960-c65f1eca84cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# one folder\n",
    "tic = time.perf_counter()\n",
    "lidar_folder = 'MRB/Merced_lidar/ASO/ASO_MRB_20210429/laz/'\n",
    "output_folder = 'MRB/Merced_lidar/ASO/ASO_MRB_20210429/tindex/original/'\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "        full_path = [os.path.join(lidar_folder, s) for s in onlyfiles]\n",
    "        # change depending on directory formats\n",
    "        #output_path = [os.path.join(os.path.basename(lidar_folder),'tindex/tiles/' + s[:-3] + 'sqlite') for s in onlyfiles]\n",
    "        output_path = [output_folder + s[:-3] + 'sqlite' for s in onlyfiles]\n",
    "        executor.map(lidar_functions.create_tindex, full_path, output_path) #running 10 times\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59151d2-faa6-445f-b84a-86ecb4fa60fe",
   "metadata": {},
   "source": [
    "### Copy Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4528ffe9-6116-491e-967c-7fbfb1fdbade",
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.perf_counter()\n",
    "lidar_folder = 'MRB/Merced_lidar/ASO/ASO_MRB_20210429/laz/'\n",
    "output_path = 'MRB/Merced_lidar/ASO/ASO_MRB_20210429/ICB_tiles/'\n",
    "onlyfiles = [f for f in os.listdir(lidar_folder) if os.path.isfile(os.path.join(lidar_folder, f))]\n",
    "full_paths = [os.path.join(lidar_folder, s) for s in onlyfiles] \n",
    "output_paths = np.repeat(output_path, len(full_paths))\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        executor.map(lidar_functions.copy_lid_by_ext_ICB, full_paths, output_paths)\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a8934cc-5009-45a1-9b5b-852df2c3a908",
   "metadata": {},
   "source": [
    "## Create DEM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c05e3051-45c4-4ab1-8076-68fe36d1e098",
   "metadata": {},
   "outputs": [],
   "source": [
    "DEM_json = json_base_path+'DEM_from_las.json' # define path to json files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "cd7b5912-904f-4064-9a78-32ac0ad932aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/NAD83_NAD83_epoch2010/'\n",
    "output_tif = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/DEM/NCALM_2014_BE.tif'\n",
    "onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "input_list = [input_path + s for s in onlyfiles]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0850ec42-97c2-4944-88b4-256f8d4dbe51",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_dict = {}\n",
    "tags = ['']*len(input_list)\n",
    "filenames = ['']*len(input_list)\n",
    "for i in range(len(input_list)):\n",
    "    filename_dict['filename_'+str(i)] = {'filename':input_list[i], 'tag':'A_'+str(i)}\n",
    "    tags[i] = 'A_'+str(i)\n",
    "    filenames[i] = filename_dict[list(filename_dict)[i]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b114ffa3-9aef-4a9c-91cb-0528a5a27d9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter out the ground points of the tiles\n",
    "filter_range = {\"type\":\"filters.range\",\n",
    "                \"limits\":\"Classification[2:2]\"}\n",
    "# merge all las files or stages\n",
    "filter_merge = {\"type\":\"filters.merge\",\n",
    "               \"tag\": \"merged\",\n",
    "               \"inputs\": tags}\n",
    "# write merged las to raster\n",
    "writers_gdal= {\"type\": \"writers.gdal\",\n",
    "               'output_type': 'mean',\n",
    "              'resolution': '1.0',\n",
    "              'radius': '0.7',\n",
    "               'window_size':3,\n",
    "               'filename': output_tif}\n",
    "# Append each stage to a list prior to saving to json \n",
    "pipeline_list = filenames.copy()\n",
    "pipeline_list.append(filter_merge)\n",
    "pipeline_list.append(filter_range)\n",
    "pipeline_list.append(writers_gdal)\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# save to json\n",
    "with open(DEM_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c1324e2-3971-44fc-a2f8-10f60ab76ab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.perf_counter()\n",
    "pdal_cmd = ['pdal','pipeline', DEM_json]\n",
    "subprocess.run(pdal_cmd)\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13399307-5a71-472d-9039-262c703e83e7",
   "metadata": {},
   "source": [
    "### Heigh Above Ground"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8c1511e-856a-426e-9794-75b704b146bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# parallel processing\n",
    "# time = 1.1 min\n",
    "tic = time.perf_counter()\n",
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/NAD83_NAD83_epoch2010/'\n",
    "output_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/HAG/'\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "        full_path = [input_path + s for s in onlyfiles]\n",
    "        output_path_full = [output_path  + s for s in onlyfiles]\n",
    "        executor.map(lidar_functions.HAG_dem, full_path, output_path_full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "8be62742-e0d1-4f5b-b0ff-0632b0286250",
   "metadata": {},
   "outputs": [],
   "source": [
    "# parallel processing\n",
    "# time = 1.1 min\n",
    "tic = time.perf_counter()\n",
    "input_path = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160417/NAD83_NAD83_epoch2010/'\n",
    "output_path = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160417/HAG/'\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "        full_path = [input_path + s for s in onlyfiles]\n",
    "        output_path_full = [output_path  + s for s in onlyfiles]\n",
    "        executor.map(lidar_functions.HAG_dem, full_path, output_path_full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "a6b9822c-9db8-4132-b641-06605deb1ba3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# parallel processing\n",
    "# time = 1.1 min\n",
    "tic = time.perf_counter()\n",
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/NAD83_NAD83_epoch2010/'\n",
    "output_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/HAG/'\n",
    "if __name__ == '__main__':\n",
    "    with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:\n",
    "        onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]\n",
    "        full_path = [input_path + s for s in onlyfiles]\n",
    "        output_path_full = [output_path  + s for s in onlyfiles]\n",
    "        executor.map(lidar_functions.HAG_dem, full_path, output_path_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad2a53a-72cd-4fe7-8c9f-fe68102cdbc5",
   "metadata": {},
   "source": [
    "## Vertical Bias Correction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "926a5b5b-e820-473b-8016-c6844b1d40a0",
   "metadata": {},
   "source": [
    "### Move Files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "061a1496-380c-494f-8338-9db6fe277b29",
   "metadata": {},
   "source": [
    "Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "f5818571-a968-43f1-9514-6ef591da593a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining source and destination\n",
    "# paths\n",
    "src = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160417/HAG/*.la*'\n",
    "trg = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160417/hwy89_vertical_bias/target_lid'\n",
    "file_paths = glob.glob(src)\n",
    "\n",
    "for files in file_paths:\n",
    "    filename = os.path.basename(files)\n",
    "    if int(files[-18:-12]) > 737000 and int(files[-11:-4]) > 4370000:\n",
    "    # copying the files to the\n",
    "    # destination directory\n",
    "        shutil.copy2(files, trg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "29e6e743-7858-49c8-ac0f-6975603ee308",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining source and destination\n",
    "# paths\n",
    "src = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/HAG/*.la*'\n",
    "trg = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/hwy89_vertical_bias/target_lid'\n",
    "file_paths = glob.glob(src)\n",
    "\n",
    "for files in file_paths:\n",
    "    filename = os.path.basename(files)\n",
    "    if int(files[-18:-12]) > 737000 and int(files[-11:-4]) > 4370000:\n",
    "    # copying the files to the\n",
    "    # destination directory\n",
    "        shutil.copy2(files, trg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c18a7b-8898-45fb-bc28-af0a5b5dca70",
   "metadata": {},
   "source": [
    "**NCALM 2008**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "9879ea73-5d96-4aa1-8360-415f8d00e9b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extents extracted from QGIS\n",
    "min_x = 731945\n",
    "max_x = 734327\n",
    "min_y = 4368517\n",
    "max_y = 4375256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "b102e611-851a-4749-8a7f-4ec0ec1f3094",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining source and destination\n",
    "# paths\n",
    "src = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/HAG/*.la*'\n",
    "trg = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/target_lid'\n",
    "file_paths = glob.glob(src)\n",
    "\n",
    "for files in file_paths:\n",
    "    filename = os.path.basename(files)\n",
    "    if int(files[-18:-12]) < max_x and int(files[-11:-4]) > min_y:\n",
    "    # copying the files to the\n",
    "    # destination directory\n",
    "        shutil.copy2(files, trg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8451906-bed0-43fb-be02-7a1f242dc0e0",
   "metadata": {},
   "source": [
    "### Merge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "78b38fda-766e-48b1-847f-94ae5a1c585c",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160326/hwy89_vertical_bias/target_lid/*.la*' # define path of input files\n",
    "input_fname = glob.glob(input_path) # save to list\n",
    "output_fname = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160326/hwy89_vertical_bias/target_lid/ASO_SCB_20160326_hwy89_merge.'+input_fname[0][-3:]# set output filename\n",
    "pdal_merge_command = input_fname\n",
    "pdal_merge_command.insert(len(pdal_merge_command),output_fname) # insert output file to list\n",
    "#pdal_merge_command.insert(0,'-f')\n",
    "pdal_merge_command.insert(0,'merge')\n",
    "pdal_merge_command.insert(0,'pdal')\n",
    "subprocess.run(pdal_merge_command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "dd4a9e5b-323e-48ad-8d07-077e16ffcc08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/target_lid/*.la*' # define path of input files\n",
    "# input_fname = glob.glob(input_path) # save to list\n",
    "# output_fname = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/target_lid/NCALM_SCB_20080210_merge.'+input_fname[0][-3:]# set output filename\n",
    "# pdal_merge_command = input_fname\n",
    "# pdal_merge_command.insert(len(pdal_merge_command),output_fname) # insert output file to list\n",
    "# #pdal_merge_command.insert(0,'-f')\n",
    "# pdal_merge_command.insert(0,'merge')\n",
    "# pdal_merge_command.insert(0,'pdal')\n",
    "# subprocess.run(pdal_merge_command)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebdd7829-317e-42c3-8f2d-9a882d93c793",
   "metadata": {},
   "source": [
    "### Clip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35d800a-8876-447f-86a7-89e8c4997fc8",
   "metadata": {},
   "source": [
    "#### ASO 2016"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86122500-4ae9-445b-a3af-005a4c4ff601",
   "metadata": {},
   "source": [
    "Applied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "71402413-e728-42b6-91cb-a000371da216",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_las = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/hwy89_vertical_bias/target_lid/ASO_SCB_20160518_hwy89_merge.las'\n",
    "output_txt = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/hwy89_vertical_bias/clipped/ASO_SCB_20160518_hwy89_clip.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "9d09b842-a504-4bab-8c81-8af8d202f3c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdal_commands = ['pdal', 'translate', input_las, output_txt, '--json', clip_json]\n",
    "subprocess.run(pdal_commands)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5974506f-0990-414e-98db-ad79a9eb29e4",
   "metadata": {},
   "source": [
    "#### NCALM 2008"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "c7551b5d-a8a1-40eb-84f5-25e940002bc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name JSON file\n",
    "extract_las_atPoint = json_base_path +'extract_las_atPoint.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "id": "cd1c8ed4-08d7-4172-845c-ed078758a861",
   "metadata": {},
   "outputs": [],
   "source": [
    "# path_to_shapefile = 'SCB/supporting_files/Harpold_data_paper/snotel_20080210_poly_b.shp'\n",
    "# # extract wkt from hwy 89 polygon\n",
    "# snotel_pts = shapefile.Reader(path_to_shapefile)\n",
    "# snotel_pts_data = snotel_pts.records()\n",
    "# geom=[]\n",
    "# for s in snotel_pts.shapes():\n",
    "#     geom.append(pygeoif.geometry.as_shape(s)) \n",
    "# poly_base = pygeoif.MultiPolygon(geom)\n",
    "# snote_pts = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "56467508-027c-4fda-b448-bcbed112e845",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in ground_truthed data\n",
    "def csv_to_list(input_csv):\n",
    "    src = open(input_csv)\n",
    "    csvreader = csv.reader(src)\n",
    "    header = next(csvreader)\n",
    "    output_list = []\n",
    "    for row in csvreader:\n",
    "        output_list.append(row)\n",
    "    src.close()\n",
    "    return(header, output_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "dc0026e0-8c6f-44fe-b339-aa695bba261e",
   "metadata": {},
   "outputs": [],
   "source": [
    "SNOTEL_src = \"SCB/supporting_files/Harpold_data_paper/snowdepth_filtered.csv\"\n",
    "snotel_header, snotel_data = csv_to_list(SNOTEL_src)\n",
    "hunt_src = 'SCB/supporting_files/Huntingon_2008_snow/Hungington_2008_snow.csv'\n",
    "hunt_header, hunt_data = csv_to_list(hunt_src)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "56707ce6-5465-4cd1-8021-eb0d6a245b4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Independence Creek',\n",
       "  '540',\n",
       "  '39.490053',\n",
       "  '-120.282302',\n",
       "  '131.1666667',\n",
       "  '733715',\n",
       "  '4374687'],\n",
       " ['Independence Camp',\n",
       "  '539',\n",
       "  '39.452701',\n",
       "  '-120.29369',\n",
       "  '140.0416667',\n",
       "  '732860',\n",
       "  '4370511'],\n",
       " ['Independence Lake',\n",
       "  '541',\n",
       "  '39.427512',\n",
       "  '-120.313362',\n",
       "  '212.2083333',\n",
       "  '731251',\n",
       "  '4367665']]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snotel_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "e2ec66d7-871a-4d41-86d8-9103338fa24e",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/HAG/'\n",
    "output_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/vertical_bias_correction/'\n",
    "filenames, tags = lidar_functions.create_command_template(input_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "fa41ae06-632a-4050-aff2-890962232b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_txt = output_path+'snotel_'+snotel_data[0][1]+'.csv'\n",
    "target_point = 'POINT('+snotel_data[0][5]+' '+snotel_data[0][6]+')'\n",
    "#extract_last_atPoint("
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "88d4167b-64ac-4bb9-8be1-429fd3f73238",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'POINT(733715 4374687)'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "cde511a7-90e9-460f-834a-1ccb53689c24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# target_point = 'Point(731945.81 4368517.23)'\n",
    "# # target_point = 'Point(732770.09 4370518.08)'\n",
    "# output_txt = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/clipped/snotel_541.csv'\n",
    "# # output_txt = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/clipped/snotel_539.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "2fee3fb6-292f-4727-9bb2-f691722e7e72",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_las_atPoint(filenames, tags, target_point, output_txt):\n",
    "    # merge all las files or stages\n",
    "    filter_merge = {\"type\":\"filters.merge\",\n",
    "                    \"tag\": \"merged\",\n",
    "                    \"inputs\": tags}\n",
    "    # crop\n",
    "    filter_crop = {'type':'filters.crop',\n",
    "                   'point':target_point,\n",
    "                   'distance':1,\n",
    "                   'inputs':'merged',\n",
    "                   'tag': 'cropped'}\n",
    "    # write merged las to raster\n",
    "    writers_gdal = {\"type\":\"writers.text\",\n",
    "                    \"format\":\"csv\",\n",
    "                    \"order\":\"Z\",\n",
    "                    'keep_unspecified':False,\n",
    "                    'filename':output_txt}\n",
    "    # Append each stage to a list prior to saving to json \n",
    "    pipeline_list = filenames.copy()\n",
    "    pipeline_list.append(filter_merge)\n",
    "    pipeline_list.append(filter_crop)\n",
    "    pipeline_list.append(writers_gdal)\n",
    "\n",
    "    pipeline_dict = {'pipeline' : pipeline_list}\n",
    "    # save to json\n",
    "    with open(extract_las_atPoint, 'w') as out:\n",
    "        json.dump(pipeline_dict, out, indent=4)\n",
    "    pdal_cmd = ['pdal','pipeline', extract_las_atPoint]\n",
    "    subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bc5f304-84b8-4980-b076-dbce779c67d8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "5d1acba4-37c9-4a5d-ab0e-5221f5757369",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # merge all las files or stages\n",
    "# filter_merge = {\"type\":\"filters.merge\",\n",
    "#                \"tag\": \"merged\",\n",
    "#                \"inputs\": tags}\n",
    "# # crop\n",
    "# filter_crop = {'type':'filters.crop',\n",
    "#                  'point':target_point,\n",
    "#                'distance':2,\n",
    "#                'inputs':'merged',\n",
    "#               'tag': 'cropped'}\n",
    "# # write merged las to raster\n",
    "# writers_gdal = {\"type\":\"writers.text\",\n",
    "#                 \"format\":\"csv\",\n",
    "#                 \"order\":\"Z\",\n",
    "#                 'keep_unspecified':False,\n",
    "#                'filename':output_txt}\n",
    "# # Append each stage to a list prior to saving to json \n",
    "# pipeline_list = filenames.copy()\n",
    "# pipeline_list.append(filter_merge)\n",
    "# pipeline_list.append(filter_crop)\n",
    "# pipeline_list.append(writers_gdal)\n",
    "\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# # save to json\n",
    "# with open(clip_json_SNOTEL, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "71f0c22b-03d4-405b-ad5e-494af9b40e91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['pdal', 'pipeline', 'piske_processing/PDAL_workflow/JSON/clip_to_geometries_snotel.json'], returncode=0)"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdal_cmd = ['pdal','pipeline', clip_json_SNOTEL]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bbc45e7-d895-43f8-9aab-af78f1328c3a",
   "metadata": {},
   "source": [
    "### Calculate Stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b75ae5-8299-468b-a120-67249d868a86",
   "metadata": {},
   "source": [
    "Applied\n",
    "Using 2014 NCALM Flight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bbe0cd43-447f-40c8-9bd1-73711bfbe4cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "ASO_20160326_hwy89_stats = lidar_functions.calculate_vertical_bias('SCB/Sagehen_lidar/ASO/ASO_SCB_20160326/hwy89_vertical_bias/clipped/ASO_SCB_20160326_hwy89_clip.laz')\n",
    "ASO_20160417_hwy89_stats = lidar_functions.calculate_vertical_bias('SCB/Sagehen_lidar/ASO/ASO_SCB_20160417/hwy89_vertical_bias/clipped/ASO_SCB_20160417_hwy89_clip.laz')\n",
    "ASO_20160518_hwy89_stats = lidar_functions.calculate_vertical_bias('SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/hwy89_vertical_bias/clipped/ASO_SCB_20160518_hwy89_clip.las')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c724eab8-e366-4c29-a3d0-33d69b753df8",
   "metadata": {},
   "outputs": [],
   "source": [
    "NCALM_2014_hwy89_stats = lidar_functions.calculate_vertical_bias('SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/hwy89_vertical_bias/clipped/NCALM_SCB_2014_hwy89_clip.las')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6670984d-73d8-462e-b036-31e6d3dd2015",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['lowest_10th', 0.24, 'mean', 0.3113292433537832, 'median', 0.31]\n",
      "['lowest_10th', 0.27, 'mean', 0.3502506714413608, 'median', 0.35]\n",
      "['lowest_10th', 0.37, 'mean', 0.4511492281303602, 'median', 0.44]\n",
      "['lowest_10th', -0.05, 'mean', 0.02644937635008014, 'median', 0.02]\n"
     ]
    }
   ],
   "source": [
    "print(ASO_20160326_hwy89_stats)\n",
    "print(ASO_20160417_hwy89_stats)\n",
    "print(ASO_20160518_hwy89_stats)\n",
    "print(NCALM_2014_hwy89_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f456a359-3004-44c3-b3a5-e2cea8e8d92f",
   "metadata": {},
   "source": [
    "**NCALM 2008**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "id": "3a31fa74-0489-4f75-9ddc-2801b3bfa425",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_las_txt = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/clipped/snotel_541.csv'\n",
    "input_las_txt = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/SNOTEL_vertical_bias/clipped/snotel_539.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "id": "b32fe203-316a-4976-81e5-6d41d5e8730b",
   "metadata": {},
   "outputs": [],
   "source": [
    "hag_arr = np.loadtxt(input_las_txt,skiprows=1)\n",
    "lowest_10th_per = np.nanpercentile(hag_arr, 10)\n",
    "mean_hag = np.nanmean(hag_arr)\n",
    "median_hag = np.nanmedian(hag_arr)\n",
    "stats_539 = [\"lowest_10th\",lowest_10th_per, \"mean\",mean_hag, \"median\", median_hag]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "id": "d72b947e-20e3-4679-9070-5f27a200bfc1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['lowest_10th', 1.22, 'mean', 1.3270967741935482, 'median', 1.33]"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats_539"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "id": "e4f8b688-2561-4f3c-9510-4b02e13b3409",
   "metadata": {},
   "outputs": [],
   "source": [
    "SD_539_error = stats_539[1] - SD_539_true\n",
    "# SD_541_error = stats_541[1] - SD_541_true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "id": "4101b646-33fa-40f3-9da3-7438337df036",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.18041666700000003"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SD_539_error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e5bcbb-8274-4847-b129-d1c92bb3cb58",
   "metadata": {},
   "source": [
    "### Correct and Rasterize\n",
    "In this case we know that the May flight has the most limited extent. One way to check for this would be to use a similar code to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49bf8021-ed32-4579-8754-818f8a1468f5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a60dcefd-7390-4041-844c-da5c58049da4",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_merge_rasterize_json = json_base_path + 'correct_merge_rasterize.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "008460b9-598d-4dae-9ce1-eacabbc00175",
   "metadata": {},
   "outputs": [],
   "source": [
    "# input_path = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/HAG/'\n",
    "# filenames, tags = lidar_functions.create_command_template(input_path)\n",
    "# output_tif = 'SCB/Sagehen_lidar/ASO/ASO_SCB_20160518/corrected_tif/ASO_SCB_20160518_vbc.tif'\n",
    "# target_assign = 'Z-'+str(abs(ASO_20160518_hwy89_stats[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "id": "ea3c9439-5269-4964-94e3-40a61bb02147",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/HAG/'\n",
    "filenames, tags = lidar_functions.create_command_template(input_path)\n",
    "output_tif = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_20080210/corrected_tif/NCALM_SCB_20080210_vbc.tif'\n",
    "target_assign = 'Z+'+str(abs(SD_539_error))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62d93f1b-301e-4406-b58f-c8dbcf62cf2a",
   "metadata": {},
   "source": [
    "Create the reader stages of the pipeline. Each file is read as an individual reader stage here, and we cheat here by copying the formatting of json files and creating a dictionary with the values as the correctly formatted inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "28fec394-70ca-404f-9fb5-67b32e934ed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# [730235.96, 738826.45, 4364741.66, 4372273.16, 8590.48999999999, 7531.5]\n",
    "\n",
    "writers_bounds = '(['+ str(ASO_SCB_20160518_ext[0])+','+ str(ASO_SCB_20160518_ext[1])+'],['+str(ASO_SCB_20160518_ext[2])+','+str(ASO_SCB_20160518_ext[3])+'])'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "id": "822e3cdd-58ac-48a0-b748-663ea519090d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge all las files or stages\n",
    "filter_merge = {\"type\":\"filters.merge\",\n",
    "               \"tag\": \"merged\",\n",
    "               \"inputs\": tags}\n",
    "# filter out the ground points of the tiles\n",
    "filter_assign = {'type': 'filters.assign',\n",
    "                 'value':\"Z=\"+target_assign,\n",
    "                 'inputs':'merged',\n",
    "                 'tag':'corrected'}\n",
    "# write merged las to raster\n",
    "writers_gdal= {\"type\": \"writers.gdal\",\n",
    "               'output_type': 'mean',\n",
    "              'resolution': 1.0,\n",
    "              'radius': '0.7',\n",
    "               'bounds': writers_bounds,\n",
    "               'inputs': 'corrected',\n",
    "               'filename':output_tif}\n",
    "# Append each stage to a list prior to saving to json \n",
    "pipeline_list = filenames.copy()\n",
    "pipeline_list.append(filter_merge)\n",
    "pipeline_list.append(filter_assign)\n",
    "pipeline_list.append(writers_gdal)\n",
    "\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# save to json\n",
    "with open(correct_merge_rasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "id": "bb980f62-9d55-40a9-99cc-1136f519df41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['pdal', 'pipeline', 'piske_processing/PDAL_workflow/JSON/correct_merge_rasterize.json'], returncode=0)"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# time = ~3 min\n",
    "pdal_cmd = ['pdal','pipeline', correct_merge_rasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b466837-460a-4f8b-9202-f81a226fa651",
   "metadata": {},
   "source": [
    "## Snow-Off Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa814a07-3c70-4bc8-b6d3-3b35f3336c30",
   "metadata": {},
   "source": [
    "### Vegetation Strata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7cba995-877a-4a96-a828-784ec57aade0",
   "metadata": {},
   "source": [
    "**Combined Pipeline Method**\n",
    "We'll use the same bounds as above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "106d03d8-7a0f-4f32-9f9d-6848384eca22",
   "metadata": {},
   "outputs": [],
   "source": [
    "#filterMergeRasterize_json = json_base_path+'filterMergeRasterize.json'\n",
    "# filterMergeRasterize_json = json_base_path+'filterMergeRasterize_neg0pt15_0pt15.json'\n",
    "# filterMergeRasterize_json = json_base_path+'filterMergeRasterize_0pt15_2.json'\n",
    "# filterMergeRasterize_json = json_base_path+'filterMergeRasterize_2.json'\n",
    "filterMergeRasterize_json = json_base_path+'filterMergeRasterize_neg0pt15_0pt15.json'\n",
    "# filterMergeRasterize_json = json_base_path+'filterMergeRasterize_2_nonground.json'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "af094e83-88cd-46f4-801d-7f31c2bbbd05",
   "metadata": {},
   "outputs": [],
   "source": [
    "writers_bounds = '(['+ str(ASO_SCB_20160518_ext[0])+','+ str(ASO_SCB_20160518_ext[1])+'],['+str(ASO_SCB_20160518_ext[2])+','+str(ASO_SCB_20160518_ext[3])+'])'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "fa124ce7-908c-4981-947b-57d296be799a",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/HAG/'\n",
    "output_tif = 'SCB/Sagehen_lidar/NCALM/NCALM_SCB_2014/veg_strata/veg_classes/vegStrata_neg0pt15_0pt1.tif'\n",
    "filenames, tags = lidar_functions.create_command_template(input_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "24f04675-a114-4c79-b9a6-8bd0ec645a21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge all las files or stages\n",
    "filter_merge = {\"type\":\"filters.merge\",\n",
    "               \"tag\": \"merged\",\n",
    "               \"inputs\": tags}\n",
    "# filter out the ground points of the tiles\n",
    "filter_range = {\"type\":\"filters.range\",\n",
    "                \"limits\":\"Z[-0.15:0.15)\",\n",
    "               'inputs':'merged',\n",
    "               'tag':'filtered'}\n",
    "# write merged las to raster\n",
    "writers_gdal= {\"type\": \"writers.gdal\",\n",
    "               'output_type': 'count',\n",
    "              'resolution': '1.0',\n",
    "              'radius': '0.7',\n",
    "               'bounds': writers_bounds,\n",
    "               'inputs': 'filtered',\n",
    "               'filename':output_tif}\n",
    "# Append each stage to a list prior to saving to json \n",
    "pipeline_list = filenames.copy()\n",
    "pipeline_list.append(filter_merge)\n",
    "pipeline_list.append(filter_range)\n",
    "pipeline_list.append(writers_gdal)\n",
    "\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# save to json\n",
    "with open(filterMergeRasterize_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "850dd26e-cc34-4731-9c2a-1b27b0d9c723",
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "ea7b3ba5-0b1c-4f7c-a068-180d263aba09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['pdal', 'pipeline', 'piske_processing/PDAL_workflow/JSON/filterMergeRasterize_neg0pt15_0pt15.json'], returncode=0)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdal_cmd = ['pdal','pipeline', filterMergeRasterize_json]\n",
    "subprocess.run(pdal_cmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "4464b547-4114-43ad-8c6c-7ac3256b62ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "efb4c3fe-af8d-4b65-b53f-04338df3d9e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.017118186666661"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(toc-tic) / 60"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe8463e-2c0e-4d0d-b2a7-1b132a8e1f4e",
   "metadata": {},
   "source": [
    "**Move to Raster workflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e93513b-14f1-40a0-97d0-56a7f770d024",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f814af3-a9e9-447e-a255-793df9e3b3f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e27ca4b-59b8-4cfb-8f0c-2857a4a29aa9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77c34c87-3a18-4906-ac34-6f83dea1d45c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3a658118-ad67-41a7-97e2-7d64913b8d80",
   "metadata": {},
   "source": [
    "# ------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f221a885-025a-44c5-a762-a9fdafc5a11f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "706a4c5e-29e3-4dce-93b0-3639e61925d6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c740714-5881-4763-bcb2-b2387741f477",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abbd18c8-82e0-4a86-b878-9a53710554f2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26833f00-aa67-426b-8e20-c1c3eff46862",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56715656-b594-4285-9b2a-5a875aaabd01",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "33e66bb1-b42b-445d-af25-de4b376c9652",
   "metadata": {},
   "source": [
    "# Extras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bf738b4-4eae-41dc-9cc8-898b177ac071",
   "metadata": {},
   "source": [
    "## File format conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be10eab7-1dc5-4561-9ff8-dbabb55abf57",
   "metadata": {},
   "source": [
    "### Convert .las to .txt\n",
    "see: https://pdal.io/stages/writers.text.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "486c745d-b984-41bc-a04a-b54dcfbbaf18",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "069497d0-a99b-4952-bfa2-3af3d3c0ef5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up json file commands\n",
    "output_txt = '/Volumes/cpiske/lidar_processing/python_scripts/PDAL/test_las/mcc_part_b_tile_004_000Test.asc'\n",
    "output_json = 'lidar_processing/python_scripts/PDAL_workflow/JSON/las_to_txt.json'\n",
    "\n",
    "# create a pipeline and save to a json file \n",
    "\n",
    "filter_dict = {'type':'readers.las',\n",
    "               'override_srs': \"EPSG:4326\",\n",
    "              'filename': input_las} # we are reading in a las file\n",
    "rasterize_dict = {'type':'writers.las',\n",
    "'format':'geojson',\n",
    "'order':'X,Y,Z',\n",
    "'keep_unspecified':'false',\n",
    "'filename':output_txt}\n",
    "\n",
    "\n",
    "pipeline_list = [filter_dict, rasterize_dict]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "with open(output_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f570972-2f18-40d1-b4d2-06f0ed5ec7bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_path = 'lidar_processing/python_scripts/PDAL/JSON/las_to_txt.json'\n",
    "pdal_commands = ['pdal', 'pipeline', json_path]\n",
    "subprocess.run(pdal_commands)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142a3a30-1ae6-481b-96ac-f95a1f6f4ddc",
   "metadata": {},
   "source": [
    "### .laz to .las"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe33f9e-84de-49ea-a538-0cd37b7f7877",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # set up json file commands\n",
    "# input_laz = 'lidar_processing/python_scripts/PDAL/test_las/ASO_USCAMB20180425f1a1_180425_1_dem_filter.laz'\n",
    "# output_las = 'lidar_processing/python_scripts/PDAL/test_las/ASO_USCAMB20180425f1a1_180425_1_dem_filter.las'\n",
    "# output_json = 'lidar_processing/python_scripts/PDAL/JSON/laz_to_las.json'\n",
    "\n",
    "# # create a pipeline and save to a json file \n",
    "\n",
    "# filter_dict = {'type':'readers.las',\n",
    "#                'filename': input_las} # we are reading in a las file\n",
    "# translate_dict = {'type':'writers.las',\n",
    "#                   \"a_srs\": \"EPSG:4326\",\n",
    "#                   'filename':output_las}\n",
    "\n",
    "\n",
    "# pipeline_list = [filter_dict, translate_dict]\n",
    "# pipeline_dict = {'pipeline' : pipeline_list}\n",
    "# with open(output_json, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb37c88-c5d9-4859-91ab-5c35e53bd514",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe40151-4f3c-4fc8-acc0-ff2e092d788b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# json_path = 'lidar_processing/python_scripts/PDAL/JSON/laz_to_las.json'\n",
    "# pdal_commands = ['pdal', 'pipeline', json_path]\n",
    "# subprocess.run(pdal_commands)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea292cbb-156e-4be0-9fa8-62560bd55feb",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "# Raster Caluclations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b90302-4ff2-461e-84b4-28f68e90575f",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in [0,1,2,3,4,5]:\n",
    "    apr_elev = 'lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/rasterize/ASO_20160417/mcc_part_b_tile_004_00'+str(i)+'.tif'\n",
    "    may_elev = 'lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/rasterize/ASO_20160518/mcc_part_b_tile_004_00'+str(i)+'.tif'\n",
    "    output = 'lidar_processing/PDAL_testFiles_tutorials/test_las/SCB/raster_subtract/mcc_part_b_tile_004_00'+str(i)+'.tif'\n",
    "    raster_sub = ['gdal_calc.py', '-a', apr_elev, '-b', may_elev, '--calc=\"a - b\"', '--outfile', output]\n",
    "    subprocess.run(raster_sub)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef95aabd-bd70-47a0-be17-993b89088961",
   "metadata": {},
   "source": [
    "# Theo Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "784a85fc-2834-4d0d-81a9-19f790b08678",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_las = 'SCB/kost_lidar_data/ASO_2016/2016_05_18/WGS84_G1762_to_NAD83_NAVD88/mcc_part_b_tile_004_000.las'\n",
    "\n",
    "z_min = 0.15\n",
    "z_max = 2\n",
    "z_range = 'Z[' + str(z_min) + ':' + str(z_max) + ']'\n",
    "output_raster = 'lidar_processing/python_scripts/PDAL/test_file/mcc_part_b_tile_004_000.tif'\n",
    "output_json = 'lidar_processing/python_scripts/PDAL/JSON/las_to_tif.json'\n",
    "resolution = 0.01\n",
    "\n",
    "filter_dict = {'type':'filters.range', 'limits':z_range}\n",
    "rasterize_dict = {'filename':output_raster,\n",
    "'gdaldriver':'GTiff',\n",
    "'output_type':'count',\n",
    "'resolution':resolution,\n",
    "'type': 'writers.gdal'}\n",
    "\n",
    "\n",
    "pipeline_list = [input_las, filter_dict, rasterize_dict]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "with open(output_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1929f364-00c7-482e-8d5f-537652eed3cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_las = 'SCB/kost_lidar_data/ASO_2016/2016_05_18/WGS84_G1762_to_NAD83_NAVD88/mcc_part_b_tile_004_000.las'\n",
    "\n",
    "# z_min = 0.15\n",
    "# z_max = 2\n",
    "# z_range = 'Z[' + str(z_min) + ':' + str(z_max) + ']'\n",
    "output_raster = 'lidar_processing/python_scripts/PDAL/test_file/mcc_part_b_tile_004_000.asc'\n",
    "output_json = 'lidar_processing/python_scripts/PDAL/JSON/las_to_asc.json'\n",
    "resolution = 0.01\n",
    "\n",
    "filter_dict = {'type':'filters.range', 'limits':z_range}\n",
    "rasterize_dict = {'filename':output_raster,\n",
    "'gdaldriver':'XYZ',\n",
    "'output_type':'count',\n",
    "'resolution':resolution,\n",
    "'type': 'writers.gdal'}\n",
    "\n",
    "\n",
    "pipeline_list = [input_las, filter_dict, rasterize_dict]\n",
    "pipeline_dict = {'pipeline' : pipeline_list}\n",
    "with open(output_json, 'w') as out:\n",
    "    json.dump(pipeline_dict, out, indent=4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4f6338-edab-4c85-8941-b05d63584dd8",
   "metadata": {},
   "source": [
    "sudo docker run -v path_to_laz_folder:/input 0b pdal info /input/test.laz\n",
    "\n",
    "where path_to_laz_folder is the path to the LAS/LAZ file (you just need the folder path, not the file path).\n",
    "\n",
    ":/input is the new folder that will be created in your Docker container that will hold your point cloud.\n",
    "\n",
    "0b is just the image id of pdal\n",
    "\n",
    "/input/test.laz is the path to the point cloud in the Docker container.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a941783e-828f-4226-a25a-b772a51d12d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_laz_folder = 'lidar_processing/python_scripts/PDAL/test_las'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aee8f9b-8d85-490a-a3cd-22d4b06bb646",
   "metadata": {},
   "outputs": [],
   "source": [
    "sudo docker run -v path_to_laz_folder:/input 0b pdal info /input/test.laz\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac352849-3336-4cd7-b7d4-2f0c2a2c79fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import subprocess\n",
    "import pdal\n",
    "\n",
    "def assemblePipeline(input_las, list_of_dicts):\n",
    "    pipeline_list = [input_las]\n",
    "    pipeline_list.extend(list_of_dicts)\n",
    "    pipeline_dict = {'pipeline' : pipeline_list}\n",
    "    return pipeline_dict\n",
    "\n",
    "def makeHeightFilter(height, buffer):\n",
    "    z_min = height - buffer/2\n",
    "    z_max = height + buffer/2\n",
    "    z_range = 'Z[' + str(z_min) + ':' + str(z_max) + ']'\n",
    "    heightDict = {'type':'filters.range', 'limits':z_range}\n",
    "    return heightDict\n",
    "\n",
    "def makeRasterizeFilter(output_raster, resolution, epsg):\n",
    "    rasterize_dict = {'filename':output_raster,\n",
    "                      'gdaldriver':'GTiff',\n",
    "                      'output_type':'count',\n",
    "                      'resolution':resolution,\n",
    "                      'override_srs' : epsg,\n",
    "                      'type': 'writers.gdal'}\n",
    "    return rasterize_dict\n",
    "\n",
    "def convertTifForPIL(input_raster, output_raster, epsg):\n",
    "    ''' GDAL bindings are an alien concept to me, so I gave up and used\n",
    "        subprocess.'''\n",
    "    commands = ['gdal_translate', input_raster, output_raster, '-ot', 'Byte', '-a_srs', epsg]\n",
    "    subprocess.run(commands)\n",
    "\n",
    "\n",
    "def buildHeightSlice(input_las, height, buffer, output_raster, resolution, epsg, json_path=None):\n",
    "    filter_dict = makeHeightFilter(height, buffer)\n",
    "    rasterize_dict = makeRasterizeFilter(output_raster, resolution, epsg)\n",
    "    filter_list = [filter_dict, rasterize_dict]\n",
    "    pipeline_dict = assemblePipeline(input_las, filter_list)\n",
    "\n",
    "    if json_path is not None:\n",
    "        with open(json_path, 'w') as out:\n",
    "            json.dump(pipeline_dict, out, indent=4)\n",
    "        pdal_commands = ['pdal', 'pipeline', json_path]\n",
    "        subprocess.run(pdal_commands)\n",
    "    else:\n",
    "        pdal_commands = json.dumps(pipeline_dict)\n",
    "        pipeline = pdal.Pipeline(pdal_commands)\n",
    "        pipeline.execute()\n",
    "\n",
    "input_las = '/Users/theo/data/las/TLS_0244_20180612_01_v003_30m_clip_height_norm.las'\n",
    "height = 1.37\n",
    "buffer = 0.05\n",
    "z_min = height - buffer/2\n",
    "z_max = height + buffer/2\n",
    "z_range = 'Z[' + str(z_min) + ':' + str(z_max) + ']'\n",
    "temp_raster = '/Users/theo/Pictures/almost_cool.tif'\n",
    "final_raster = '/Users/theo/Pictures/cool.tif'\n",
    "resolution = 0.01\n",
    "epsg = 'EPSG:3310'\n",
    "\n",
    "buildHeightSlice(input_las, height, buffer, temp_raster, resolution, epsg)\n",
    "convertTifForPIL(temp_raster, final_raster, epsg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d188910-a9a7-4c6c-b164-8a163c1d5445",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import subprocess\n",
    "import pdal\n",
    "import argparse\n",
    "\n",
    "# Create flags for the user to utilize.\n",
    "parser = argparse.ArgumentParser(description=\"Generate JSON pipeline to generate DTM from a point cloud.\")\n",
    "      \n",
    "required = parser.add_argument_group('Required arguments')\n",
    "required.add_argument('-crs', '--coordinate_system', required=True, action='store', help=\"EPSG code.\")\n",
    "required.add_argument('-i', '--infile', required=True, action='store', help=\"Input path to point cloud\")\n",
    "required.add_argument('-o', '--outfile', required=True, action='store', help=\"Output path.\")\n",
    "args = parser.parse_args()\n",
    "\n",
    "def generateJSON(infile, list_of_dicts):\n",
    "    pipeline_list = [infile]\n",
    "    pipeline_list.extend(list_of_dicts)\n",
    "    pipeline_dict = {'pipeline': pipeline_list}\n",
    "    with open(\"pipeline.json\", 'w') as out:\n",
    "        json.dump(pipeline_dict, out, indent=4)\n",
    "\n",
    "def generateDTM(epsg, infile, outfile):\n",
    "    reproject_dict = {\"type\": \"filters.reprojection\",\n",
    "                      \"out_srs\": \"EPSG:{}\".format(epsg)}\n",
    "    reclassify_zero_dict = {\"type\": \"filters.assign\",\n",
    "                       \"assignment\": \"Classification[:]=0\"}\n",
    "    elm_dict = {\"type\": \"filters.elm\"}\n",
    "    outlier_dict = {\"type\": \"filters.outlier\"}\n",
    "    smrf_dict = {\"type\": \"filters.smrf\", \"ignore\": \"Classification[7:7]\",\n",
    "                 \"slope\": 0.2, \"window\": 16, \"threshold\": 0.45, \"scalar\": 1.2}\n",
    "    range_dict = {\"type\":\"filters.range\", \"limits\":\"Classification[2:2]\"}\n",
    "    output_dict = {\"filename\": outfile, \"gdaldriver\": \"GTiff\", \"output_type\": \"all\", \"resolution\": 0.01, \"type\": \"writers.gdal\"}\n",
    "    list_of_dicts = list([reproject_dict, reclassify_zero_dict, elm_dict, outlier_dict, smrf_dict, range_dict, output_dict])\n",
    "    generateJSON(infile, list_of_dicts)\n",
    "    pdal_cmds = ['pdal', 'pipeline', 'pipeline.json']\n",
    "    subprocess.run(pdal_cmds)\n",
    "    \n",
    "generateDTM(args.coordinate_system, args.infile, args.outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0efcfb16-ecfc-4ce2-8a5e-6bf2974c4510",
   "metadata": {},
   "source": [
    "# More helpful things"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84929bbc-a6c6-49ad-bfc5-9789fe5dd577",
   "metadata": {},
   "source": [
    "### Get stats of a dataset\n",
    "see: https://www.spatialised.net/lidar-qa-with-pdal-part-1/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa7713f5-fbfd-412c-8488-04489fa868db",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name JSON file\n",
    "\n",
    "stats_json = 'lidar_processing/python_scripts/PDAL_workflow/JSON/stats.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cc3f4a2-adf9-4d2e-a62e-a2a538437396",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader_las = {\"type\":\"readers.las\",\n",
    "              \"filename\": input_las_stats}\n",
    "filter_stats = {\"type\":\"filters.stats\",\n",
    "                \"dimensions\":\"Z\",\n",
    "                \"global\":\"Z\",\n",
    "                \"advanced\":\"true\"}\n",
    "pipeline_list = [reader_las, filter_stats]\n",
    "#pipeline_dict = {reader_las, filter_stats}\n",
    "# with open(stats_json, 'w') as out:\n",
    "#     json.dump(pipeline_dict, out, indent=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ff5cfca-34a5-471d-8b40-bef477ef6438",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94a58e53-92c7-47c2-b75b-f9c9466b9e2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = pdal.Pipeline(json.dumps(pipeline_list))\n",
    "pipeline.execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "722c7605-47f2-4698-bec0-f2aa20a26f4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "json.loads(pipeline.metadata)[\"metadata\"][\"filters.stats\"][\"statistic\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c67b19a-0d9f-4b10-ad08-ccf9593fa659",
   "metadata": {},
   "source": [
    "## Navigating folders/files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67984138-2046-46f9-9187-26121ac09dca",
   "metadata": {},
   "source": [
    "**create list of files/folders with a wildcard (*)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8b23dd1-01e1-46d4-81e7-b9e5be2df38b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ex. list all files in folder4 that end in .laz = folder1/folder2/folder3/folder4/*.laz\n",
    "# ex. list all folders named folder3 in folder 1 = foler1/*/folder3\n",
    "# ex. list all list all contents in folder2 = folder2/* - note just folder 2, no subdirectories\n",
    "glob_cmd = 'path'\n",
    "glob_exe = glob.glob(file_glob_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "124af7f3-340e-4a3d-8b74-fbff724635f2",
   "metadata": {},
   "source": [
    "**create a list with all directories/subdirectories on a path**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "316d4080-2f26-48b8-b908-992955d0200f",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_folders = [x[0] for x in os.walk('path')]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ddee80f-1c7f-46e1-a8e8-2cee82ac6f02",
   "metadata": {},
   "source": [
    "**create a list with all directories/subdirectories on a path with specific folders**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fb7ef3c-3534-4566-9a4d-63ce9bcd074d",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_folders = [x[0] for x in os.walk('path')]\n",
    "index_pos_list = [ i for i in range(len(all_folders)) if all_folders[i][-6:] == 'retile' ] # must change this to meet requirements\n",
    "full_list = [all_folders[i] for i in index_pos_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507fb6c4-a80d-4b97-8533-1852b3a8e992",
   "metadata": {},
   "source": [
    "**get name of the directory just above one listed**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea8cc0c9-4dcb-44e7-a72f-8586c6828636",
   "metadata": {},
   "outputs": [],
   "source": [
    "subdirname = os.path.basename(os.path.dirname('path'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "141720ac-a112-4891-b727-3e4999ed4298",
   "metadata": {},
   "source": [
    "**create a list with only filenames**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5412acaf-9f08-430c-8ce2-ca0027d4d129",
   "metadata": {},
   "outputs": [],
   "source": [
    "onlyfiles = [f for f in os.listdir('path') if os.path.isfile(os.path.join('path', f))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d80f4b2-ebb0-4eb2-9a39-58d554be97b0",
   "metadata": {},
   "source": [
    "**create a list with full file paths**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfc96133-4087-4aca-a90b-302487a803e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "full_path = ['path' + '/' + s for s in onlyfiles]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b8a6064-f348-405d-bf7d-5e8c8f7bfd81",
   "metadata": {},
   "source": [
    "## Parallel Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "215beb4e-46bb-4271-b452-6849510f3a36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# time = 142.5\n",
    "tic = time.perf_counter()\n",
    "list(map(function, args));\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a4ca6cc-9fb5-48ce-8146-b41150dd9674",
   "metadata": {},
   "outputs": [],
   "source": [
    "# time = 53 s\n",
    "tic = time.perf_counter()\n",
    "if __name__ == \"__main__\":\n",
    "    pool = Pool(3)\n",
    "    pool.map(function, arg)\n",
    "    pool.close()\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "920e3ba6-7d5f-4a00-b563-ee2a53074b87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# time = very fast? .06s\n",
    "tic = time.perf_counter()\n",
    "if __name__ == \"__main__\":\n",
    "    executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)\n",
    "    executor.map(function, arg)\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb4c6986-5a20-472a-a5bd-4a854161ff30",
   "metadata": {},
   "outputs": [],
   "source": [
    "# time = 0.22\n",
    "tic = time.perf_counter()\n",
    "if __name__ == \"__main__\":\n",
    "    executor = concurrent.futures.ProcessPoolExecutor(max_workers=3)\n",
    "    executor.map(function, arg)\n",
    "toc = time.perf_counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1f97306-b0c1-4b9d-8b1e-5bd95d1732c6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}