A reload of all 3Dep DEMs from USGS was performed to refresh our data.

acquire_and_preprocess_3dep_dems.py had to be run twice, one for Alaska and once for the rest to two different folder. This is due to different CRS's. Eventually, we could merge these into one run. This also meant two separate vrt runs / files.

This also triggered a new set of pre-clips for both AK and CONUS+ but the outputs can/were put into the same folder, so fim_pipeline looks in one common pre-clip folder.

Other minor adjustment include:

• A change to chmod (permissions) files / folder for the logging folders. After careful re-analysis, it was discovered there was some duplication.
• Added a simple duration system to the sierra test system, rating_curve_comparions.py. This was added as it is expected to be used soon for a full BED/Production. The fix focuses purely on duration, but a test did detect a possible pre-existing logic problem. A separate card will be created for that.

Note:
The root folder for DEM is being changed from:
/inputs/3dep_dems/.... to
/inputs/dems/3dep_dems/....
This recognizes other DEMs that may be coming in the near future.
The same sub-folder patterns have not be changed.
No attempts will be made at this time to move older files, only new incoming from this PR.

Changes

• CITATION.cff: has not be updated for a very long time.
• fim_post_processing.sh: Update to file/folder permissions.
• data
  • `usgs\acquire_and_preprocesss_3dep_dem.pys
    • Minor text updates and updated datatime.now patterns as the old ones are not deprecated
    • An adjustment to how number of jobs are handled. The system dis-likes too many multi-procs due to open network connections to the source.
    • Change the target output folder from optional to required.
  • wbd
    • generate_pre_clip_from_huc8.py:
      • Minor text updates
    • preprocess_wbd.py
      • Minor text updates
• src\base_variables.env: Changes to variables to reflect new dems and pre-clip paths.
• tools\rating_curve_comparisons.py
  • Added duration system as mentioned above.

Testing

• The full DEMs have been re-loaded and the same as the pre-clips. It will now be copied to all enviros from the EFS folder.
• rating_curve_comparion was tested but when detected a possible logic problem, the logic code was comments to validate just the duration system.
• Tests and analysis of file/folder permissions based on fim_post_processing.sh

Deployment Plan (For developer use)

Files/folders to be copied to other enviros

• From: /efs..../inputs/dems

• From: /efs.../inputs/pre_clip_huc8/20240917/

• Copy to:

  • Fim S3
  • The two OWP enviro, including the dev_fim_share and the local for the OWP prod server
  • ESIP

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set? (done already)
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• (sort of) Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations