Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[1pt] PR: Change Docker base image and stop Whitebox downloads #1291

Open
wants to merge 13 commits into
base: dev
Choose a base branch
from
Open

[1pt] PR: Change Docker base image and stop Whitebox downloads #1291

wants to merge 13 commits into from
+1,063 −598

Conversation

mluck
Copy link
Contributor

@mluck mluck commented Sep 16, 2024 •
edited
Loading

This PR incorporates a number of changes to the Docker environment:

During testing, it was discovered that many files which are not in the src directory, can no longer see the src\utils files. Adjusting the dockerfile to add extra values to the PYTHONPATH variable fixed it.
Note: This triggers new docker images to be made.

Changes

  • Dockerfile: Changes base image to gdal:ubuntu-small-3.8.4 and removes code related to JDK
  • Pipfile and Pipfile.lock: Upgrades fiona, jupyterlab, and whitebox
  • fim_pre_processing: Removes WBT_PATH assignment
  • src/
    • agreedem.py and unique_pixel_and_allocation.py: sets whitebox_dir to WBT_PATH

Testing

Ran fim_pipeline.sh and synthesize_test_cases.py on HUC 03100204.

  • Also ran pipeline on 12040101, including synthesize and eval_plots.

Deployment Plan (For developer use)

How does the changes affect the product?

  • Code only?
  • If applicable, has a deployment plan be created with the deployment person/team?
  • Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
  • If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
  • Require new pre-clip set?
  • Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@mluck mluck added High Priority dependencies Pull requests that update a dependency file labels Sep 16, 2024
@RobHanna-NOAA
Copy link
Contributor

The Docker fix has been tested. Removed all images from our OWP "u" servers, had ITSG check to make sure there was no residue. Then loaded our new container from here and ITSG said it was still fixed. So we are good to go on the docker base image fix

@mluck mluck changed the title [1pt] PR: Change Docker base image [1pt] PR: Change Docker base image and stop Whitebox downloads Sep 19, 2024
@RobHanna-NOAA
Copy link
Contributor

built the image on one of my EC2's, then ran sudo find /var/lib/docker/ -type f -name "java-17" to ensure it didn't pop back up. All good on that part. Doing a pipeline test next.

@mluck
Copy link
Contributor Author

mluck commented Sep 23, 2024

Tested the new changes by building a new Docker image and running fim_pipeline, synthesize_test_cases, and eval_plots. Verified no import error and the eval plots were created successfully.

RobHanna-NOAA
RobHanna-NOAA approved these changes Sep 23, 2024
View reviewed changes
Copy link
Contributor

@RobHanna-NOAA RobHanna-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matt and I went back and forth on testing and validate. All is well now

@mluck
Copy link
Contributor Author

mluck commented Sep 24, 2024

Note: the message [Errno 2] No such file or directory: './whitebox_tools' (see opengeos/whitebox-python#72 (comment)) is still occurring but doesn't seem to affect the functionality of whitebox.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RobHanna-NOAA RobHanna-NOAA RobHanna-NOAA approved these changes

Assignees

@mluck mluck

@CarsonPruitt-NOAA CarsonPruitt-NOAA

Labels
dependencies Pull requests that update a dependency file High Priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[8pt] Open JDK 17 re-appeared [8pt] Whitebox calling URL at runtime
3 participants
@mluck @RobHanna-NOAA @CarsonPruitt-NOAA