Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance improvement - change crs on hand data loads and not each forecast rebuild #930

Closed
RobHanna-NOAA opened this issue Oct 24, 2024 · 3 comments
Assignees
Labels
bug Something isn't working Data Loads Priority - High Priority Level Request
Milestone

Comments

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Oct 24, 2024

Currently, we are re-projecting the data crs on every feature for each forecast update. If we change the projection to 3857 when hand data is loaded in the first place, we can drop the re-project for each forecast. This could make a big impact on calcs for forecasts.

Note: hand is coming in as epsg:5070 and hand Alaska is coming in as epsg:3338. When we data load, we can ensure all huc starting with "19" (which is Alaska), then set the crs as 3338 when the object is loaded before re-projecting. Note. knowing the right incoming projection is super important before reprojecting.

Check the Alaska versus Conus for catchments as well.
Is ras2fim covered as well to convert to 3857 at data load? I think it is, but need to check forecast loads to ensure they are also 3857 as well.

@RobHanna-NOAA RobHanna-NOAA added Request bug Something isn't working Priority - High Priority Level Data Loads labels Oct 24, 2024
@RobHanna-NOAA RobHanna-NOAA self-assigned this Oct 24, 2024
@RobHanna-NOAA RobHanna-NOAA added this to the V2.1.8 milestone Oct 24, 2024
@RobHanna-NOAA RobHanna-NOAA changed the title Performance improvement - change crs on hand data loads and not on daily forecasts Performance improvement - change crs on hand data loads and not each forecast rebuild Oct 24, 2024
@RobHanna-NOAA
Copy link
Contributor Author

See (Issue 932)[https://github.com//pull/932] which changed
Core/LAMBDA/viz_functions/image_based/viz_hand_fim_processing/lambda_function.py. That script is called every time forecast data is loaded and it reprojects all HUCs each time.

Change the initial fim data load so it does the reprojected there instead of each forecast. This means one reproject for any release and not per forecast. We are basically copy/pasting part of this lambda code into the initial hand data load. Watch for catchments too for this.

@RobHanna-NOAA
Copy link
Contributor Author

Put this optimization on hold for now. It might a similar fix for rems and catchments. Research required for current catchments and rem load and handling for crs, even for hand data. in the lambda above, it was thought the hand data was coming in as ESRI:102039 but for conus it really was EPSG:5070. In 932, it fixed both CONUS and Alaska.

@RobHanna-NOAA
Copy link
Contributor Author

deemed to big and part of a much bigger picture of optimizing catchments, REMS and hydrotables

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Data Loads Priority - High Priority Level Request
Projects
None yet
Development

No branches or pull requests

1 participant