HRRR module #27
Merged
HRRR module #27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Start a new HRRR Data handling class and initialize with moving the filename finder method from utils. More functionality to be moved there soon.
Create a base class for all unit tests that use data of RME. Contains all required directories and some other basic basin info as attributes. Also takes care of creating and cleaning up of outputs created in each test case.
Add a helper method to be able to skip tests that require an external HTTP request via environment variable. This speeds up running the whole suite against only local data.
Change the VARIABLE of the HRRRPreprocessor to be an immutable tuple. This changes the instance to use a copy of that constant and enables changing of an individual instance without affecting the set on the class.
Add a central method to the utils that sets up the logging either to a given config or a default instance.
Also slightly change the message formatting and add a space before the message.
Move the logic that retrieves data via FTP to a dedicated class. Start on breaking up the HRRR class tome.
Move the logic that retrieves data via HTTP to a dedicated class. Chipping away on the HRRR class tome.
Looks a lot like the get_data method and was commented out to begin with
Move HTTP request specific attributes to the dedicated class.
Create basic class that knows how to parse a config file and sets up a logger. See doc for more details.
Add a base class to serve as super class for local HRRR files to be read.
Use ConfigFile as super class for HTTPRetrieval. Also add HTTP specific configuration options to be read from a configuration file and have them accessible as properties. Last change is to use the FileHandler for date formatting of HRRR files.
Add the start and end date as a class constant so they are readily accessible in tests.
Extract logic from the HRRR tome to the hrrr data module. This separates the logic to load data on disk independent of the data format to one class, while separate classes contain the logic for each format. Added some basic specs around the loader logic
All logic from the HRRR tome was moved to the data module. Keeping the HRRR class around for backwards compatibility. To be retired.
jomey
commented
Dec 28, 2020
scotthavens
suggested changes
Dec 28, 2020
|
Nice work on this Joe! Was the skiing that bad? Opened #28 to remove the FTP and OpenDAP so I'm not super concerned about those tests. How does this play with SMRF? Would be interested to hear if the SMRF tests pass using this branch . |
Yeah, our storms keep under-producing and the coverage isn't consistent yet. Many sharks still sticking out. SMRF was my mini integration suite. I have WFR installed in pip editable mode. Then every time I changed things in WRF I also run the HRRR test ( |
Also adds a test for this parameter option.
scotthavens
approved these changes
Dec 29, 2020
The logic never raised an error once all six forecast hours were not successfully loaded.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When going through the code of loading HRRR files, I found myself a bit lost. The current
hrrr.pyhas all the logic around loading of files on disk as well as fetching from remote sources.This PR breaks the HRRR tome (as I called it) apart into separate pieces and establishes a structure that allows for different weather sources. (Thinking of GOES here).
Centralized a few pieces as for instances fetching a name for a HRRR file. Also established a testing infrastructure similar to SMRF, where there is a basic class that uses RME data.
Looks like a lot of new code, but essentially moves pieces around.
The slight change I introduced is around the logic for getting data from files on disk. There, the call to combine by coordinates is now outside the loading data from disk.