Issue reading NDFD GRIB data

[sapiennervosa]

Hello,

I've found that grib-rs is unable to read GRIB2 files from NDFD. This is true of API usage and the gribber CLI alike. Examples of NDFD GRIB files can be found here:

https://tgftp.nws.noaa.gov/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003

One thing I did check was that the issue wasn't simply the header that NDFD puts into its files. Deleting that, unfortunately, didn't solve the issue. These files can be decoded using wgrib2 and I did verify the ones I tried with grib-rs weren't malformed by reading them with wgrib2 successfully.

Now, to be clear; I've done zero debugging myself yet. I have written (most) of a GRIB2 parser in Python myself, and am fairly familiar with the format, so I am not wholly unqualified to figure this out. I also do very much need a rust implementation of a GRIB decoder for a project I'm working on so I have some selfish altruism motivating me here, and whatever fix I make I can offer up via a pull request if anyone's interested. However, I wanted to share this in case there was obvious oversight on my part before I dive into it.

[noritada]

Thanks for the report and for providing the URL for the data.
Although I have not downloaded all the data available from the URL, I have identified the problem, at least for ds.phail.bin.

The problem is due to the fact that the grid system and data compression methods used in this data are not yet supported by the library.
I have not been able to add support for these because I have not been able to find data files to test with.
Since you have provided one, I can now proceed with development using that data.
I will be able to complete addition of support for the compression method used this weekend.

Additional information

The GRIB2 data consists of 9 sections.
Among them, Section 3 defines the grid system and requires the library to implement various grid systems such as latitude/longitude grid, Lambert grid, and so on.
Section 5 defines compression methods and requires the library to implement decoding of various compression methods.

The currently supported grid systems and compression methods in this project are listed here.

[sapiennervosa]

I appreciate you looking into this!

I'm not sure how familiar you are with meteorology here in the US, but pretty much all the weather data we get comes at least in part from observations and forecasts by the NOAA (National Oceanic and Atmospheric Administration). NDFD is just one of the many ways and contexts by which they distribute it. As such, it would be likely be very useful to many people to have full support for how they use GRIB. If you have the interest, you may find their documentation useful: https://vlab.noaa.gov/web/mdl/ndfd-grib-all-sections

I hesitate to ask for so much without offering anything in return. FWIW my team is making a big investment in Rust and, though we are all fairly new to the language, we will likely be able to make contributions to this project as we come across features we need.

[noritada]

I think I finally have a decent understanding of the issue you are writing about.

The lack of support for grid systems in Section 3 does not affect users if they do not use coordinate information, and the lack of support for compression methods in Section 5 does not affect users if they do not decode the values, but there is an issue with reading NDFD GRIB2 message structure, which is a problem for scanning GRIB2.
This affects all operations.

One thing I did check was that the issue wasn't simply the header that NDFD puts into its files. Deleting that, unfortunately, didn't solve the issue. These files can be decoded using wgrib2 and I did verify the ones I tried with grib-rs weren't malformed by reading them with wgrib2 successfully.

The problem is that the "header" you are writing is sandwiched between NDFD GRIB2 messages. This means that when we read multiple GRIB2 messages in a file, simply deleting the very first header will not resolve the error.
I will attempt to fix this issue.

00000000  2a 2a 2a 2a 30 30 30 30  37 35 37 38 37 33 2a 2a  |****0000757873**|
00000010  2a 2a 0a 4c 4e 55 5a 39  38 20 4b 57 4e 53 20 30  |**.LNUZ98 KWNS 0|
00000020  32 30 36 32 35 0d 0d 0a  2a 2a 2a 2a 30 30 30 30  |20625...****0000|
00000030  31 38 35 32 38 33 2a 2a  2a 2a 0a 4c 4e 55 42 31  |185283****.LNUB1|
00000040  32 20 4b 57 4e 53 20 30  32 30 36 32 35 0d 0d 0a  |2 KWNS 020625...|
00000050  47 52 49 42 00 00 00 02  00 00 00 00 00 02 d3 ae  |GRIB............|
:
0002d3f0  00 00 00 00 00 00 00 00  00 00 37 37 37 37 2a 2a  |..........7777**|
0002d400  2a 2a 30 30 30 30 31 39  30 38 33 31 2a 2a 2a 2a  |**0000190831****|
0002d410  0a 4c 4e 55 43 31 32 20  4b 57 4e 53 20 30 32 30  |.LNUC12 KWNS 020|
0002d420  36 32 35 0d 0d 0a 47 52  49 42 00 00 00 02 00 00  |625...GRIB......|

(Each message starts with GRIB and ends with 7777. Byte sequence between 7777 and GRIB causes an error.)

[sapiennervosa]

Apologies; I should've either tried removing all of them or communicated that NOAA inserted them before each message. The wgrib2 tool NOAA provides is fairly fault tolerant and is able to find the start and end byte sequences with or without the NDFD specific stuff they stick in between them. I clearly forgot my own efforts at a parser were more or less porting that one and assumed a similar behavior. So, again, my apologies there.

With that in mind, it could also be helpful (but not essential) if there was more detail in grib-rs error messages relating to reading GRIB data. At the moment it's very much just pass or fail. I have no love of wgrib2, but it does at least provide some helpful insight when attempting to parse malformed files. For example, this is an error it spat out when I gave it a file with some random bytes removed:

*** FATAL ERROR: rd_grib2_msg_seq_file for .../ndfd_data/SL.us008001/ST.opnl/DF.gr2/DC.ndfd/AR.conus/VP.001-003/ds.temp.bin, missing end section ('7777') ***

Even that is somewhat misleading because the end section is present but simply not at the correct offset. However, it does give the user some indication of where to start looking for the issue.

[noritada]

Thank you very much.

I agree with your opinion that the current error messages are not informative enough. So far, my most important theme has been to deal with the variety of GRIBs (increasing the number of types that can be processed), but now that I have achieved some of that, I will be focusing on ease of use.

I went through all the NDFD data types last week to find out which data to use as test data (which data would allow me to test as many things as possible), and if my analysis was correct, the changes made up until a few days ago allowed users to properly retrieve grid point values from any NDFD GRIB files. I am still working on improving the code base, but since the changes contains an important fix, I would like to release it once over the weekend.

As for grid system, support for the Lambert grid is not yet implemented, and latitude/longitude for each grid point cannot be calculated. Since all NFDF GRIB files seem to use the Lambert grid, I will continue to work on this as a remaining issue.

[noritada]

I have released v0.9.0, which contains support for Lambert conformal grid used for NDFD GRIB data.

[sapiennervosa]

Fantastic! I cannot thank you enough for your work in this regard. I'll be integrating this into our application in the near future and will keep you apprised of our experience with it.

…

On Tue, Feb 6, 2024 at 4:39 AM Noritada Kobayashi ***@***.***> wrote: I have released v0.9.0, which contains support for Lambert conformal grid used for NDFD GRIB data. — Reply to this email directly, view it on GitHub <#59 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWM5BAUYSZFPWUJKC2JJ33YSH26TAVCNFSM6AAAAAA6YAP3YOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGOBQGMYDM> . You are receiving this because you authored the thread.Message ID: ***@***.***>