Major rewrite

[peterdudfield]

I thought it we are considering a major rewrite, then it might be worth say why? and what functionality we have right now.
Re writing the nwp-consumer has been a success.

Why

• make code easier to read
• easier to debug
• easier to expand on
• join together training dataset code and live consumer. Currently training dataset code is in dagster repo (TODO add link)

Requirements

• Get 3 different satellite data, 0, 9 and 45.5.
• For 0 and 9, get both the HRV and non-HRV data
• Optional to scale data, to match exsisting google public dataset data
• Dataset: Make training dataset, and merge to monthly zarrs
• Live: Get live data
• Live: When getting live data, save raw and process timestamp files so that we dont have to repeat this everytime the consumer runs
• Live: Save 'latest' file, so ML models can just load that file
• Live: If 0 degrees fails, pull 9 degree satellite
• Live: Pull 0 degree in 5 minut intervals, pull 9 and 45.5 deg in 15 minute intervals
• Dockerize service, so that live and dataset can use the same docker file
• Live: Save raw data and processed data, so that we only have to pull raw data once

Optional

• Live: For live data, with 9 degrees, use data tailor service (we might find we might not need to). If we do this, we need a way to run the data tailor clean up service too.
• Add options for Goes satellite back in

[peterdudfield]

@devsjc could you add any other requirements you think we need