Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client.latest can not establish latest date #11

Open
rahanozturk2 opened this issue Jul 8, 2022 · 2 comments · May be fixed by #22
Open

Client.latest can not establish latest date #11

rahanozturk2 opened this issue Jul 8, 2022 · 2 comments · May be fixed by #22

Comments

@rahanozturk2
Copy link

Hello,

I've recently started to use ecmwf opendata and have a problem when I want to automate the download.

It says; "Cannot establish latest date for {'date': ['2022-07-07 18:00:00'], 'resol': ['0p4-beta'], 'stream': ['oper'], 'type': ['fc'], 'step': ['0'], 'url': ['https://data.ecmwf.int/forecasts']}"
---> 26 if client.latest().strftime("%Y-%m-%d%H") == date:

while True:
    files = [f for f in listdir(path) if f.endswith(".grib2")]
    if files == []:
        date = client.latest().strftime("%Y-%m-%d_%H")
        filename = path+'{}.grib2'.format(date)
        client.retrieve(request, filename)
    else:
        date = str(files[-1])[-20:-6]
        if client.latest().strftime("%Y-%m-%d_%H") == date:               #### 26
                pass
        else:
            date = client.latest().strftime("%Y-%m-%d_%H")
            filename = path+'{}.grib2'.format(date)
            client.retrieve(request, filename)
    time.sleep(60*60)
@packman2008
Copy link

I get the same error but only when the latest data is from the 18:00 run on the previous day.

@packman2008 packman2008 mentioned this issue Sep 4, 2022
Closed
@packman2008
Copy link

I think I understand what's going wrong and have a fix, although there may be better fixes available.

Client.latest() works by setting a search start time to 18z on the current date, e.g. at 02:00 on 05/09 the start time would be 05/090 18z, a stop date which is 1 day 6 hours before the start time, e.g. 04/09 12z and a delta time of 6 hours (this is for the case when a specific run time isn't requested using the time parameter). It then loops back from start time whilst the loop time is greater than the stop time looking for a file that meets the search requirements.

If you're looking for a file which only exists in the 00z and 12z runs, e.g. 144, then this will always fail when using latest() to identify whether the 18z run is available. The loop runs as follows at, say, 02:00 (the time you'd start checking for the 18z run data to become available):

05/09 18z - Run time is in the future so no files exist yet
05/09 12z - Run time is in the future so no files exist yet
05/09 06z - Run time is in the future so no files exist yet
05/09 00z - Run has started but too early for files to be available yet
04/09 18z - Run has started but too early for files to be available yet
04/09 12z - Loop ends because loop time is not > than stop time.
Error is raised

There is another error condition when the time parameter is specified. If you want to find the latest 18z run, the same loop setup for start and stop date, but the delta is set to one day. In this case the loop does this (for start date= 05/09 18z and stop date of 04/09 12z and delta=1day)

05/09 18z - Run time is in the future so no files exist yet
04/09 18z - Run has started but too early for files to be available yet
03/09 18z - Loop ends because loop time is not > than stop time.
Error is raised

A fix which resolves both of these issues is to change the line in the definition of latest() in client.py from

stop = date - datetime.timedelta(days=1, hours=6)

to

stop = date - datetime.timedelta(days=2, hours=0)

I believe that for any valid combination of execution date/time, run times, time parameter, etc, setting a delta of 2 days will always find a file that matches the search requirements rather than raising an error in some circumstances.

packman2008 added a commit to packman2008/ecmwf-opendata that referenced this issue Jul 7, 2023
@packman2008
Update client.py
4ce5725 
Resolves ecmwf#11
@packman2008 packman2008 linked a pull request Jul 7, 2023 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update client.py to allow client.latest() to identify 18z runs packman2008/ecmwf-opendata
2 participants
@packman2008 @rahanozturk2