Loading in already downloaded files offline

[tlg7]

This is a great piece of software for downloading model data easily and efficiently.

I have directory of HRRR data downloaded already. To read in that data, I use two commands:

# Load the Herbie object
H = Herbie(pstest.strftime("%Y-%m-%d %H"), fxx=0)

# Reads the variable as a dataset.
ds = H.xarray("CAPE:surface", remove_grib=False)

This works well on my own computer, but on an HPC cluster fails, most likely due to internet restrictions.

My question: is it possible to read in the files I already have downloaded without using any internet connection? The files names are saved so that I can't understand them (ex. subset_44ef1277__hrrr.t00z.wrfsfcf00.grib2), or else I would simply call the file with the matching time/variable.

Thanks!

[blaylockbk]

Thanks @tlg7 for the feedback. I'll have to look into this more to remember how it works, but you have a valid point...if you are offline, but the grib2 file exists, Herbie should be able to work.

FYI: The naming structure for a subsetted file is

subset_<hash-that-represents-the-grib-messages>_<original-filename>

You can use Herbie to predict that filename

H.get_localFilePath("CAPE:surface")

OUT:
WindowsPath('C:/Users/blaylock/data/hrrr/20230301/subset_8eefabec__hrrr.t06z.wrfsfcf00.grib2')

H.get_localFilePath("CAPE:surface").exists()

OUT:
True  (if you haven't downloaded that file yet)

---

So, what happens on your HPC when you try to create the Herbie object? Does it spin forever?

Herbie should check if the file exists locally before it looks on the internet or the file. That said, Herbie always tries to find the index file on the internet because Herbie does not download index files.

[tlg7]

It doesn't spin for too long, maybe only a few minutes. Pasted below it the error output.

Those get_localFilePath commands do work if I have a herbie object loaded, but if there is not internet connection, I've no object file to run that command on.

`
Traceback (most recent call last):
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib
3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib
3/util/connection.py", line 95, in create_connection
raise err
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib
3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib
3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib
3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib3/connection.py", line 179, in _new_conn
raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x153306b5b9d0>, 'Connection to noaa-hrrr-bdp-pds.s3.amazonaws.com timed out. (connect timeout=None)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='noaa-hrrr-bdp-pds.s3.amazonaws.com', port=443): Max retries exceeded with url: /hrrr.20210703/conus/hrrr.t18z.wrfsfcf00.grib2 (Caused by Co
nnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x153306b5b9d0>, 'Connection to noaa-hrrr-bdp-pds.s3.amazonaws.com timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/gpfs/home/tlg14c/CapeSheartoPSFinal_5.py", line 62, in MRMStoPS
H=Herbie(pstest.strftime("%Y-%m-%d %H"), fxx=0)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/herbie/archive.py", line 239, in init
self.grib, self.grib_source = self.find_grib()
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/herbie/archive.py", line 409, in find_grib
if self._check_grib(grib_url):
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/herbie/archive.py", line 333, in _check_grib
head = requests.head(url)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/api.py", line 100, in head
return request("head", url, **kwargs)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/site-packages/requests/adapters.py", line 553, in send
raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='noaa-hrrr-bdp-pds.s3.amazonaws.com', port=443): Max retries exceeded with url: /hrrr.20210703/conus/hrrr.t18z.wrfsfcf00.grib2 (Caused by
ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x153306b5b9d0>, 'Connection to noaa-hrrr-bdp-pds.s3.amazonaws.com timed out. (connect timeout=None)'))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/gpfs/home/tlg14c/CapeSheartoPSFinal_5.py", line 107, in
p.map(MRMStoPS,dir_ps)
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/gpfs/home/tlg14c/.conda/envs/herbie/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
requests.exceptions.ConnectTimeout: None: Max retries exceeded with url: /hrrr.20210703/conus/hrrr.t18z.wrfsfcf00.grib2 (Caused by None)`