Cannot Create Forcing Data

[jarq6c]

On Step 2 of Preconditioning, I receive the following error.

(base) bash-4.4# python /scripts/forcing/nwm/nwm_2.0_to_adhydro.py -s 2019-07-01:00 -e 2019-12-31:23 -m 76 -c 2 /data /data/forcing/data/
WARNING: Missing data file for time 2019-09-06 08:00:00; Ommiting from forcing output.
WARNING: Missing data file for time 2019-09-06 09:00:00; Ommiting from forcing output.
WARNING: Missing data file for time 2019-09-06 10:00:00; Ommiting from forcing output.
WARNING: Missing data file for time 2019-11-22 17:00:00; Ommiting from forcing output.
WARNING: Missing data file for time 2019-11-22 20:00:00; Ommiting from forcing output.
WARNING: Missing data file for time 2019-11-30 17:00:00; Ommiting from forcing output.
Running with 2 cores.
Mem after reading ADHydro polygon geometries: 0.168964740546875 GB
Time to read ADHydro geometry: 0.2160508632659912
Extracting geom
Creating dataframe
Building NWM grid spatial index.
Processing 1376 mesh elements, 109 channel elements on 2 cores.
Intersecting mesh/channel elements with grid...
Error running polygonIntersect
ERROR intersecting nwm grid
list index out of range
Error running polygonIntersect
ERROR intersecting nwm grid
list index out of range
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 358, in <module>
    intersectMesh = pd.read_parquet('meshIntersect.parquet')
  File "/opt/conda/lib/python3.7/site-packages/pandas/io/parquet.py", line 460, in read_parquet
    path, columns=columns, use_nullable_dtypes=use_nullable_dtypes, **kwargs
  File "/opt/conda/lib/python3.7/site-packages/pandas/io/parquet.py", line 218, in read
    mode="rb",
  File "/opt/conda/lib/python3.7/site-packages/pandas/io/parquet.py", line 82, in _get_path_or_handle
    handles = get_handle(path_or_handle, mode, is_text=False)
  File "/opt/conda/lib/python3.7/site-packages/pandas/io/common.py", line 651, in get_handle
    handle = open(handle, ioargs.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'meshIntersect.parquet'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 288, in intersectMP
    raise(e)
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 284, in intersectMP
    result = intersectGrid(adHydro_frame, nwmCoords, nwmIndex, how)
  File "/scripts/forcing/nwm/overlay.py", line 207, in intersectGrid
    raise(e)
  File "/scripts/forcing/nwm/overlay.py", line 203, in intersectGrid
    nwmFrame = adHydroFrame.apply(polygonIntersect, args=(nwmCoords, nwmIndex, None), axis=1)
  File "/opt/conda/lib/python3.7/site-packages/geopandas/geodataframe.py", line 1362, in apply
    func, axis=axis, raw=raw, result_type=result_type, args=args, **kwargs
  File "/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py", line 7768, in apply
    return op.get_result()
  File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 185, in get_result
    return self.apply_standard()
  File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 276, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 290, in apply_series_generator
    results[i] = self.f(v)
  File "/opt/conda/lib/python3.7/site-packages/pandas/core/apply.py", line 110, in f
    return func(x, *args, **kwds)
  File "/scripts/forcing/nwm/overlay.py", line 118, in polygonIntersect
    idx = nn[0]
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 373, in <module>
    results = pool.map(partial(intersectMP, whichADHydro='mesh', how='poly'), mesh_chunks)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
IndexError: list index out of range

[hellkite500]

Did you run this with data from the release artifacts?

[hellkite500]

Short term work around, I've uploaded a forcing artifact you can extract to skip this processing step:

https://github.com/hellkite500/adhydro-training/releases/tag/v0.0.3-alpha

[jarq6c]

Did you run this with data from the release artifacts?

Ah, I assumed the only thing missing was the superfile. I'll try again with the new data.

[jarq6c]

I was able to to use the preprocessed forcing data. I got down to the simulation step and it failed. I got a bunch of errors about groundwater variables not being set. I may try to start from scratch. Maybe I missed a step.

[hellkite500]

From a fresh setup, a separate, yet maybe related error

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 238, in getVarMP
    meshDataArray[ var_index , mesh.index[0]:mesh.index[-1]+1] = result['mesh'].loc[var].values
UnboundLocalError: local variable 'meshDataArray' referenced before assignment
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scripts/forcing/nwm/nwm_2.0_to_adhydro.py", line 593, in <module>
    results = pool.map(partial(getVarMP, minRow=minY, maxRow=maxY, minCol=minX, maxCol=maxX), chunks )
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
UnboundLocalError: local variable 'meshDataArray' referenced before assignment