Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError in build_renewable_profiles #22

Closed
2 tasks done
ekatef opened this issue Oct 22, 2023 · 2 comments
Closed
2 tasks done

ValueError in build_renewable_profiles #22

ekatef opened this issue Oct 22, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@ekatef
Copy link
Member

ekatef commented Oct 22, 2023

Checklist

  • I am using the current main branch or the latest release. Please indicate.
  • I am running on an up-to-date pypsa-earth environment. Update via conda env update -f pypsa-earth/envs/environment.yaml.

Describe the Bug

When changing the region to one outside of Nigeria, an error appears in build_renewable_profiles (a full listing is bellow).

It looks like the reason is that the curreng copernicus land-cover file covers only Nigeria:

image

As a quick fix, that is enough to manually replace pypsa-distribution/pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif with a full version (1.71 GB; can be loaded from pypsa-earth datakit via https://drive.google.com/file/d/1jN5iV_iaRa5lc1XOaEtletKPUgPNOVuf/view).

Error Message

rule build_renewable_profiles:
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log
    jobid: 3
    benchmark: benchmarks/build_renewable_profiles_solar
    reason: Missing output files: resources/renewable_profiles/profile_solar.nc; Set of input files has changed since last execution
    wildcards: technology=solar
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=20000, mem_mib=19074
INFO:snakemake.logging:rule build_renewable_profiles:
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log
    jobid: 3
    benchmark: benchmarks/build_renewable_profiles_solar
    reason: Missing output files: resources/renewable_profiles/profile_solar.nc; Set of input files has changed since last execution
    wildcards: technology=solar
    resources: tmpdir=/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T, mem_mb=20000, mem_mib=19074

INFO:snakemake.logging:
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
paths.regions
resources/shapes/microgrid_bus_shapes.geojson
INFO:__main__:correction_factor is set as 0.854337
INFO:__main__:Calculate landuse availabilities...
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 80, in raster_geometry_mask
    window = geometry_window(dataset, shapes, pad_x=pad_x, pad_y=pad_y)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/features.py", line 477, in geometry_window
    window = window.intersection(raster_window)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 775, in intersection
    return intersection([self, other])
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 125, in wrapper
    return function(*args[0])
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 239, in intersection
    return functools.reduce(_intersection, windows)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/windows.py", line 257, in _intersection
    raise WindowError(f"Intersection is empty {w1} {w2}")
rasterio.errors.WindowError: Intersection is empty Window(col_off=57098, row_off=-23804, width=1009, height=1009) Window(col_off=0, row_off=0, width=25092, height=23488)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 673, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 373, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 314, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 229, in projected_mask
    masked, transform_ = mask(raster, geom, crop=True, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 178, in mask
    shape_mask, transform, window = raster_geometry_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 86, in raster_geometry_mask
    raise ValueError('Input shapes do not overlap raster.')
ValueError: Input shapes do not overlap raster.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/pypsa-distribution/.snakemake/scripts/tmpxrlwxru8.build_renewable_profiles.py", line 525, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 748, in compute_availabilitymatrix
    availability = list(pool.map(_process_func, shapes.index))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 673, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 373, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 314, in shape_availability
    masked, transform = projected_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/atlite/gis.py", line 229, in projected_mask
    masked, transform_ = mask(raster, geom, crop=True, **kwargs)
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 178, in mask
    shape_mask, transform, window = raster_geometry_mask(
  File "~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/rasterio/mask.py", line 86, in raster_geometry_mask
    raise ValueError('Input shapes do not overlap raster.')
ValueError: Input shapes do not overlap raster.
[Sun Oct 22 15:32:24 2023]
INFO:snakemake.logging:[Sun Oct 22 15:32:24 2023]
Error in rule build_renewable_profiles:
    jobid: 3
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_renewable_profiles:
    jobid: 3
    input: ~/pypsa-distribution/pypsa-earth/resources/natura.tiff, ~/pypsa-distribution/pypsa-earth/resources/shapes/country_shapes.geojson, ~/pypsa-distribution/pypsa-earth/resources/shapes/offshore_shapes.geojson, pypsa-earth/data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, pypsa-earth/data/gebco/GEBCO_2021_TID.nc, pypsa-earth/data/hydro_capacities.csv, pypsa-earth/data/eia_hydro_annual_generation.csv, resources/powerplants.csv, resources/shapes/microgrid_bus_shapes.geojson, pypsa-earth/cutouts/casr-2013-era5.nc
    output: resources/renewable_profiles/profile_solar.nc
    log: logs/build_renewable_profile_solar.log (check log file(s) for error details)

Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: pypsa-earth/.snakemake/log/2023-10-22T153212.944843.snakemake.log
WARNING:snakemake.logging:Complete log: pypsa-earth/.snakemake/log/2023-10-22T153212.944843.snakemake.log
@ekatef ekatef added the bug Something isn't working label Oct 22, 2023
@ekatef
Copy link
Member Author

ekatef commented Dec 16, 2023

The issue can be resolved in a more elegant way if specifying tutorial: false in config.pypsa-earth.yaml and re-running the workflow. It feels like a note in the docs may be enough to fix it.

@ekatef ekatef mentioned this issue Dec 16, 2023
Open
6 tasks
@ekatef
Copy link
Member Author

ekatef commented Dec 16, 2023

Closed in favour of #13.

@ekatef ekatef closed this as completed Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ekatef