Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

library: Add imagery Python library module to grass.script #3756

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion python/grass/script/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ include $(MODULE_TOPDIR)/include/Make/Python.make

DSTDIR = $(ETC)/python/grass/script

MODULES = core db raster raster3d vector array setup task utils
MODULES = core db imagery raster raster3d vector array setup task utils

PYFILES := $(patsubst %,$(DSTDIR)/%.py,$(MODULES) __init__)
PYCFILES := $(patsubst %,$(DSTDIR)/%.pyc,$(MODULES) __init__)
Expand Down
1 change: 1 addition & 0 deletions python/grass/script/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

from .core import *
from .db import *
from .imagery import *
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
from .raster import *
from .raster3d import *
from .vector import *
Expand Down
150 changes: 150 additions & 0 deletions python/grass/script/imagery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
"""
Imagery related functions to be used in Python scripts.

Usage:

::

import grass.script as gs

gs.imagery.group_to_dict(imagery_group)
Comment on lines +8 to +10
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you mean for it to be used this way, then don't add that to the __init__.py or leave .imagery here out. (I don't know if it would even not work the way it is written now.) It is basically a decision between the imagery being part of the internal structure versus part of the API. Simpler imports versus more freedom in naming (no danger of collisions).

I added from . import setup there so that there is no need for extra import, but that the namespace stays separate. However setup is a special case.

For consistency with raster and others, having from .imagery import * seems appropriate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should tend to avoid adding new star imports

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should tend to avoid adding new star imports

In general, of course, but here in __init__.py and next to from .raster import *?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I included the star import is do have it consistent with the other modules...

And with that: gs.imagery.group_to_dict(imagery_group) does work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said, gs.group_to_dict(imagery_group) works too. I do what you prefer regarding the star-import. Not being able to do gs.group_to_dict(imagery_group) without the star-import would be fine with me...
Let me know what to do to merge it...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm... tests fail without the import of imagery in __init__.py

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course the long way, import grass.script.imagery will work.
if you add it, then gs.imagery should work too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. Why it works right now? I thought from . import imagery is needed for gs.imagery.group_to_dict to work. I think I had to do it that way for from . import setup and gs.setup.init.

...

(C) 2024 by Stefan Blumentrath and the GRASS Development Team
This program is free software under the GNU General Public
License (>=v2). Read the file COPYING that comes with GRASS
for details.

.. sectionauthor:: Stefan Blumentrath
"""

from .core import read_command, warning, fatal
from .raster import raster_info


def group_to_dict(
imagery_group_name,
subgroup=None,
dict_keys="semantic_labels",
dict_values="map_names",
fill_semantic_label=True,
env=None,
):
"""Create a dictionary to represent an imagery group with metadata.

Depending on the dict_keys option, the returned dictionary uses either
the names of the raster maps ("map_names"), their row indices in the group
("indices") or their associated semantic_labels ("semantic_labels") as keys.
The default is to use semantic_labels. Note that map metadata
of the maps in the group have to be read to get the semantic label,
in addition to the group file. The same metadata is read when the
"metadata" is requested as dict_values. Other supported dict_values
are "map_names" (default), "semantic_labels", or "indices".

The function can also operate on the level of subgroups. In case a
non-existing (or empty sub-group) is requested a warning is printed
and an empty dictionary is returned (following the behavior of i.group).

Example::

>>> run_command("g.copy", raster="lsat7_2000_10,lsat7_2000_10")
>>> run_command("r.support", raster="lsat7_2000_10", semantic_label="L8_1")
>>> run_command("g.copy", raster="lsat7_2000_20,lsat7_2000_20")
>>> run_command("r.support", raster="lsat7_2000_20", semantic_label="L8_2")
>>> run_command("g.copy", raster="lsat7_2000_30,lsat7_2000_30")
>>> run_command("r.support", raster="lsat7_2000_30", semantic_label="L8_3")
>>> run_command("i.group", group="L8_group",
>>> input="lsat7_2000_10,lsat7_2000_20,lsat7_2000_30")
>>> group_to_dict("L8_group") # doctest: +ELLIPSIS
{"L8_1": "lsat7_2000_10", ... "L8_3": "lsat7_2000_30"}
>>> run_command("g.remove", flags="f", type="group", name="L8_group")
>>> run_command("g.remove", flags="f", type="raster",
>>> name="lsat7_2000_10,lsat7_2000_20,lsat7_2000_30")

:param str imagery_group_name: Name of the imagery group to process (or None)
:param str subgroup: Name of the imagery sub-group to process (or None)
:param str dict_keys: What to use as key for dictionary. Can bei either
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
"semantic_labels" (default), "map_names" or "indices"
:param str dict_values: What to use as values for dictionary. Can bei either
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
"map_names" (default), "semanic_labels", "indices" or
"metadata" (to return dictionaries with full map metadata)
:param bool fill_semantic_label: If maps in a group do not have a semantic
label, their index in the group is used
instead (default). Otherwise None / "none"
is used.
:param dict env: Environment to use when parsing the imagery group

:return: dictionary representing an imagery group with it's maps and their
semantic labels, row indices in the group, or metadata
:rtype: dict
"""
group_dict = {}
maps_in_group = (
read_command(
"i.group",
group=imagery_group_name,
subgroup=subgroup,
flags="g",
quiet=True,
env=env,
)
.strip()
.split()
)

if dict_keys not in {"indices", "map_names", "semantic_labels"}:
raise ValueError(f"Invalid dictionary keys <{dict_keys}> requested")

if dict_values not in {"indices", "map_names", "semantic_labels", "metadata"}:
raise ValueError(f"Invalid dictionary values <{dict_values}> requested")

if subgroup and not maps_in_group:
warning(
_("Empty result returned for subgroup <{sg}> in group <{g}>").format(
sg=subgroup, g=imagery_group_name
)
)

for idx, raster_map in enumerate(maps_in_group):
raster_map_info = None
# Get raster metadata if needed
if (
dict_values in {"semantic_labels", "metadata"}
or dict_keys == "semantic_labels"
):
raster_map_info = raster_info(raster_map, env=env)

# Get key for dictionary
if dict_keys == "indices":
key = str(idx + 1)
elif dict_keys == "map_names":
key = raster_map
elif dict_keys == "semantic_labels":
key = raster_map_info["semantic_label"]
if not key or key == '"none"':
if fill_semantic_label:
key = str(idx + 1)
else:
fatal(
_(
"Semantic label missing for raster map {m} in group <{g}>."
).format(m=raster_map, g=imagery_group_name)
)

if dict_values == "indices":
val = str(idx + 1)
elif dict_values == "map_names":
val = raster_map
elif dict_values == "semantic_labels":
val = raster_map_info["semantic_label"]
elif dict_values == "metadata":
val = raster_map_info
if key in group_dict:
warning(
_(
"Key {k} from raster map {m} already present in group dictionary."
"Overwriting existing entry..."
).format(k=key, r=raster_map)
)
group_dict[key] = val
return group_dict
163 changes: 163 additions & 0 deletions python/grass/script/testsuite/test_imagery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
from grass.exceptions import CalledModuleError
from grass.gunittest.case import TestCase
from grass.gunittest.main import test

import grass.script as gs


class TestImageryGroupToDict(TestCase):
"""Tests function `group_to_dict` that returns raster maps
from an imagery group and their metadata."""

@classmethod
def setUpClass(cls):
cls.bands = [1, 2, 3]
cls.raster_maps = [f"lsat7_2002_{band}0" for band in cls.bands]
cls.group = "L8_group"
cls.subgroup = "L8_group_subgroup"
# Create input maps with label and group
for band in cls.bands:
cls.runModule(
"g.copy", raster=[f"lsat7_2002_{band}0", f"lsat7_2002_{band}0"]
)
cls.runModule(
"r.support", map=f"lsat7_2002_{band}0", semantic_label=f"L8_{band}"
)
cls.runModule("i.group", group=cls.group, input=cls.raster_maps)

@classmethod
def tearDownClass(cls):
cls.runModule("g.remove", type="raster", name=cls.raster_maps, flags="f")
cls.runModule("g.remove", type="group", name=cls.group, flags="f")

def test_basic_group_dict_defaults(self):
"""Test with semantic labels as keys and map names as values (defaults)"""
ref_dict = {f"L8_{band}": f"lsat7_2002_{band}0" for band in self.bands}
group_info = gs.imagery.group_to_dict(self.group)
# Check that a dict is returned
self.assertIsInstance(group_info, dict)
self.assertListEqual(list(ref_dict.keys()), list(group_info.keys()))
self.assertListEqual(
list(ref_dict.values()), [val.split("@")[0] for val in group_info.values()]
)

def test_non_existing_group(self):
"""Test that function fails if group does not exist"""
# Non existing group
self.assertRaises(
CalledModuleError, gs.imagery.group_to_dict, "non_existing_group"
)

def test_invalid_dict_key(self):
"""Test that function fails if invalid keys are requested"""
self.assertRaises(
ValueError,
gs.imagery.group_to_dict,
self.group,
**{"dict_keys": "invalid_dict_key"},
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
)

def test_invalid_dict_value(self):
"""Test that function fails if invalid values are requested"""
self.assertRaises(
ValueError,
gs.imagery.group_to_dict,
self.group,
**{"dict_values": "invalid_dict_value"},
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
)

def test_missing_subgroup(self):
"""Test that empty dict is returned if subgroup does not exist"""
group_info = gs.imagery.group_to_dict(
self.group, subgroup="non_existing_subgroup"
)

# Check that an empty dict is returned
self.assertDictEqual(group_info, {})

def test_basic_group_map_keys(self):
"""Test with map_names as keys and semantic_labels as values"""
ref_dict = {f"lsat7_2002_{band}0": f"L8_{band}" for band in self.bands}
group_info = gs.imagery.group_to_dict(
self.group, dict_keys="map_names", dict_values="semantic_labels"
)
# Check that a dict is returned
self.assertIsInstance(group_info, dict)
self.assertListEqual(
list(ref_dict.keys()), [key.split("@")[0] for key in group_info.keys()]
)
self.assertListEqual(list(ref_dict.values()), list(group_info.values()))

def test_basic_group_index_keys(self):
"""Test with indices as keys and mapnames as values"""
ref_dict = {str(band): f"lsat7_2002_{band}0" for band in self.bands}
group_info = gs.imagery.group_to_dict(self.group, dict_keys="indices")
# Check that a dict is returned
self.assertIsInstance(group_info, dict)
self.assertListEqual(list(ref_dict.keys()), list(group_info.keys()))
self.assertListEqual(
list(ref_dict.values()),
list([val.split("@")[0] for val in group_info.values()]),
ninsbl marked this conversation as resolved.
Show resolved Hide resolved
)

def test_full_info_group_label_keys(self):
"""Test with semantic labels as keys and full map metadata as values"""
group_info = gs.imagery.group_to_dict(self.group, dict_values="metadata")
# Check that a dict is returned
self.assertIsInstance(group_info, dict)
self.assertListEqual(
[f"L8_{band}" for band in self.bands],
[key.split("@")[0] for key in group_info.keys()],
)
for band in self.bands:
# Take some metadata keys from raster_info
for metadata_key in [
"north",
"nsres",
"cols",
"datatype",
"map",
"date",
"semantic_label",
"comments",
]:
self.assertIn(metadata_key, group_info[f"L8_{band}"])

def test_full_info_group_label_keys_subgroup(self):
"""Test with map names as keys and full map metadata as values"""
metadata_keys = {
"north",
"nsres",
"cols",
"datatype",
"map",
"date",
"semantic_label",
"comments",
}
self.runModule(
"i.group", group=self.group, subgroup=self.subgroup, input=self.raster_maps
)
group_info = gs.imagery.group_to_dict(
self.group,
subgroup=self.subgroup,
dict_keys="map_names",
dict_values="metadata",
)
# Check that a dict is returned
self.assertIsInstance(group_info, dict)
self.assertListEqual(
[f"lsat7_2002_{band}0" for band in self.bands],
[key.split("@")[0] for key in group_info.keys()],
)
for key, val in group_info.items():
# Check keys
self.assertTrue(key.startswith("lsat7_2002_"))
# Check values
self.assertIsInstance(val, dict)
# Take some metadata keys from raster_info
self.assertTrue(metadata_keys.issubset(set(val.keys())))


if __name__ == "__main__":
test()
Loading