-
-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
library: Add imagery Python library module to grass.script #3756
Open
ninsbl
wants to merge
20
commits into
OSGeo:main
Choose a base branch
from
ninsbl:image_group_dict
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
ec43c65
new group_to_dict function
ninsbl 149c84a
new group_to_dict function
ninsbl a7764b1
add imagery module
ninsbl 3635ea8
add imagery module
ninsbl f78b813
add imagery module
ninsbl cc86c0b
address code review
ninsbl f5aab58
address code review
ninsbl e2b7880
address test faliures
ninsbl 88c2e25
improve logic
ninsbl 2525777
update docstring
ninsbl 85d36a0
address review
ninsbl 9e7923c
fix tests
ninsbl 569dff6
fix semantic_labels values
ninsbl 89c431d
address review
ninsbl 25f1f97
Merge branch 'main' into image_group_dict
ninsbl 1da985a
Apply suggestions from code review
ninsbl ebf46dd
Apply suggestions from code review
ninsbl 0a6a697
Revert "add imagery module"
ninsbl e2965f0
Merge branch 'main' into image_group_dict
ninsbl a753a3b
Merge branch 'main' into image_group_dict
echoix File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,150 @@ | ||
""" | ||
Imagery related functions to be used in Python scripts. | ||
|
||
Usage: | ||
|
||
:: | ||
|
||
import grass.script as gs | ||
|
||
gs.imagery.group_to_dict(imagery_group) | ||
... | ||
|
||
(C) 2024 by Stefan Blumentrath and the GRASS Development Team | ||
This program is free software under the GNU General Public | ||
License (>=v2). Read the file COPYING that comes with GRASS | ||
for details. | ||
|
||
.. sectionauthor:: Stefan Blumentrath | ||
""" | ||
|
||
from .core import read_command, warning, fatal | ||
from .raster import raster_info | ||
|
||
|
||
def group_to_dict( | ||
imagery_group_name, | ||
subgroup=None, | ||
dict_keys="semantic_labels", | ||
dict_values="map_names", | ||
fill_semantic_label=True, | ||
env=None, | ||
): | ||
"""Create a dictionary to represent an imagery group with metadata. | ||
|
||
Depending on the dict_keys option, the returned dictionary uses either | ||
the names of the raster maps ("map_names"), their row indices in the group | ||
("indices") or their associated semantic_labels ("semantic_labels") as keys. | ||
The default is to use semantic_labels. Note that map metadata | ||
of the maps in the group have to be read to get the semantic label, | ||
in addition to the group file. The same metadata is read when the | ||
"metadata" is requested as dict_values. Other supported dict_values | ||
are "map_names" (default), "semantic_labels", or "indices". | ||
|
||
The function can also operate on the level of subgroups. In case a | ||
non-existing (or empty sub-group) is requested a warning is printed | ||
and an empty dictionary is returned (following the behavior of i.group). | ||
|
||
Example:: | ||
|
||
>>> run_command("g.copy", raster="lsat7_2000_10,lsat7_2000_10") | ||
>>> run_command("r.support", raster="lsat7_2000_10", semantic_label="L8_1") | ||
>>> run_command("g.copy", raster="lsat7_2000_20,lsat7_2000_20") | ||
>>> run_command("r.support", raster="lsat7_2000_20", semantic_label="L8_2") | ||
>>> run_command("g.copy", raster="lsat7_2000_30,lsat7_2000_30") | ||
>>> run_command("r.support", raster="lsat7_2000_30", semantic_label="L8_3") | ||
>>> run_command("i.group", group="L8_group", | ||
>>> input="lsat7_2000_10,lsat7_2000_20,lsat7_2000_30") | ||
>>> group_to_dict("L8_group") # doctest: +ELLIPSIS | ||
{"L8_1": "lsat7_2000_10", ... "L8_3": "lsat7_2000_30"} | ||
>>> run_command("g.remove", flags="f", type="group", name="L8_group") | ||
>>> run_command("g.remove", flags="f", type="raster", | ||
>>> name="lsat7_2000_10,lsat7_2000_20,lsat7_2000_30") | ||
|
||
:param str imagery_group_name: Name of the imagery group to process (or None) | ||
:param str subgroup: Name of the imagery sub-group to process (or None) | ||
:param str dict_keys: What to use as key for dictionary. It can be either | ||
"semantic_labels" (default), "map_names" or "indices" | ||
:param str dict_values: What to use as values for dictionary. It can be either | ||
"map_names" (default), "semanic_labels", "indices" or | ||
"metadata" (to return dictionaries with full map metadata) | ||
:param bool fill_semantic_label: If maps in a group do not have a semantic | ||
label, their index in the group is used | ||
instead (default). Otherwise None / "none" | ||
is used. | ||
:param dict env: Environment to use when parsing the imagery group | ||
|
||
:return: dictionary representing an imagery group with it's maps and their | ||
semantic labels, row indices in the group, or metadata | ||
:rtype: dict | ||
""" | ||
group_dict = {} | ||
maps_in_group = ( | ||
read_command( | ||
"i.group", | ||
group=imagery_group_name, | ||
subgroup=subgroup, | ||
flags="g", | ||
quiet=True, | ||
env=env, | ||
) | ||
.strip() | ||
.split() | ||
) | ||
|
||
if dict_keys not in {"indices", "map_names", "semantic_labels"}: | ||
raise ValueError(f"Invalid dictionary keys <{dict_keys}> requested") | ||
|
||
if dict_values not in {"indices", "map_names", "semantic_labels", "metadata"}: | ||
raise ValueError(f"Invalid dictionary values <{dict_values}> requested") | ||
|
||
if subgroup and not maps_in_group: | ||
warning( | ||
_("Empty result returned for subgroup <{sg}> in group <{g}>").format( | ||
sg=subgroup, g=imagery_group_name | ||
) | ||
) | ||
|
||
for idx, raster_map in enumerate(maps_in_group): | ||
raster_map_info = None | ||
# Get raster metadata if needed | ||
if ( | ||
dict_values in {"semantic_labels", "metadata"} | ||
or dict_keys == "semantic_labels" | ||
): | ||
raster_map_info = raster_info(raster_map, env=env) | ||
|
||
# Get key for dictionary | ||
if dict_keys == "indices": | ||
key = str(idx + 1) | ||
elif dict_keys == "map_names": | ||
key = raster_map | ||
elif dict_keys == "semantic_labels": | ||
key = raster_map_info["semantic_label"] | ||
if not key or key == '"none"': | ||
if fill_semantic_label: | ||
key = str(idx + 1) | ||
else: | ||
fatal( | ||
_( | ||
"Semantic label missing for raster map {m} in group <{g}>." | ||
).format(m=raster_map, g=imagery_group_name) | ||
) | ||
|
||
if dict_values == "indices": | ||
val = str(idx + 1) | ||
elif dict_values == "map_names": | ||
val = raster_map | ||
elif dict_values == "semantic_labels": | ||
val = raster_map_info["semantic_label"] | ||
elif dict_values == "metadata": | ||
val = raster_map_info | ||
if key in group_dict: | ||
warning( | ||
_( | ||
"Key {k} from raster map {m} already present in group dictionary." | ||
"Overwriting existing entry..." | ||
).format(k=key, r=raster_map) | ||
) | ||
group_dict[key] = val | ||
return group_dict |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,163 @@ | ||
from grass.exceptions import CalledModuleError | ||
from grass.gunittest.case import TestCase | ||
from grass.gunittest.main import test | ||
|
||
import grass.script as gs | ||
|
||
|
||
class TestImageryGroupToDict(TestCase): | ||
"""Tests function `group_to_dict` that returns raster maps | ||
from an imagery group and their metadata.""" | ||
|
||
@classmethod | ||
def setUpClass(cls): | ||
cls.bands = [1, 2, 3] | ||
cls.raster_maps = [f"lsat7_2002_{band}0" for band in cls.bands] | ||
cls.group = "L8_group" | ||
cls.subgroup = "L8_group_subgroup" | ||
# Create input maps with label and group | ||
for band in cls.bands: | ||
cls.runModule( | ||
"g.copy", raster=[f"lsat7_2002_{band}0", f"lsat7_2002_{band}0"] | ||
) | ||
cls.runModule( | ||
"r.support", map=f"lsat7_2002_{band}0", semantic_label=f"L8_{band}" | ||
) | ||
cls.runModule("i.group", group=cls.group, input=cls.raster_maps) | ||
|
||
@classmethod | ||
def tearDownClass(cls): | ||
cls.runModule("g.remove", type="raster", name=cls.raster_maps, flags="f") | ||
cls.runModule("g.remove", type="group", name=cls.group, flags="f") | ||
|
||
def test_basic_group_dict_defaults(self): | ||
"""Test with semantic labels as keys and map names as values (defaults)""" | ||
ref_dict = {f"L8_{band}": f"lsat7_2002_{band}0" for band in self.bands} | ||
group_info = gs.imagery.group_to_dict(self.group) | ||
# Check that a dict is returned | ||
self.assertIsInstance(group_info, dict) | ||
self.assertListEqual(list(ref_dict.keys()), list(group_info.keys())) | ||
self.assertListEqual( | ||
list(ref_dict.values()), [val.split("@")[0] for val in group_info.values()] | ||
) | ||
|
||
def test_non_existing_group(self): | ||
"""Test that function fails if group does not exist""" | ||
# Non existing group | ||
self.assertRaises( | ||
CalledModuleError, gs.imagery.group_to_dict, "non_existing_group" | ||
) | ||
|
||
def test_invalid_dict_key(self): | ||
"""Test that function fails if invalid keys are requested""" | ||
self.assertRaises( | ||
ValueError, | ||
gs.imagery.group_to_dict, | ||
self.group, | ||
dict_keys="invalid_dict_key", | ||
) | ||
|
||
def test_invalid_dict_value(self): | ||
"""Test that function fails if invalid values are requested""" | ||
self.assertRaises( | ||
ValueError, | ||
gs.imagery.group_to_dict, | ||
self.group, | ||
dict_values="invalid_dict_value", | ||
) | ||
|
||
def test_missing_subgroup(self): | ||
"""Test that empty dict is returned if subgroup does not exist""" | ||
group_info = gs.imagery.group_to_dict( | ||
self.group, subgroup="non_existing_subgroup" | ||
) | ||
|
||
# Check that an empty dict is returned | ||
self.assertDictEqual(group_info, {}) | ||
|
||
def test_basic_group_map_keys(self): | ||
"""Test with map_names as keys and semantic_labels as values""" | ||
ref_dict = {f"lsat7_2002_{band}0": f"L8_{band}" for band in self.bands} | ||
group_info = gs.imagery.group_to_dict( | ||
self.group, dict_keys="map_names", dict_values="semantic_labels" | ||
) | ||
# Check that a dict is returned | ||
self.assertIsInstance(group_info, dict) | ||
self.assertListEqual( | ||
list(ref_dict.keys()), [key.split("@")[0] for key in group_info.keys()] | ||
) | ||
self.assertListEqual(list(ref_dict.values()), list(group_info.values())) | ||
|
||
def test_basic_group_index_keys(self): | ||
"""Test with indices as keys and mapnames as values""" | ||
ref_dict = {str(band): f"lsat7_2002_{band}0" for band in self.bands} | ||
group_info = gs.imagery.group_to_dict(self.group, dict_keys="indices") | ||
# Check that a dict is returned | ||
self.assertIsInstance(group_info, dict) | ||
self.assertListEqual(list(ref_dict.keys()), list(group_info.keys())) | ||
self.assertListEqual( | ||
list(ref_dict.values()), | ||
[val.split("@")[0] for val in group_info.values()], | ||
) | ||
|
||
def test_full_info_group_label_keys(self): | ||
"""Test with semantic labels as keys and full map metadata as values""" | ||
group_info = gs.imagery.group_to_dict(self.group, dict_values="metadata") | ||
# Check that a dict is returned | ||
self.assertIsInstance(group_info, dict) | ||
self.assertListEqual( | ||
[f"L8_{band}" for band in self.bands], | ||
[key.split("@")[0] for key in group_info.keys()], | ||
) | ||
for band in self.bands: | ||
# Take some metadata keys from raster_info | ||
for metadata_key in [ | ||
"north", | ||
"nsres", | ||
"cols", | ||
"datatype", | ||
"map", | ||
"date", | ||
"semantic_label", | ||
"comments", | ||
]: | ||
self.assertIn(metadata_key, group_info[f"L8_{band}"]) | ||
|
||
def test_full_info_group_label_keys_subgroup(self): | ||
"""Test with map names as keys and full map metadata as values""" | ||
metadata_keys = { | ||
"north", | ||
"nsres", | ||
"cols", | ||
"datatype", | ||
"map", | ||
"date", | ||
"semantic_label", | ||
"comments", | ||
} | ||
self.runModule( | ||
"i.group", group=self.group, subgroup=self.subgroup, input=self.raster_maps | ||
) | ||
group_info = gs.imagery.group_to_dict( | ||
self.group, | ||
subgroup=self.subgroup, | ||
dict_keys="map_names", | ||
dict_values="metadata", | ||
) | ||
# Check that a dict is returned | ||
self.assertIsInstance(group_info, dict) | ||
self.assertListEqual( | ||
[f"lsat7_2002_{band}0" for band in self.bands], | ||
[key.split("@")[0] for key in group_info.keys()], | ||
) | ||
for key, val in group_info.items(): | ||
# Check keys | ||
self.assertTrue(key.startswith("lsat7_2002_")) | ||
# Check values | ||
self.assertIsInstance(val, dict) | ||
# Take some metadata keys from raster_info | ||
self.assertTrue(metadata_keys.issubset(set(val.keys()))) | ||
|
||
|
||
if __name__ == "__main__": | ||
test() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you mean for it to be used this way, then don't add that to the
__init__.py
or leave.imagery
here out. (I don't know if it would even not work the way it is written now.) It is basically a decision between the imagery being part of the internal structure versus part of the API. Simpler imports versus more freedom in naming (no danger of collisions).I added
from . import setup
there so that there is no need for extra import, but that the namespace stays separate. Howeversetup
is a special case.For consistency with raster and others, having
from .imagery import *
seems appropriate.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should tend to avoid adding new star imports
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, of course, but here in
__init__.py
and next tofrom .raster import *
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason I included the star import is do have it consistent with the other modules...
And with that:
gs.imagery.group_to_dict(imagery_group)
does work.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That said,
gs.group_to_dict(imagery_group)
works too. I do what you prefer regarding the star-import. Not being able to dogs.group_to_dict(imagery_group)
without the star-import would be fine with me...Let me know what to do to merge it...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm... tests fail without the import of
imagery
in__init__.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't it similar to https://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-submodules-are-added-to-the-package-namespace-trap
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or https://stackoverflow.com/a/40823467
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course the long way, import grass.script.imagery will work.
if you add it, then gs.imagery should work too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused. Why it works right now? I thought
from . import imagery
is needed forgs.imagery.group_to_dict
to work. I think I had to do it that way forfrom . import setup
andgs.setup.init
.