Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove geomedian fy aliases #1086

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Remove geomedian fy aliases #1086

wants to merge 2 commits into from

Conversation

jmettes
Copy link
Member

@jmettes jmettes commented Jun 27, 2022

Remove changed names for geomedian measurements from FY product definitions (at least temporarily). I've run some sample years for testing, but am not using the latest odc-stats image which has changed to use aliases instead

$ s3-to-dc s3://dea-public-data-dev/derivative/ga_ls5t_nbart_gm_fyear_3/3-0-0/**/*.odc-metadata.yaml ga_ls5t_nbart_gm_fyear_3

...
ERROR:root:Failed to index dataset s3://dea-public-data-dev/derivative/ga_ls5t_nbart_gm_fyear_3/3-0-0/x50/y26/2010-07-01--P1Y/ga_ls5t_nbart_gm_fyear_3_x50y26_2010-07-01--P1Y_final.odc-metadata.yaml with error Failed to create dataset with error The dataset is not specifying all of the measurements in this product.
Missing fields are;
{'nbart_swir_1', 'nbart_swir_2', 'nbart_nir', 'nbart_blue', 'nbart_red', 'nbart_green'}
 The URI was s3://dea-public-data-dev/derivative/ga_ls5t_nbart_gm_fyear_3/3-0-0/x50/y26/2010-07-01--P1Y/ga_ls5t_nbart_gm_fyear_3_x50y26_2010-07-01--P1Y_final.odc-metadata.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 58, in dump_to_odc
    index_update_dataset(metadata, uri, dc, doc2ds, update, update_if_exists, allow_unsafe)
  File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 179, in index_update_dataset
    raise IndexingException(
odc.apps.dc_tools.utils.IndexingException: Failed to create dataset with error The dataset is not specifying all of the measurements in this product.
Missing fields are;
{'nbart_swir_1', 'nbart_swir_2', 'nbart_nir', 'nbart_blue', 'nbart_red', 'nbart_green'}
 The URI was s3://dea-public-data-dev/derivative/ga_ls5t_nbart_gm_fyear_3/3-0-0/x50/y26/2010-07-01--P1Y/ga_ls5t_nbart_gm_fyear_3_x50y26_2010-07-01--P1Y_final.odc-metadata.yaml
Added 0 datasets and failed 1035 datasets.

@jmettes jmettes requested review from tebadi, emmaai and omad June 27, 2022 04:27
@codecov-commenter
Copy link

codecov-commenter commented Jun 27, 2022

Codecov Report

Merging #1086 (e6ab154) into master (6e505bc) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1086   +/-   ##
=======================================
  Coverage   99.61%   99.61%           
=======================================
  Files         121      121           
  Lines        1027     1027           
=======================================
  Hits         1023     1023           
  Misses          4        4           

Copy link
Contributor

@emmaai emmaai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change is obsolete, since the bands name of geomedian are changed to nbart_xx in this pr opendatacube/odc-stats#51 and merged.

@jmettes
Copy link
Member Author

jmettes commented Aug 17, 2022

For some further context, testing data of 2010-2011 FY and 2015-2016 FY was produced for the LCCS ABS release. The version of odc-stats used to produce that data proceeded merging of the latest odc-stats which fixed the problem of the difference of geomedian alias names. Because of this, the product definitions in this PR had to be used to index the geomedian products in dev.

2020-2021 FY geomedian data will still need to be produced using the same process for consistency of the LCCS testing and release to ABS.

After testing and products are delivered, we'll need to re-run of all of the financial years using the latest odc-stats. At that point, this branch can be deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants