Make sure reconstruction.log and/or metadata.yaml is copied to downsampled datasets

[dstansby]

No description provided.

[dstansby]

@JosephBrunet is the goal here to put metadata inside the JP2 files, so if users download the JP2 zip files they have a copy of the metadata? If so I think we should just put a copy of the metadata txt file that gets released on the HOA portal, instead of new metadata (ie the .json and .log files).

Or, we should upload the .json and .log files to HOA too if they are important?

[JosephBrunet]

I think having a metadata file is easier than adding to jp2 (but I'm not again the idea). It's easier for user to check quickly metadata
The idea in the end is to merge the metadata.json, the reconstruction.log, and the .txt file into a single json file.
The dataset with json are only from last september so it will be a long time before we need to upload them

I think it is fine to include the reconstruction.log, it is included in all the datasets currently on the HOA website.

But yes in the future we need to start moving toward a single metadata file

[dstansby]

I'm just checking for reconstruction.log files over at https://github.com/HiPCTProject/hipct-data-tools/pull/148 and it looks like some datasets have a meatdata.yml file, but no reconstruction.log. Is this expected? If so, could this be fixed it so all datasets have a reconstruction.log file?

As a general rule we want to make sure that the metdata and format for all datasets is the same. If we make a change to the metadata format or metadata keys, we need to backfill this change to all previous datasets (even if it means empty metadata fields for the older datasets if information is missing). That way we (and anyone who downloads the data) can process all datasets with the same code. cc @Guillaume-Gaisne here since this is to do with metadata.