Bugs when metadata records from different positions/field of views are merged

[mjosch]

Hi @karlduderstadt ,

I noticed some bugs when MARS merges the metadata record for different positions (fields of view) within the same experiment. First of all, most metadata records are not merged at all. However, when they are merged, it seems like Metadata properties are lost and overwritten with subsequent merged ones. For instance, the parameter DnaMoleculeCount in DnaMoleculeArchives keeps being overwritten but in this case it should be summed up. In other cases a different action might be preferred. To keep it simple, maybe metadata merging is not absolutely required and they could kept separate?

[karlduderstadt]

Hi @mjosch

Thanks for pointing out this problem. It seems there are a couple issues here. First, all positions from a single experiment should get the same metadata UID and I guess that didn't happen. This could be a bug in mars-scifio or could be a difference in the metadata records, such that the UUIDs don't match. I will investigate this further. In any case, I think in micromanager 2 they removed the global UUID, so it might be even harder to ensure all positions from an experiment get the same metadata UID.

I completely agree with you. I think by default all metadata records should not be merged, so I will change this. But I do think there might be some special situations when this feature is desired. So I have added a radio button to all image commands that create archives with two options "unique from dataset" or "random" with "random" as the default. I hope this addresses the main issue.

If two metadata records have the same metadata UID, but you don't want them to be merged when running the merge archives command, you will need to change the metadataUID of one of the datasets using the follow script:

#@ MoleculeArchive archive

import de.mpg.biochem.mars.util.*
import org.apache.commons.lang3.reflect.FieldUtils

def metadata = archive.getMetadata(0)

archive.removeMetadata(metadata)
newUID = MarsMath.getUUID58().substring(0, 10)
FieldUtils.writeField(metadata, "uid", newUID, true);
archive.putMetadata(metadata)

archive.molecules().forEach{ molecule ->
	molecule.setMetadataUID(newUID)
}

archive.rebuildIndexes()

The uid is a private variable with no set method by design because it shouldn't typically be changed. This script uses java reflection to force the change. It assumes all molecule in the archive are associated with the same metadata records and also updates those.

Ultimately, the merge archives command should have a checkbox or some options that provide more possibilities, but this will take some time to implement. I think using the script above before merging is a suitable workaround for situations where a lot of analysis has been completed and you don't want two datasets with the same metadata UID to be merged.

If you happen to be in a position where you have two metadata records with different metadataUIDs and you still want to merge them. This can also be done very easily in a script using the merge method.

#@ MoleculeArchive archive1
#@ MoleculeArchive archive2

def metadata1 = archive1.getMetadata(0)
archive2.getMetadata(0).merge(metadata1)

A similar merge method is also available for merging molecule records. As you discovered, when this method is used (it is called in the merge command) last one in wins in terms of parameters, regions, positions, etc... more care is taken when merging molecule tables and metadata record OME images to ensure all information is retained.

I am not sure there is a simpler way to deal with this. I could add a rule for the DnaMoleculeCount and adds them I suppose, but I think not merging is a better solution.

The changes have all been implemented in image commands, but I didn't reference this issue in the commits, sorry. For now I will leave it open until I work on the merge command and add the checkbox.