Enable annotation/image tags import on self-hosted instance

[davidsean]

I would like to move a volume (files, annotations, tags and tree) from biigle.de to a self-hosted instance. I've seen this answered a few times, but cannot figure it out.

The manual and this discussion suggests creating a volume from coco format, but I can only upload CSVs (Select a metadata file) to initialize a volume. I am running the latest (1.4.0) version.

I assume I am missing an environmental variable here?

One option to move volumes between instances is the Export/Import from the biigle/sync module, but it leaves biigle.de out of the picture.
If all else fails, I can always try to loop through the individual objects using the API to sync the two.

[davidsean]

I think I just found the answer by finding the biigle/metadata-coco module

[mzur]

Hi, the easiest would indeed be to use biigle/sync but only the biigle.de admins can create an export file for you. Please write us an email to [email protected] with the project/volumes you want to export and we check if this is feasible.

If you want to do it with the COCO file, make sure to export and import the label tree(s) first. BIIGLE will then be able to automatically link the labels of the imported annotations. You can find the label tree export behind the gear icon of the label tree and the "download" option. Then use the "import" button of the view to create a new label tree.

[davidsean]

Thanks Martin,
Sort of got things working, but I cannot figure out how to transfer the file labels. The COCO report does not seem to contain them. Exporting the file annotations is fine, I`ve written a small script that could loop and POST them using the API which works. But I am looking for a cleaner solution for my users.

If you upload such a metadata file, the form to create a new volume will offer the options to import the annotations and/or file labels for the new volume, too.

I`ve tried initializing a volume with file metadata, but did not get the additional dialogue. The one thing that I've yet to try is the iFDO, which is a bit harder to get started with (I assume you cannot export an iFDO unless the volume was created with one).

I am considering writing a module for combining file metadata+file labels+labeltree (for the file labels) as a zip and handling import export. I am a bit worried about issues relating to database key issues ( image_label_id , user_id, image_id, label_id).
For my usecase, the user_id can be the logged in user (for all the image labels) so that's not a problem. And I am assuming that somehow the label tree zip import/exports don't suffer from have key/id collisions, so is a solution for this. The tricky part is that the module would need to be more than a simple parser, as it needs to run sequential tasks (create volume, create tree, create labels). Which may be a bit too much for me right now....

I think I can hack up an a fake export zip with what I need, and use the biigle/sync module to handle the import. I'll give this a try first.

[mzur]

I`ve tried initializing a volume with file metadata, but did not get the additional dialogue. The one thing that I've yet to try is the iFDO, which is a bit harder to get started with (I assume you cannot export an iFDO unless the volume was created with one).

This is correct. iFDO is currently also the only supported file format that can contain both annotations and file labels.

I am considering writing a module for combining file metadata+file labels+labeltree (for the file labels) as a zip and handling import export. [...]

This is exactly what biigle/sync does! So if you tell me (at [email protected]) which volumes/projects you want to export, I'm sure I can provide you with an export file that you can import into your instance.

The only thing that is missing in the biigle/sync export is additional file metadata (https://github.com/biigle/sync/issues/18). But if you were able to upload these before via a separate metadata file, you can do this again in your instance.

Also you might have to manually update the user UUIDs in the export file if you already have some users with accounts in both instances, which were not created through biigle/sync.