Merge pull request #6 from pwalczysko/metadata

Add metadata import section
ome · Jun 17, 2020 · bcdaa27 · bcdaa27
2 parents d6cdd28 + 74d43a1
commit bcdaa27
Show file tree

Hide file tree

Showing 19 changed files with 371 additions and 30 deletions.
diff --git a/docs/images/metadata1.png b/docs/images/metadata1.png
diff --git a/docs/images/metadata2.png b/docs/images/metadata2.png
diff --git a/docs/images/metadata3.png b/docs/images/metadata3.png
diff --git a/docs/images/metadata3a.png b/docs/images/metadata3a.png
diff --git a/docs/images/metadata4.png b/docs/images/metadata4.png
diff --git a/docs/images/metadata5.png b/docs/images/metadata5.png
diff --git a/docs/images/metadata6.png b/docs/images/metadata6.png
diff --git a/docs/images/metadata7.png b/docs/images/metadata7.png
diff --git a/docs/images/metadata8.png b/docs/images/metadata8.png
diff --git a/docs/images/metadata9.png b/docs/images/metadata9.png
diff --git a/docs/import-cli.rst b/docs/import-cli.rst
@@ -1,8 +1,8 @@
 Import data using the Command Line Interface (CLI)
 ==================================================
 
-Description:
-------------
+Description
+-----------
 
 This chapter will show how to import data for another user, using Command Line Interface (CLI).
 
@@ -18,8 +18,8 @@ We will show:
 
 -  How to deal with imports of large amounts of data in CLI, using the --bulk option and helper csv and yml files which define what is to be imported and how.
 
-**Resources:**
---------------
+Resources
+---------
 
 -  Documentation:
 
@@ -45,8 +45,8 @@ We will show:
 
    -  :download:`bulk.yml <../scripts/bulk.yml>`
 
-Setup:
-------
+Setup
+-----
 
 **CLI Importer installation**
 
@@ -56,8 +56,8 @@ found at \ https://docs.openmicroscopy.org/latest/omero/users/cli/installation.h
 Note: When importing for another user using the CLI, the importer1 does not have to be a member of the target group.
 
 
-**Step-by-step:**
------------------
+Step-by-step
+------------
 
 #.  Open a terminal and connect to the server as importer1 using ``ssh``.
 

diff --git a/docs/import-desktop-client.rst b/docs/import-desktop-client.rst
@@ -1,8 +1,8 @@
 Import data using the Desktop Client
 ====================================
 
-Description:
-------------
+Description
+-----------
 
 In the first part, we first show how to import data by yourself and for yourself into OMERO using various import strategies. This will be mainly done using the OMERO.insight desktop client.
 
@@ -23,8 +23,8 @@ We will show:
 
 -  How to import data for other users in OMERO.insight.
 
-Setup:
-------
+Setup
+-----
 
 OMERO.insight desktop client installation instructions:.
 
@@ -79,8 +79,8 @@ Linux
 
    .. image:: images/importinsight8.png
 
-Step-by-step:
--------------
+Step-by-step
+------------
 
 #. Open OMERO.insight and in the login dialog, click on the wrench icon \ |image8a|
 

diff --git a/docs/import-dropbox.rst b/docs/import-dropbox.rst
@@ -1,8 +1,8 @@
 Import data using OMERO.dropbox
 ===============================
 
-**Description:**
-----------------
+Description
+-----------
 
 OMERO.dropbox allows to import files into OMERO automatically, by means
 of offline import from a watched directory. Typically, each user has a
@@ -21,8 +21,8 @@ We will show
 
 -  How to import several files for two different users using OMERO.dropbox. The image files are manually copied into the prepared folders watched by OMERO.dropbox.
 
-**Resources:**
---------------
+Resources
+---------
 
 -  Documentation:
 
@@ -32,8 +32,8 @@ We will show
 
 -  Data: example images from \ https://downloads.openmicroscopy.org/images/DV/alexia/cajal-bodies/
 
-**Setup:**
-----------
+Setup
+-----
 
 This example setup will help you to understand OMERO.dropbox functionality. Later, you can choose the setup you like for your OMERO.server studying detailed instructions at \ https://docs.openmicroscopy.org/latest/omero/sysadmins/dropbox.html#advanced-use
 
@@ -74,8 +74,8 @@ Below are the installation instructions.
 
 -  ``$ mkdir user-2``
 
-Step-by-step:
--------------
+Step-by-step
+------------
 
 #. Open a browser window.
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -8,3 +8,4 @@ Contents:
 
    import
    render
+   metadata
diff --git a/docs/metadata.rst b/docs/metadata.rst
@@ -0,0 +1,165 @@
+Import metadata using the Command Line Interface (CLI)
+======================================================
+
+Description
+-----------
+
+This chapter will show how to import metadata starting from a local CSV file and ending with OMERO.tables on images or Key-Value pairs on images using the Command Line Interface (CLI) and the OMERO server-side scripts.
+
+This action is typically done after a successful import of images.
+
+We will show:
+
+- How to import metadata from local CSV file in a bulk manner and turn them into OMERO.tables on images using CLI
+
+- How to turn the OMERO.tables on images into Key-Value pairs on images in bulk manner using CLI
+
+- How to import metadata from local CSV file and use a server-side script in OMERO to turn these into OMERO.tables on images
+
+- How to construct a simple file to turn the metadata stored in OMERO.tables into Key-Value pairs on images using CLI
+
+Resources
+---------
+
+-  Documentation:
+
+   -  `CLI installation <https://docs.openmicroscopy.org/omero/latest/users/cli/installation.html>`_
+
+   -  `CLI <https://docs.openmicroscopy.org/omero/latest/users/cli/index.html>`__
+
+-  Data: example images from
+
+   -  IDR data `idr0021-lawo-pericentriolarmaterial/experimentA <https://idr.openmicroscopy.org/webclient/?show=project-51>`_
+   -  `siRNAi-HeLa <https://downloads.openmicroscopy.org/images/DV/siRNAi-HeLa/>`_ dataset
+
+-  Metadata plugin for OMERO
+
+   - https://pypi.org/project/omero-metadata/
+
+-  Bulkmap config yml files defining the various Key-Value pairs parameters, such as the groups and other parameters.
+
+   - `idr0021-experimentA-bulkmap-config.yml <https://github.com/IDR/idr0021-lawo-pericentriolarmaterial/blob/9479af85f19487f215e3dfdd31a1b587370ed3cf/experimentA/idr0021-experimentA-bulkmap-config.yml>`_
+   - :download:`simple-annotation-bulkmap-config.yml <../scripts/simple-annotation-bulkmap-config.yml>`
+
+-  Annotation CSV files define the content of OMERO.tables for each image.
+
+   - `idr0021-experimentA-annotation.csv <https://github.com/IDR/idr0021-lawo-pericentriolarmaterial/blob/9479af85f19487f215e3dfdd31a1b587370ed3cf/experimentA/idr0021-experimentA-annotation.csv>`_
+   - :download:`simple-annotation.csv <../scripts/simple-annotation.csv>`
+   - :download:`four-images.csv <../scripts/four-images.csv>`
+
+Setup
+-----
+
+**Metadata plugin installation**
+
+- Go to the environment where you installed your OMERO.cli as specified under - `CLI installation <https://docs.openmicroscopy.org/omero/latest/users/cli/installation.html>`_.
+
+- Activate the virtual environment.
+
+- Run::
+    
+    $ pip install omero-metadata
+
+Step-by-step
+------------
+
+#.  On your local machine, open a terminal
+
+#.  If you did not do so already, activate the virtual environment where ``omero-py`` is installed or add it to ``PATH`` e.g.::
+
+    $ export PATH=/opt/omero/server/venv3/bin:$PATH
+
+#.  Download the CSV from `idr0021-experimentA-annotation.csv <https://github.com/IDR/idr0021-lawo-pericentriolarmaterial/blob/9479af85f19487f215e3dfdd31a1b587370ed3cf/experimentA/idr0021-experimentA-annotation.csv>`_ if you have access to the `idr0021 <https://idr.openmicroscopy.org/webclient/?show=project-51>`_ data  in your OMERO.server. Alternatively, download :download:`simple-annotation.csv <../scripts/simple-annotation.csv>`, which will allow you to work with the `siRNAi-HeLa <https://downloads.openmicroscopy.org/images/DV/siRNAi-HeLa/>`_ dataset.
+
+#.  The variable ``$ID`` below is the ID of the Project, in this example case it is the Project containing the idr0021 study. If you are working with the siRNAi-HeLa data, replace in the following example the "Project" with a "Dataset" and the ``idr0021-experimentA-annotation.csv`` with ``simple-annotation.csv``. To add annotations from a local CSV file to the images in the said Project or Dataset in the form of OMERO.tables, run::
+    
+    $ omero metadata populate --report --batch 1000 --file local/path/to/idr0021-experimentA-annotation.csv Project:$ID
+
+#.  Open your browser and login to the OMERO.web. Navigate to the Project or Dataset you just worked with, expand the "Attachments" harmonica in the right-hand pane and verify that a new attachment is on that Project named ``bulk_annotations``.
+
+    |image0|
+
+#.  You can inspect its content by clicking on the "eye" icon |image1| inside the annotation.
+
+#.  Select an image inside the Project/Dataset and expand the "Tables" harmonica in the right-hand pane. These tables contain the appropriate line from the ``bulk_annotations`` attachment you just created for that particular image.
+
+    |image2|
+
+#.  Go back to your terminal. Download the `idr0021-experimentA-bulkmap-config.yml <https://github.com/IDR/idr0021-lawo-pericentriolarmaterial/blob/9479af85f19487f215e3dfdd31a1b587370ed3cf/experimentA/idr0021-experimentA-bulkmap-config.yml>`_ file . Alternatively, in case you are working with the siRNAi-HeLa Dataset, download :download:`simple-annotation-bulkmap-config.yml <../scripts/simple-annotation-bulkmap-config.yml>`.
+
+#.  If you are working with the IDR data, open the downloaded ``idr0021-experimentA-bulkmap-config.yml`` file in a text editor and delete the ``Advanced options...`` section. Save the file and run::
+
+    $ omero metadata populate --context bulkmap --cfg local/path/to/idr0021-experimentA-bulkmap-config.yml --batch 100 Project:$ID
+
+#.  If you work with the siRNAi-HeLa data, open the downloaded ``simple-annotation-bulkmap-config.yml`` and study the comments in the file itself, which will give you hints about how to manipulate the file to fit your particular needs with respect to the resulting Key-Value pairs layout. Make your changes (no need to change anything if you do not want), save the file locally and run::
+
+    $ omero metadata populate --context bulkmap --cfg local/path/to/simple-annotation-bulkmap-config.yml --batch 100 Dataset:$ID
+
+#.  Go to your browser and in OMERO.web, select the images in the Project or Dataset you targeted and verify that they have now new Key-Value pairs displayed in the right-hand pane.
+
+    |image3a|
+
+#.  Still in OMERO.web, create a new Dataset and copy into it four images, preferably images which have neither OMERO.tables on them nor any Key-Value pairs attached. Note the name of the images you are copying in.
+
+    |image4|
+
+#.  Go to the https://pypi.org/project/omero-metadata/ and find the section named ``populate``. Study the ``project.csv``. You can either take the ``project.csv`` file from there, or more conveniently, you can download directly its copy :download:`four-images.csv <../scripts/four-images.csv>`. Open the CSV file in Excel and edit the name of the images in the first column to match the names of the images you copied into your Dataset in the previous step. Also, edit the name of the Dataset in the second column to match the name of your Dataset in OMERO.web. Save the file locally as CSV.
+
+#.  In your OMERO.web, upload the CSV file you just saved and attach it onto the Dataset you created previously.
+
+    |image5|
+
+#.  Select the Dataset you created and attached to it the CSV. Find the script icon |image6| above the central pane, expand it and find the ``Import scripts`` section. In there, select the ``Populate metadata`` script.
+
+    |image7| 
+
+#.  Run the script.
+
+#.  Click again onto the Dataset in the left-hand pane to refresh and observe that there is a new Attachment in the right hand pane under "Attachments" harmonica. 
+
+    |image8|
+
+#.  Click on single images inside the Dataset and observe that in the "Tables" harmonica in the right-hand pane there are new values coming originally from your edited CSV.
+
+    |image9|
+
+
+.. |image0| image:: images/metadata1.png
+   :width: 4in
+   :height: 1in
+
+.. |image1| image:: images/metadata2.png
+   :width: 0.35in
+   :height: 0.3in
+
+.. |image2| image:: images/metadata3.png
+   :width: 4in
+   :height: 3.5in
+
+.. |image3a| image:: images/metadata3a.png
+   :width: 4in
+   :height: 3.3in
+
+.. |image4| image:: images/metadata4.png
+   :width: 5in
+   :height: 1.5in
+
+.. |image5| image:: images/metadata5.png
+   :width: 4in
+   :height: 1in
+
+.. |image6| image:: images/metadata6.png
+   :width: 0.35in
+   :height: 0.3in
+
+.. |image7| image:: images/metadata7.png
+   :width: 2in
+   :height: 0.7in
+
+.. |image8| image:: images/metadata8.png
+   :width: 4in
+   :height: 1.3in
+
+.. |image9| image:: images/metadata9.png
+   :width: 4in
+   :height: 2in
diff --git a/docs/render.rst b/docs/render.rst
@@ -1,8 +1,8 @@
 Change image rendering settings and channel names using the Command Line Interface (CLI)
 ========================================================================================
 
-Description:
-------------
+Description
+-----------
 
 This chapter will show how to change rendering settings on images using the Command Line Interface (CLI).
 
@@ -12,8 +12,8 @@ We will show:
 
 -  How to change rendering settings of large amount of images on the CLI in a repeatable manner.
 
-**Resources:**
---------------
+Resources
+---------
 
 -  Documentation:
 
@@ -42,8 +42,8 @@ We will show:
 
    - https://github.com/ome/training-scripts/blob/master/maintenance/scripts/apply_rnd_settings_as.sh
 
-Setup:
-------
+Setup
+-----
 
 **Rendering plugin installation**
 
@@ -55,8 +55,8 @@ Setup:
     
     $ pip install omero-cli-render
 
-**Step-by-step:**
------------------
+Step-by-step
+------------
 
 #.  On your local machine, open a terminal
 

diff --git a/scripts/four-images.csv b/scripts/four-images.csv
@@ -0,0 +1,6 @@
+# header s,s,d,l,s
+Image Name,Dataset Name,ROI_Area,Channel_Index,Channel_Name
+Centrin_PCNT_Cep215_20110506_Fri-1608_0_SIR_PRJ.dv,dataset01,0.0469,1,DAPI
+Cep215_PCNT_gtub_20110506_Fri-1721_0_SIR_PRJ.dv,dataset01,0.142,2,GFP
+Cep215_PCNT_NEDD1_20110505_Thu-1403_0_SIR_PRJ.dv,dataset01,0.093,3,TRITC
+siControl_N20_Cep215_I_20110411_Mon-1509_0_SIR_PRJ.dv,dataset01,0.429,4,Cy5