From 4b0c1e8b70dc2551514a273a64057151029a1f6a Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Thu, 7 Apr 2022 13:14:20 +0100
Subject: [PATCH 01/24] Updating README
Updating README.rst to include the changes to default behaviour
---
README.rst | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/README.rst b/README.rst
index 1e568745..513df1fb 100644
--- a/README.rst
+++ b/README.rst
@@ -70,8 +70,12 @@ object IDs in the ``OMERO.table``.
The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``.
-If you wish to ensure that ``number`` columns are created for numerical data, this will
-allow you to make numerical queries on the table.
+
+
+Automatic header
+^^^^^^^^^
+
+**The default behaviour of the script is to automatically detect the column types and specific object types from an input ``CSV`` using the list below.**
Column Types are:
- ``d``: ``DoubleColumn``, for floating point numbers
@@ -80,8 +84,14 @@ Column Types are:
- ``b``: ``BoolColumn``, for true/false
- ``plate``, ``well``, ``image``, ``dataset``, ``roi`` to specify objects
-These can be specified in the first row of a ``CSV`` with a ``# header`` tag (see examples below).
-The ``# header`` row is optional. Default column type is ``String``.
+
+
+Manual Header
+^^^^^^^^^
+
+However, it is possible to override the default behaviour, ignoring the automatic header detection, and manually assign the header to define the column type if a ``CSV`` with with a ``# header`` tag is passed (see examples below).
+
+Automatic header detection can also be ignored if using the ``--manual_headers`` flag. If the ``# header`` is not present and this flag is used, column types will default to ``String``
NB: Column names should not contain spaces if you want to be able to query
by these columns.
From 115dcbf77fca9a2c31da8286acadbd2fef05709a Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Thu, 7 Apr 2022 13:16:08 +0100
Subject: [PATCH 02/24] Clearer titles
---
README.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.rst b/README.rst
index 513df1fb..c4347203 100644
--- a/README.rst
+++ b/README.rst
@@ -72,7 +72,7 @@ The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``
-Automatic header
+Automatic Column Types
^^^^^^^^^
**The default behaviour of the script is to automatically detect the column types and specific object types from an input ``CSV`` using the list below.**
@@ -86,7 +86,7 @@ Column Types are:
-Manual Header
+Manual Column Types
^^^^^^^^^
However, it is possible to override the default behaviour, ignoring the automatic header detection, and manually assign the header to define the column type if a ``CSV`` with with a ``# header`` tag is passed (see examples below).
From 0d291cc2263e0f5145c6a82c64b902b4bb13047e Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 8 Apr 2022 11:04:31 +0100
Subject: [PATCH 03/24] Fixed grammer. Added more examples.
Fixed grammer. Added more examples for automatic header detection. Added linkable titles to some headings
---
README.rst | 64 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 46 insertions(+), 18 deletions(-)
diff --git a/README.rst b/README.rst
index c4347203..321747f6 100644
--- a/README.rst
+++ b/README.rst
@@ -71,10 +71,6 @@ object IDs in the ``OMERO.table``.
The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``.
-
-Automatic Column Types
-^^^^^^^^^
-
**The default behaviour of the script is to automatically detect the column types and specific object types from an input ``CSV`` using the list below.**
Column Types are:
@@ -85,25 +81,25 @@ Column Types are:
- ``plate``, ``well``, ``image``, ``dataset``, ``roi`` to specify objects
+However, it is possible to manually define the column types , ignoring the automatic header detection, if a ``CSV`` with a ``# header`` row is passed (see examples below).
-Manual Column Types
-^^^^^^^^^
-
-However, it is possible to override the default behaviour, ignoring the automatic header detection, and manually assign the header to define the column type if a ``CSV`` with with a ``# header`` tag is passed (see examples below).
-
-Automatic header detection can also be ignored if using the ``--manual_headers`` flag. If the ``# header`` is not present and this flag is used, column types will default to ``String``
+Automatic header detection can also be ignored if using the ``--manual_headers`` flag. If the ``# header`` is not present and this flag is used, column types will default to ``String`` (unless the column names correspond to OMERO objects such as ``image`` or ``plate``).
NB: Column names should not contain spaces if you want to be able to query
by these columns.
+Examples
+^^^^^^^^^
+
**Project / Dataset**
+^^^^^^^^^
To add a table to a Project, the ``CSV`` file needs to specify ``Dataset Name``
and ``Image Name`` or ``Image ID``::
$ omero metadata populate Project:1 --file path/to/project.csv
-
-project.csv::
+
+project.csv (manual column types definition)::
# header s,s,d,l,s
Image Name,Dataset Name,ROI_Area,Channel_Index,Channel_Name
@@ -112,7 +108,15 @@ project.csv::
img-03.png,dataset01,0.093,3,TRITC
img-04.png,dataset01,0.429,4,Cy5
-This will create an OMERO.table linked to the Project like this with
+project.csv (automatic column types detection)::
+
+ Image Name,Dataset Name,ROI_Area,Channel_Index,Channel_Name
+ img-01.png,dataset01,0.0469,1,DAPI
+ img-02.png,dataset01,0.142,2,GFP
+ img-03.png,dataset01,0.093,3,TRITC
+ img-04.png,dataset01,0.429,4,Cy5
+
+Both manual definition or automatic detection of column types will create an OMERO.table linked to the Project as folows with
a new ``Image`` column with IDs:
========== ============ ======== ============= ============ =====
@@ -128,11 +132,14 @@ If the target is a Dataset instead of a Project, the ``Dataset Name`` column is
**Screen / Plate**
+^^^^^^^^^
To add a table to a Screen, the ``CSV`` file needs to specify ``Plate`` name and ``Well``.
-If a ``# header`` is specified, column types must be ``well`` and ``plate``.
+If a ``# header`` is specified, column types must be ``well`` and ``plate``::
-screen.csv::
+ $ omero metadata populate Screen:1 --file path/to/screen.csv
+
+screen.csv (manual column types definition)::
# header well,plate,s,d,l,d
Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
@@ -141,7 +148,15 @@ screen.csv::
A3,plate01,DMSO,5.5,550,4
B1,plate01,DrugX,12.3,50,44.43
-This will create an OMERO.table linked to the Screen, with the
+screen.csv (automatic column types detection)::
+
+ Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
+ A1,plate01,DMSO,10.1,10,25.4
+ A2,plate01,DMSO,0.1,1000,2.54
+ A3,plate01,DMSO,5.5,550,4
+ B1,plate01,DrugX,12.3,50,44.43
+
+Similarly, this will create an OMERO.table linked to the Screen, with the
``Well Name`` and ``Plate Name`` columns added and the ``Well`` and
``Plate`` columns used for IDs:
@@ -157,6 +172,7 @@ Well Plate Drug Concentration Cell_Count Percent_Mitotic Well Name Plat
If the target is a Plate instead of a Screen, the ``Plate`` column is not needed.
**ROIs**
+^^^^^^^^^
If the target is an Image or a Dataset, a ``CSV`` with ROI-level or Shape-level data can be used to create an
``OMERO.table`` (bulk annotation) as a ``File Annotation`` linked to the target object.
@@ -168,9 +184,11 @@ NB: Columns of type ``shape`` aren't yet supported on the OMERO.server.
Alternatively, if the target is an Image, the ROI input column can be
``Roi Name`` (with type ``s``), and an ``roi`` type column will be appended containing ROI IDs.
-In this case, it is required that ROIs on the Image in OMERO have the ``Name`` attribute set.
+In this case, it is required that ROIs on the Image in OMERO have the ``Name`` attribute set::
-image.csv::
+ $ omero metadata populate Image:1 --file path/to/image.csv
+
+image.csv (manual column types definition)::
# header roi,l,l,d,l
Roi,shape,object,probability,area
@@ -179,6 +197,16 @@ image.csv::
503,1068,3,0.2,25
503,1069,4,0.8,400
503,1070,5,0.5,200
+
+
+image.csv (automatic column types detection)::
+
+ Roi,shape,object,probability,area
+ 501,1066,1,0.8,250
+ 502,1067,2,0.9,500
+ 503,1068,3,0.2,25
+ 503,1069,4,0.8,400
+ 503,1070,5,0.5,200
This will create an OMERO.table linked to the Image like this:
From 494314a04b5b1f38eb3076b03edd740fd99a5423 Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 8 Apr 2022 14:46:58 +0100
Subject: [PATCH 04/24] Improved clarify
---
README.rst | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/README.rst b/README.rst
index 321747f6..370597cf 100644
--- a/README.rst
+++ b/README.rst
@@ -70,19 +70,20 @@ object IDs in the ``OMERO.table``.
The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``.
+OMERO.tables have defined column types to specify the data-type such as `double` or `long` and special object-types of each column for storing OMERO object IDs such as ``ImageColumn`` or ``WellColumn``
-**The default behaviour of the script is to automatically detect the column types and specific object types from an input ``CSV`` using the list below.**
-Column Types are:
+**The default behaviour of the script is to automatically detect these column types from an input ``CSV``**. This behaviour works as folows:
+* Columns named with a supported object-type (e.g. 'image', 'plate'...) or with object_id (e.g. 'image_id', 'dataset_id' ) will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
+* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
+
+However, it is possible to manually define the column types , ignoring the automatic header detection, if a ``CSV`` with a ``# header`` row is passed. The ``# header`` row should be the first row of the CSV and defines columns according to the following list (see examples below):
- ``d``: ``DoubleColumn``, for floating point numbers
- ``l``: ``LongColumn``, for integer numbers
- ``s``: ``StringColumn``, for text
- ``b``: ``BoolColumn``, for true/false
- ``plate``, ``well``, ``image``, ``dataset``, ``roi`` to specify objects
-
-However, it is possible to manually define the column types , ignoring the automatic header detection, if a ``CSV`` with a ``# header`` row is passed (see examples below).
-
Automatic header detection can also be ignored if using the ``--manual_headers`` flag. If the ``# header`` is not present and this flag is used, column types will default to ``String`` (unless the column names correspond to OMERO objects such as ``image`` or ``plate``).
NB: Column names should not contain spaces if you want to be able to query
From 70a1c92a795759342cf4f98fe9d6c03ba85a0a19 Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 8 Apr 2022 15:55:37 +0100
Subject: [PATCH 05/24] Fixed formatting
---
README.rst | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/README.rst b/README.rst
index 370597cf..02bb4455 100644
--- a/README.rst
+++ b/README.rst
@@ -73,8 +73,9 @@ The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``
OMERO.tables have defined column types to specify the data-type such as `double` or `long` and special object-types of each column for storing OMERO object IDs such as ``ImageColumn`` or ``WellColumn``
**The default behaviour of the script is to automatically detect these column types from an input ``CSV``**. This behaviour works as folows:
-* Columns named with a supported object-type (e.g. 'image', 'plate'...) or with object_id (e.g. 'image_id', 'dataset_id' ) will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
-* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
+
+* Columns named with a supported object-type (e.g. 'image', 'plate'...) or with object_id (e.g. 'image_id', 'dataset_id' ) will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
+* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
However, it is possible to manually define the column types , ignoring the automatic header detection, if a ``CSV`` with a ``# header`` row is passed. The ``# header`` row should be the first row of the CSV and defines columns according to the following list (see examples below):
From 2bc1a06aafd76f143d0c246e70ba940720e0df5e Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 8 Apr 2022 15:57:41 +0100
Subject: [PATCH 06/24] Added full list for supported object-type
---
README.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.rst b/README.rst
index 02bb4455..91830e7a 100644
--- a/README.rst
+++ b/README.rst
@@ -74,7 +74,7 @@ OMERO.tables have defined column types to specify the data-type such as `double`
**The default behaviour of the script is to automatically detect these column types from an input ``CSV``**. This behaviour works as folows:
-* Columns named with a supported object-type (e.g. 'image', 'plate'...) or with object_id (e.g. 'image_id', 'dataset_id' ) will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
+* Columns named with a supported object-type ('plate', 'well', 'image', 'dataset', or 'roi') or with object_id (e.g. 'image_id', 'dataset_id') will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
From f6610a20bbb6914c5843474f2e80d46b9bf8c92d Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 15 Apr 2022 14:21:03 +0100
Subject: [PATCH 07/24] Updating linting
---
README.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.rst b/README.rst
index 91830e7a..d8f33849 100644
--- a/README.rst
+++ b/README.rst
@@ -79,6 +79,7 @@ OMERO.tables have defined column types to specify the data-type such as `double`
However, it is possible to manually define the column types , ignoring the automatic header detection, if a ``CSV`` with a ``# header`` row is passed. The ``# header`` row should be the first row of the CSV and defines columns according to the following list (see examples below):
+
- ``d``: ``DoubleColumn``, for floating point numbers
- ``l``: ``LongColumn``, for integer numbers
- ``s``: ``StringColumn``, for text
From 0cc977f9c4d2309b8712f2ef20ad93cba68295f2 Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Fri, 15 Apr 2022 16:00:17 +0100
Subject: [PATCH 08/24] Fixed Title underline too short
---
README.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.rst b/README.rst
index d8f33849..9f4aa6a8 100644
--- a/README.rst
+++ b/README.rst
@@ -95,7 +95,7 @@ Examples
^^^^^^^^^
**Project / Dataset**
-^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^
To add a table to a Project, the ``CSV`` file needs to specify ``Dataset Name``
and ``Image Name`` or ``Image ID``::
@@ -135,7 +135,7 @@ If the target is a Dataset instead of a Project, the ``Dataset Name`` column is
**Screen / Plate**
-^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^
To add a table to a Screen, the ``CSV`` file needs to specify ``Plate`` name and ``Well``.
If a ``# header`` is specified, column types must be ``well`` and ``plate``::
From b3610bcad0ea9cf58543eec14be9b9fe0f2d5b89 Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Mon, 18 Apr 2022 16:05:05 +0100
Subject: [PATCH 09/24] removed double star
Co-authored-by: pwalczysko
---
README.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.rst b/README.rst
index 9f4aa6a8..c154563d 100644
--- a/README.rst
+++ b/README.rst
@@ -72,7 +72,7 @@ The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``
OMERO.tables have defined column types to specify the data-type such as `double` or `long` and special object-types of each column for storing OMERO object IDs such as ``ImageColumn`` or ``WellColumn``
-**The default behaviour of the script is to automatically detect these column types from an input ``CSV``**. This behaviour works as folows:
+The default behaviour of the script is to ``automatically detect these column types`` from an input ``CSV``. This behaviour works as folows:
* Columns named with a supported object-type ('plate', 'well', 'image', 'dataset', or 'roi') or with object_id (e.g. 'image_id', 'dataset_id') will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
From bf24a6b7c1ffabe231e9da83c0e3e0f5b30f35eb Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Mon, 18 Apr 2022 16:14:12 +0100
Subject: [PATCH 10/24] Removed extra backticks
---
README.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.rst b/README.rst
index c154563d..bfac6f80 100644
--- a/README.rst
+++ b/README.rst
@@ -70,9 +70,9 @@ object IDs in the ``OMERO.table``.
The ``CSV`` file must be provided as local file with ``--file path/to/file.csv``.
-OMERO.tables have defined column types to specify the data-type such as `double` or `long` and special object-types of each column for storing OMERO object IDs such as ``ImageColumn`` or ``WellColumn``
+OMERO.tables have defined column types to specify the data-type such as ``double`` or ``long`` and special object-types of each column for storing OMERO object IDs such as ``ImageColumn`` or ``WellColumn``
-The default behaviour of the script is to ``automatically detect these column types`` from an input ``CSV``. This behaviour works as folows:
+The default behaviour of the script is to automatically detect the column types from an input ``CSV``. This behaviour works as folows:
* Columns named with a supported object-type ('plate', 'well', 'image', 'dataset', or 'roi') or with object_id (e.g. 'image_id', 'dataset_id') will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
From fcbc1c8a63fad20aec91560bbc93bc948b87816e Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Thu, 12 May 2022 11:12:27 +0100
Subject: [PATCH 11/24] Added dataset id example
---
README.rst | 34 ++++++++++++++++++++++++++++++++--
1 file changed, 32 insertions(+), 2 deletions(-)
diff --git a/README.rst b/README.rst
index bfac6f80..20fd176a 100644
--- a/README.rst
+++ b/README.rst
@@ -97,7 +97,7 @@ Examples
**Project / Dataset**
^^^^^^^^^^^^^^^^^^^^^^
-To add a table to a Project, the ``CSV`` file needs to specify ``Dataset Name``
+To add a table to a Project, the ``CSV`` file needs to specify ``Dataset Name`` or ``Dataset ID``
and ``Image Name`` or ``Image ID``::
$ omero metadata populate Project:1 --file path/to/project.csv
@@ -131,8 +131,38 @@ img-03.png dataset01 0.093 3 TRITC 36640
img-04.png dataset01 0.429 4 Cy5 36641
========== ============ ======== ============= ============ =====
-If the target is a Dataset instead of a Project, the ``Dataset Name`` column is not needed.
+Example using ``Image ID`` and ``Dataset ID``.
+
+project.csv (manual column types definition)::
+
+ # header image,dataset,d,l,s
+ image id,Dataset ID,ROI_Area,Channel_Index,Channel_Name
+ 36638,101,0.0469,1,DAPI
+ 36639,101,0.142,2,GFP
+ 36640,101,0.093,3,TRITC
+ 36641,101,0.429,4,Cy5
+
+project.csv (automatic column types detection)::
+ image id,Dataset ID,ROI_Area,Channel_Index,Channel_Name
+ 36638,101,0.0469,1,DAPI
+ 36639,101,0.142,2,GFP
+ 36640,101,0.093,3,TRITC
+ 36641,101,0.429,4,Cy5
+
+Both manual definition or automatic detection of column types will create an OMERO.table linked to the Project as folows with
+a new ``Image`` column with Names:
+
+===== ======= ======== ============= ============ ==========
+Image Dataset ROI_Area Channel_Index Channel_Name Image Name
+===== ======= ======== ============= ============ ==========
+36638 101 0.0469 1 DAPI img-01.png
+36639 101 0.142 2 GFP img-02.png
+36640 101 0.093 3 TRITC img-03.png
+36641 101 0.429 4 Cy5 img-04.png
+===== ======= ======== ============= ============ ==========
+
+If the target is a Dataset instead of a Project, the ``Dataset Name`` column is not needed.
**Screen / Plate**
^^^^^^^^^^^^^^^^^^^
From 6c5500b60b648e3f3f7de86e01af53d18df22bb5 Mon Sep 17 00:00:00 2001
From: Muhanad Zahra <86613209+muhanadz@users.noreply.github.com>
Date: Thu, 12 May 2022 11:38:28 +0100
Subject: [PATCH 12/24] fixing language
---
README.rst | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/README.rst b/README.rst
index 20fd176a..c157d3b5 100644
--- a/README.rst
+++ b/README.rst
@@ -74,8 +74,8 @@ OMERO.tables have defined column types to specify the data-type such as ``double
The default behaviour of the script is to automatically detect the column types from an input ``CSV``. This behaviour works as folows:
-* Columns named with a supported object-type ('plate', 'well', 'image', 'dataset', or 'roi') or with object_id (e.g. 'image_id', 'dataset_id') will generate the corresponding column type in the OMERO.table. e.g (ImageColumn, PlateColumn, DatasetColumn, etc)
-* Other column types will be detected based on the column's data using the pandas library (e.g. column of data type double).
+* Columns named with a supported object-type (e.g. 'plate', 'well', 'image', 'dataset', or 'roi'), with