populate command fails if the target is a plate and the CSV contains a plate column

[pwalczysko]

When the following command is ueed with a csv as specified below, it ends in error

omero metadata populate --context csv --file simple-screen.csv Plate:51
Using session for trainer-1@localhost:4064. Idle timeout: 10 min. Current group: Lab1
Traceback (most recent call last):
  File "/opt/omero/server/venv3/bin/omero", line 118, in <module>
    rv = omero.cli.argv()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1754, in argv
    cli.invoke(args[1:])
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1187, in invoke
    stop = self.onecmd(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1264, in onecmd
    self.execute(line, previous_args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/cli.py", line 1346, in execute
    args.func(args)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/cli.py", line 505, in populate
    ctx.parse()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1107, in parse
    self.preprocess_from_handle(f1)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1049, in preprocess_from_handle
    self.preprocess_data(reader)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1123, in preprocess_data
    column, original_value, row)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_metadata/populate.py", line 448, in resolve
    return self.wrapper.resolve_plate(column, row, value)
AttributeError: 'PlateWrapper' object has no attribute 'resolve_plate'

The same command where the target object is a Screen containing the above Plate is successful.

Suggestion: Make the plugin work in both situations, the present behaviour is too strict.

cc @will-moore @sbesson

The used CSV

# header well,plate,s,d,l,d
Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
A1,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,10.1,10,25.4
A2,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,0.1,1000,2.54
A3,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,5.5,550,4
B1,INMAC384-DAPI-CM-eGFP_59223_1,DrugX,12.3,50,44.43

[pwalczysko]

cc @will-moore @sbesson

[sbesson]

I am not sure I agree the current behavior is too strict.

As it stands, plate, well, image, dataset and roi are reserved column names. The expectation of the API is that if specified the values of these columns must be resolvable into objects found in the hierarchy of the target container.

Supporting the scenario above would require making a decision about whether these columns should be ignored (treated as standard columns) or checked. Looking at a slightly more complex example than the one above, what should be the behavior if a plate was annotated with as CSV as follows?

# header well,plate,s,d,l,d
Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
A1,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,10.1,10,25.4
A2,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,0.1,1000,2.54
A3,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,5.5,550,4
B1,INMAC384-DAPI-CM-eGFP_59223_1,DrugX,12.3,50,44.4
A1,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,10.1,10,25.4
A2,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,0.1,1000,2.54
A3,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,5.5,550,4
B1,INMAC384-DAPI-CM-eGFP_59223_2,DrugX,12.3,50,44.4

In that case, should Plate be ignored, checked against the

[pwalczysko]

what should be the behavior if a plate was annotated with as CSV as follows?

I would suggest throw an error complaining about non-uniqueness of the well names.

I guess this is already what the same plugin does if I try to pass a CSV on a Dataset with non-unique image names in it ? See below:

# header s,s,d,l,s
Image Name,Dataset Name,ROI_Area,Channel_Index,Channel_Name
Centrin_PCNT_Cep215_20110506_Fri-1608_0_SIR_PRJ.dv,dataset01,0.0469,1,DAPI
Cep215_PCNT_gtub_20110506_Fri-1721_0_SIR_PRJ.dv,dataset01,0.142,2,GFP
Cep215_PCNT_NEDD1_20110505_Thu-1403_0_SIR_PRJ.dv,dataset01,0.093,3,TRITC
Centrin_PCNT_Cep215_20110506_Fri-1608_0_SIR_PRJ.dv,dataset01,0.429,4,Cy5

The confusion here is caused by the apparent discrepancy between the behaviour of this plugin when run on P/D/I as opposed to when run on SPW. The discrepancy starts with the constraint on the SPW case where the well,plate... columns are compulsory, whereas the corresponding PDI case has no such demands on the CSV with regard to the corresponding image,dataset... columns, as the example above shows.

Edit: Maybe to try to specify the behaviour in the highlighted case more precisely:

In that case, should Plate be ignored

If the target object is a Plate, then the Plate column should be ignored and the Well column should be insisted on having unique names. If the target object is a Screen, then the Plate column should be taken into account and the Well name uniqueness is not to be insisted upon.

[abhamacher]

I absolutely agree with @pwalczysko since this error is pretty annoying for me as a user. The "plate" column should be just ignored when populating to a plate and not throw the above mentioned error.

[JensWendt]

Hello,

I experienced the same issue as @pwalczysko that I cannot process a .csv with a plate column if I run the script on a plate.
Unfortunately I also cannot process it, as described in the original post, if I run the script with same .csv on a screen containing said plate.
I tried to delete the plate column in the .csv (also according to the Note panel at the end of the How-To) and run the script again on the plate.
Now I get an error message like this:

Traceback (most recent call last):
File "./script", line 192, in run_script()
File "./script", line 184, in run_script
message = populate_metadata(client, conn, script_params)
File "./script", line 138, in populate_metadata
ctx.parse_from_handle_stream(f2)
File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1093, in parse_from_handle_stream
table = self.create_table()
File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1110, in create_table
{'omero.group': native_str(group)})
File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_SharedResources_ice.py", line 393, in newTable
return _M_omero.grid.SharedResources._op_newTable.invoke(self, ((repoId, path), _ctx))
omero.InternalException: exception ::omero::InternalException
{
serverStackTrace =
serverExceptionClass =
message = null table as argument
}

I have no idea where the "null table" exception might come from.
I am using this on a plate "Overlap 2" under a screen "Test Screen _ Script".
There are wells L9-24 and M9-24, each containing two images.
Attached is my .csv that I used for testing.
TEST table_02.csv

Am I missing something?
Is there a better detailed guide on how to get the metadata into the well-tables (I haven't even started with images yet...) maybe cirvumventing all these subtle pitfalls?

[will-moore]

@JensWendt I don't know what's wrong with that CSV. It looks OK. I just did a little test with the first half of your csv (editing the Well names to fit a smaller plate I have for testing) and it worked OK (using omero-metadata 0.9.0, which looks like what you have - at least populate.py", line 1093 is the same. It shouldn't care about the Plate or the number of Images in a Well. Even when I used Well M24 which doesn't exist on my plate, it didn't fail.

Sorry I can't be more help, but I don't think there's anything wrong with the CSV.
If you have the ability to run on the command line, you can get more verbose logging -vvv which might help:

$ omero metadata populate Plate:51 --file /path/to/TEST.table_02.csv -vvv

[JensWendt]

Heyo @will-moore ,

Again, thank you for the incredible quick answer.
I found a way to run it from the CLI as you proposed and it gave the following verbose Debug-Log:
Script Error.txt

Without knowing the actual structure and code behind it all, my guess from the log would be:
the sript it is able to correctly read the .csv and generate the table with all its rows, but then in the final step when it tries to convert the temporary table it created to an OMERO.table it crashes, for unknown reasons.
For clarification I run the omero-metadata 0.9.0 and OMERO 5.6.1
If you could help out with your background knowledge it would be highly appreciated!

Kind regards

Jens

[will-moore]

I wonder if pytables is installed (and working) on the server side? Do you know if populate metadata is working with any data now? Are there existing tables on the server that display OK (as in the screenshot above, when launched from the eye icon in Attachments) or data shown in Tables tab?

[JensWendt]

@will-moore you were spot on.
Pytables was installed for the omero-web venv but not the omero venv.
I pip-installed the same version into the omero venv as for the omero-web and additionally also the pyparser package, as it sounded like it might be useful as well.
Tried it again via the CLI and it worked.
I will play around with it a bit more now.
Thanks a lot!