[BUG] Changes to tables made outside of ARCitect not properly loaded

[shiltemann]

OS and framework information (please complete the following information):

• OS: Ubuntu 22.04
• ARCitect: v0.0.40

Describe the bug
I sometimes need to edit my tables outside of ARCitect (e.g. for batch operations on large numbers of rows, this is not really convenient/doable yet in ARCitect). However, the changes I make outside of ARCitect are only partially detected, more specifically, any added rows are not loaded in ARCitect.

Steps to reproduce

• Create a new ARC
• Create an assay
• Add any table template
• Add some values to the row
• Save the ARC
• Open the explorer and edit the isa.assay.xlxs file with an external program (open office in my case, also tested it with excel web)
  • change the existing values in row 1
  • add a new row with some content
• Restart ARCitect and open the ARC again
• The changes to the existing row are shown, but the additional row is not displayed

[Freymaurer]

Can you share the example file? Then we will look into it 🙂

[shiltemann]

@Freymaurer sure, here is the file on OneDrive: https://1drv.ms/x/c/f72c4b5fae1f0d2e/ESJ36nx9u2dOo2AA__TxBo0BT7JlC3vz9mHYsovQU-0InQ?e=WN4D1C

some more details: I started with 180 lines, one per sample. I have paired-end data so this means 360 files, so I wanted a line per file, so (outside of ARCitect) I duplicated the 180 lines, added filenames, then sorted on first column so each pair of lines from the same sample is shown together. ARCitect now shows me only the first 180 lines of the new sorted file. See both the "SRA - submissions" table as the "RNA-Seq" table

[Mestizia]

I did some further testing as I felt it was related to nfdi4plants/ARCtrl#462

That issue refers to macos and windows, the issue in ubuntu is probably connected.

Here is a mock table I created in arcitect (ubuntu):

Opening it in open office gives no error, I added some data in open office as you can see:

After saving and closing office, I open again arcitect and open the table. However, the values in rows that were not created in arcitect still do not show up (6). Additionally, columns where there was no data filled in before are also missing the values (5). Notably, cells for which there were already values in both row and column in arcitect do work (4). Further testing shows that I can successfully edit cells that were already filled in arcitect with open office and the changes will show up in arcitect.

[Freymaurer]

@Freymaurer sure, here is the file on OneDrive: https://1drv.ms/f/c/f72c4b5fae1f0d2e/Em7BwphmM5lGg5WPBI_AEJcBDx5NLTMmBsjaQ83kuBUOPw?e=ldIhDS

some more details: I started with 180 lines, one per sample. I have paired-end data so this means 360 files, so I wanted a line per file, so (outside of ARCitect) I duplicated the 180 lines, added filenames, then sorted on first column so each pair of lines from the same sample is shown together. ARCitect now shows me only the first 180 lines of the new sorted file. See both the "SRA - submissions" table as the "RNA-Seq" table

I do not have permission to access the isa.assay file 😅

[shiltemann]

oops, updated permissions, better now? https://1drv.ms/x/c/f72c4b5fae1f0d2e/ESJ36nx9u2dOo2AA__TxBo0BT7JlC3vz9mHYsovQU-0InQ?e=WN4D1C

[Freymaurer]

Yes! And i already found the issue! Swate (and underlying ARCtrl) work based on xlsx tables.

Tip

The content of the annotation table MUST be placed in an xlsx table whose name starts with annotationTable. Each sheet MUST contain at most one such annotation table. Only cells inside this table are considered as part of the formatted metadata.

https://github.com/nfdi4plants/ARC-specification/blob/main/ISA-XLSX.md#annotation-table-sheets

In your example file the table only spans to row 181. Therefore, ARCtrl ignores everything else.

@Mestizia

After saving and closing office, I open again arcitect and open the table. However, the values in rows that were not created in arcitect still do not show up (6).

Can you verifiy if this is the same issue? With the table not spanning row 3?

Additionally, columns where there was no data filled in before are also missing the values (5).

This is the "Term Source REF" column, which is hidden by default in Swate. Can you expand the composite column by clicking on this icon?

[shiltemann]

@Freymaurer thanks for troubleshooting! ok... I learned something new about Excel files today ;P ..and more importantly I can continue with my ARC again, so thanks a bunch for the quick response!

[Mestizia]

Yes! And i already found the issue! Swate (and underlying ARCtrl) work based on xlsx tables.

Tip

The content of the annotation table MUST be placed in an xlsx table whose name starts with annotationTable. Each sheet MUST contain at most one such annotation table. Only cells inside this table are considered as part of the formatted metadata.

https://github.com/nfdi4plants/ARC-specification/blob/main/ISA-XLSX.md#annotation-table-sheets

In your example file the table only spans to row 181. Therefore, ARCtrl ignores everything else.

@Mestizia

After saving and closing office, I open again arcitect and open the table. However, the values in rows that were not created in arcitect still do not show up (6).

Can you verifiy if this is the same issue? With the table not spanning row 3?

Additionally, columns where there was no data filled in before are also missing the values (5).

This is the "Term Source REF" column, which is hidden by default in Swate. Can you expand the composite column by clicking on this icon?

Yes it is there, sorry for the delay!
That actually means that I can edit the table in libreoffice calc.

[Freymaurer]

Awesome! I'll close this issue for now, feel free to reopen!