Adding new generation of table manipulation tools

[hechth]

This PR introduces a set of new tabular file handling tools which are a bit more modern than what is available in the core galaxy tools.

The tools are based on pandas and/or scipy and all have a python CLI using argparse and should support parquet, csv, tsv and tabular files.

• arithmetic operations
• interpolate
• rename columns
• transform on columns - log, ld, ln, sqrt, e, abs, floor, ceil - basically anything that is of the form f(x) = y
• compute masks
• scale column subset
• summary

@bgruening and @bernt-matthias and @mvdbeek please give some feedback if you think this design works well or whether it should be done differently. I'm planning to add few more tools to this to cover many use-cases in various -omics disciplines.

Also @KristinaGomoryova please try the tools locally to see if this is roughly what you had in mind.

Also, please let me know whether you would see these tools rather in tools-iuc and if you have a better idea for naming this collection of tools, I'm open to suggestions.

[bgruening]

Do you know this tool? https://github.com/galaxyproject/tools-iuc/tree/main/tools/table_compute

[hechth]

Yes I do ... but the tool is too complex in the form for what it can do and not very flexible. You can't choose specific columns etc. and overall I think it doesn't really cover the use cases. We actually adopted this schema from KNIME which provides very nice tools and operations to perform over tables.

[bernt-matthias]

I share the concerns of @bgruening and think that (in particular in case of the arithmetics and rename tool) it might be better to extend/improve existing tools.

[bernt-matthias]

For this tool I would think that an extension for table compute is more appropriate. The possibility to select specific columns / rows for arithmetic operations would definitely make sense.

[hechth]

Having the multi select for the rows works, but it would probably have to provide both options

[bernt-matthias]

It could be an option to create 1-tool workflows for the tasks where we all the complexity of the the underlying tool gets hidden.

[hechth]

@bernt-matthias yes, that is possible - though workflows don't support the column select input type properly, so you can't dynamically select the columns, which massively reduces user friendliness.

[bernt-matthias]

@bernt-matthias yes, that is possible - though workflows don't support the column select input type properly, so you can't dynamically select the columns, which massively reduces user friendliness.

Got it.

We actually adopted this schema from KNIME which provides very nice tools and operations to perform over tables.

This is a point I totally support :)

[hechth]

@bernt-matthias and @bgruening I would like to move this forward but I'm afraid there won't be a community consensus any time soon. I would recommend that we will pilot the tools locally, moving them to the toolshed but keeping them out of usegalaxy.* until we have some more experience with what is really needed and if the usability and everything is how we expected it.

[bernt-matthias]

I asked @rmassei to have a look here. He has a KNIME history and could give us some feedback in this direction. I will also try to have another look on the tools.

[hechth]

@bernt-matthias thanks a lot, that would be very helpful!

[rmassei]

Hi @bernt-matthias and @hechth!
After reviewing the PR, I have a few comments:

• The behavior of table_pandas_arithmetics.xml and tools/tables/table_pandas_transform.xml can already be replicated using the existing table compute tool. While it would require additional steps (e.g., performing the operation on a single column via cut and replace), the output would be identical. Since this operation is likely to be frequently performed by users, it might make more sense to enhance the existing table compute tool to include this functionality, especially since it is also based on pandas.

• A similar comment applies to tools/tables/table_pandas_rename_column.xml. This functionality can already be achieved using a simple workflow or by extending the table compute tool (or even the add_column_headers tool) to include this feature.

By design, KNIME offers a broader selection of data wrangling tools, which might indeed feel more user-friendly. However, if a tool with similar functionality already exists in Galaxy (such as the table compute tool) and is likely already integrated into some workflows, wouldn't it be more practical to enhance the functionality of the existing tool and provide more user-friendly documentation in its help section?