hotfix/snvphyl 1.2.3 #1035

apetkau · 2021-06-14T18:05:28Z

Description of changes

Updates SNVPhyl to version 1.2.3 (from 1.2.2) which includes all updated tools which can all be properly installed in Galaxy.
Updates the IRIDA/Galaxy Docker image to include this new version of SNVPhyl and also updates from Galaxy 20.05 to 20.09 (the latest Galaxy Docker image).
- Updates the integration test suite to use this newer Docker image with the latest SNVPhyl pipeline.

In order to test this out, you can install the newest IRIDA/Galaxy Docker by the command:

docker run -d -p 48888:80 -v /path/to/irida/data:/path/to/irida/data phacnml/galaxy-irida-20.09

Where /path/to/irida/data is the location of your IRIDA data directories and 48888 is the port that Galaxy should run on.

Once the IRIDA/Galaxy Docker container is running you can connect IRIDA to this Galaxy version and test out the pipelines.

Tested with the following versions of Galaxy:

Galaxy 21.01 (latest stable release)
Galaxy 20.09 (version in Docker)
Galaxy 18.05 (version used by our IRIDA)
- For this version of Galaxy it looks like you need to remove bcftools_view and re-install the newer version. This has been documented in our SNVPhyl install instructions.

Note: this is a hotfix and so it must be merged into master and development.

Related issue

Fix for issue #1018

Checklist

Things for the developer to confirm they've done before the PR should be accepted:

CHANGELOG.md (and UPGRADING.md if necessary) updated with information for new change.
Tests added (or description of how to test) for any new features.
User documentation updated for UI or technical changes.

This reverts commit faa0368.

innovate-invent · 2021-06-16T17:27:54Z

Is this PR ready to go other than the docker content? I'd like to patch my deployment.

apetkau · 2021-06-16T17:33:45Z

No, not quite. I have not tested it out with different versions of Galaxy yet. I tried with an older version of Galaxy and some of the tools didn't seem to install properly. In particular bcftools_view, which is the tool that was causing issues in the first place (#1018).

I think the issue here is that if you have an older version of bcftools_view in Galaxy and try to update it via the SNVPhyl tool suite in Galaxy, Galaxy won't update it. But I have not looked further.

This issue only impacts you if you are installing in a Galaxy that already had older versions of SNVPhyl though. If you are installing in a Galaxy with no existing IRIDA-related tools then everything works fine.

apetkau · 2021-06-16T17:57:35Z

If you wish to patch you can do so though. It's very unlikely I will update the actual workflow files/tools. I probably just have to add more documentation on how to handle installing some tools if you encounter the issue I described above.

innovate-invent · 2021-06-17T19:29:27Z

I see in the log

17 Jun 2021 18:53:40,950 DEBUG ca.corefacility.bioinformatics.irida.config.workflow.IridaWorkflowsConfig:190 - Adding default workflow irida.workflow.default.PHYLOGENOMICS=0e06d5c7-20ad-404b-8152-e9b9573ba723
...
17 Jun 2021 18:53:40,986 DEBUG ca.corefacility.bioinformatics.irida.service.workflow.IridaWorkflowsService:181 - Registering workflow: IridaWorkflow [workflowDescription=IridaWorkflowDescription [id=0e06d5c7-20ad-404b-8152-e9b9573ba723, name=SNVPhyl, version=1.2.3, analysisType=PHYLOGENOMICS], workflowStructure=IridaWorkflowStructure [workflowFile=/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ca/corefacility/bioinformatics/irida/model/workflow/analysis/type/workflows/SNVPhyl/1.2.3/irida_workflow_structure.ga]]
...
17 Jun 2021 18:53:40,988 DEBUG ca.corefacility.bioinformatics.irida.service.workflow.IridaWorkflowsService:181 - Registering workflow: IridaWorkflow [workflowDescription=IridaWorkflowDescription [id=f609c177-c268-4ad0-9d7f-9f9d5187fef7, name=SNVPhyl, version=1.2.2, analysisType=PHYLOGENOMICS], workflowStructure=IridaWorkflowStructure [workflowFile=/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ca/corefacility/bioinformatics/irida/model/workflow/analysis/type/workflows/SNVPhyl/1.2.2/irida_workflow_structure.ga]]

and it appears to still be running 1.2.2 and failing

17 Jun 2021 19:16:38,695 ERROR ca.corefacility.bioinformatics.irida.service.analysis.execution.AnalysisExecutionServiceAspect:65 - Error occured for submission: AnalysisSubmission [id=149, name=SNVPhyl_20210617, submitter=nolan.woods, workflowId=f609c177-c268-4ad0-9d7f-9f9d5187fef7, analysisState=SUBMITTING, analysisCleanedState=NOT_CLEANED] changing to state ERROR
ca.corefacility.bioinformatics.irida.exceptions.WorkflowException: GalaxyResponseException{status=400, responseBody={"err_msg": {"sec_filter|select_genotype": "parameter 'select_genotype': an invalid option ('__none__') was selected (valid options: ^hom,^miss,^het,het,miss,hom) Using default: 'None'."}, "err_code": 0, "err_data": {"9": {"sec_filter|select_genotype": "parameter 'select_genotype': an invalid option ('__none__') was selected (valid options: ^hom,^miss,^het,het,miss,hom) Using default: 'None'."}}}, errorMessage=null, errorCode=null, traceback=null}
	at ca.corefacility.bioinformatics.irida.pipeline.upload.galaxy.GalaxyWorkflowService.runWorkflow(GalaxyWorkflowService.java:123)
	at ca.corefacility.bioinformatics.irida.service.analysis.execution.galaxy.AnalysisExecutionServiceGalaxyAsync.executeAnalysis(AnalysisExecutionServiceGalaxyAsync.java:152)
	at ca.corefacility.bioinformatics.irida.service.analysis.execution.galaxy.AnalysisExecutionServiceGalaxyAsync$$FastClassBySpringCGLIB$$fc2d896.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
	at ca.corefacility.bioinformatics.irida.service.analysis.annotations.RunAsUserAspect.setSecurityContextFromAnalysisSubmission(RunAsUserAspect.java:100)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
	at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
	at org.springframework.security.concurrent.DelegatingSecurityContextCallable.call(DelegatingSecurityContextCallable.java:87)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: GalaxyResponseException{status=400, responseBody={"err_msg": {"sec_filter|select_genotype": "parameter 'select_genotype': an invalid option ('__none__') was selected (valid options: ^hom,^miss,^het,het,miss,hom) Using default: 'None'."}, "err_code": 0, "err_data": {"9": {"sec_filter|select_genotype": "parameter 'select_genotype': an invalid option ('__none__') was selected (valid options: ^hom,^miss,^het,het,miss,hom) Using default: 'None'."}}}, errorMessage=null, errorCode=null, traceback=null}
	at com.github.jmchilton.blend4j.galaxy.Client.buildResponseException(Client.java:18)
	at com.github.jmchilton.blend4j.BaseClient.checkResponse(BaseClient.java:138)
	at com.github.jmchilton.blend4j.BaseClient.create(BaseClient.java:47)
	at com.github.jmchilton.blend4j.BaseClient.create(BaseClient.java:41)
	at com.github.jmchilton.blend4j.galaxy.WorkflowsClientImpl.runWorkflowInvocationResponse(WorkflowsClientImpl.java:50)
	at com.github.jmchilton.blend4j.galaxy.WorkflowsClientImpl.invokeWorkflow(WorkflowsClientImpl.java:54)
	at ca.corefacility.bioinformatics.irida.pipeline.upload.galaxy.GalaxyWorkflowService.runWorkflow(GalaxyWorkflowService.java:121)
	... 29 more

Is there anything extra I have to do to get it to use 1.2.3?

apetkau · 2021-06-17T21:30:36Z

I do not know why it is still running 1.2.2 for you since it is working for me.

You should see in the log it registering all previous versions of a workflow. It needs them registered to set metadata for previously-ran workflows. But it should default to 1.2.3 (which is what irida.workflow.default.PHYLOGENOMICS=0e06d5c7-20ad-404b-8152-e9b9573ba723 is suggesting it's doing).

innovate-invent · 2021-06-17T21:39:26Z

I found it, the front end instance is still running an old container :/

apetkau · 2021-06-17T21:46:51Z

Ah, okay. Thanks for letting me know.

innovate-invent · 2021-06-18T17:44:40Z

Looks like a tool the workflow depends on is broken: phac-nml/snvphyl-galaxy#36

apetkau · 2021-07-07T17:45:42Z

phac-nml/snvphyl-galaxy#36 was fixed and is no longer an issue

tom114

Tested and working as expected. Thanks @apetkau . I'll be merging this along with #1045 for the 21.05.2 release.

apetkau and others added 14 commits June 9, 2021 19:15

Start of version 1.2.3

9f92318

Updated SNVPhyl workflow to 1.2.3

7a578ed

Updated changelog/version

e2074c7

Updated docs

66d4a86

Updated Docker to Galaxy 20.09

cf42a6b

Switched integration testing to Galaxy 20.09

668ee56

Update VM Dockerfile

cc78fb5

Added tools for plugins

14092a9

Merge steps in dockerfile

faa0368

Revert "Merge steps in dockerfile"

a1fe868

This reverts commit faa0368.

Fixed up tools so they properly install

18f7509

Updated galaxy-docker for tests

6b2d98b

Wrong tag for docker image

8c3c56f

Updated docker version to use for Galaxy

6dedb34

apetkau added 2 commits July 5, 2021 23:09

Updating documentation about installing correct version of bcftools_view

4e90214

Fixed broken link

640a2ca

apetkau changed the title ~~WIP:Fix/snvphyl 1.2.3~~ Fix/snvphyl 1.2.3 Jul 7, 2021

apetkau changed the title ~~Fix/snvphyl 1.2.3~~ hotfix/snvphyl 1.2.3 Jul 7, 2021

apetkau requested a review from tom114 July 7, 2021 17:45

apetkau assigned tom114 Jul 7, 2021

apetkau added the tools Pipelines and workflows in IRIDA label Jul 7, 2021

apetkau added the galaxy Galaxy for workflows and IRIDA's communication with Galaxy label Jul 7, 2021

tom114 approved these changes Jul 9, 2021

View reviewed changes

tom114 merged commit b221548 into phac-nml:master Jul 9, 2021

apetkau deleted the fix/snvphyl-1.2.3 branch July 9, 2021 16:37

tom114 mentioned this pull request Jul 9, 2021

SNVPhyl no-longer works in Galaxy #1018

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hotfix/snvphyl 1.2.3 #1035

hotfix/snvphyl 1.2.3 #1035

apetkau commented Jun 14, 2021 •

edited

Loading

innovate-invent commented Jun 16, 2021

apetkau commented Jun 16, 2021 •

edited

Loading

apetkau commented Jun 16, 2021

innovate-invent commented Jun 17, 2021 •

edited

Loading

apetkau commented Jun 17, 2021

innovate-invent commented Jun 17, 2021 •

edited

Loading

apetkau commented Jun 17, 2021

innovate-invent commented Jun 18, 2021

apetkau commented Jul 7, 2021

tom114 left a comment

hotfix/snvphyl 1.2.3 #1035

hotfix/snvphyl 1.2.3 #1035

Conversation

apetkau commented Jun 14, 2021 • edited Loading

Description of changes

Related issue

Checklist

innovate-invent commented Jun 16, 2021

apetkau commented Jun 16, 2021 • edited Loading

apetkau commented Jun 16, 2021

innovate-invent commented Jun 17, 2021 • edited Loading

apetkau commented Jun 17, 2021

innovate-invent commented Jun 17, 2021 • edited Loading

apetkau commented Jun 17, 2021

innovate-invent commented Jun 18, 2021

apetkau commented Jul 7, 2021

tom114 left a comment

Choose a reason for hiding this comment

apetkau commented Jun 14, 2021 •

edited

Loading

apetkau commented Jun 16, 2021 •

edited

Loading

innovate-invent commented Jun 17, 2021 •

edited

Loading

innovate-invent commented Jun 17, 2021 •

edited

Loading