Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set "-o pipefail" for filter_subsample_sequences and mafft_one_chr #173

Merged
merged 1 commit into from
Nov 25, 2020

Conversation

tomkinsc
Copy link
Member

Previously, if the commands along the way in the pipeline fail, the tasks succeed. This adds "-o pipefail" so errors lead to non-zero exits.
This change may be needed elsewhere; only adding to these two tasks for now.

Previously, if the commands along the way in the pipeline fail, the tasks succeed. This adds "-o pipefail" so errors lead to non-zero exits.
This change may be needed elsewhere; only adding to these two tasks for now.
@tomkinsc tomkinsc merged commit 52bdedf into master Nov 25, 2020
@tomkinsc tomkinsc deleted the ct-nextstrain-pipefail branch November 25, 2020 03:25
@dpark01
Copy link
Member

dpark01 commented Nov 25, 2020

Wait, what was silently failing in the pipes?

This only appears to affect the one line in each task that has cat, xargs, and the actual command. In one of these cases, there's a defensive grep in there to get rid of blank lines (that sometimes exist and sometimes don't, depending on the WDL execution engine we're running on), and I learned the hard way a while ago that grep exits 1 if it never matches the pattern (so set -e is good, but -o pipefail shouldn't be used if there's a grep in the pipe that might reasonably never match a pattern).

@tomkinsc tomkinsc restored the ct-nextstrain-pipefail branch November 25, 2020 14:06
@tomkinsc
Copy link
Member Author

Great point about the grep calls. The augur filter command was silently failing (succeeding with no output) when provided with an empty metadata file as a result of macOS Docker not staging files correctly. Maybe I'll change this to exit if the input is empty? Or we could enable pipefail only for the augur filter call?

@dpark01
Copy link
Member

dpark01 commented Nov 25, 2020

yeah either way, let's see. Maybe we can break out the grep as a separate line wrapped in a set +o pipefail; grep... ; set -o pipefail, so that the augur and mafft invocations get the proper pipefail precautions.... Separate PR though.

There's probably other reasons augur filter might fail, so checking for empty metadata won't address all of them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants