partis command not passed to shell. while: Expression Syntax error.

[Pezhvuk]

Hi Branden,

This issue is to address and solve the rather ambiguous issue with running partis, previously reported in #22. I found out running partis directly from the terminal was not working either, but that issue seemed to be resolved after our system administrator figured out some rather complex module load conflicts on our end. However, the exact same error report, by sumrep, persisted despite of resolving our internal issue with running partis. After tinkering quite a bit, I managed to extract some more error information, which, eventually I managed to figure out the source of.

while: Expression Syntax.
Error in data.table::fread(., stringsAsFactors = TRUE) : 
  File '_output/partis_output-cluster-annotations.csv' does not exist. Include one or more spaces to consider the input a system command.
Calls: getPartisAnnotations ... freduce -> withVisible -> <Anonymous> -> <Anonymous>
In addition: Warning message:
running command '/bin/tcsh /d/as2/u/mp002/sumrep_14-03-2019/inst/run_partis.sh -p /d/as7/s/partis/bin/partis -a partition -i /d/as2/u/mp002/sumrep_project/test/test_fasta/S-FV_+14d.fasta -o _output/partis_output.csv -n 16 -l ig -h _output/params -e v_gl_seq:v_qr_seqs:cdr3_seqs' had status 1 
Execution halted

The whole issue was largely concerned with the callPartis function in the PartisFunctions.R and the run_partis.sh bash script (both referenced at the bottom for the ease of navigation).
The key issue here, which, is the while: Expression Syntax, was not making sense for a while (pun intended :)), because I could not find it's exact source. Until, I modified the callPartis source code by changing command %>% system (lines 51-52 in PartisFunctions.R) to system(command, intern=TRUE) to print out the command passed to run_partis.sh, which would subsequently make clear what COMMAND from run_partis.sh was being passed to the shell. The clue is in '/bin/tcsh /d/as2/u/mp002/sumrep_14-03-2019/inst/run_partis.sh bit, which made me realise that the run_partis.sh is only written for bash shells, and would not work on my tcsh; vaguely complaing while: Expression Syntax. This would be a non-fatal error, however, if the line shell <- Sys.getenv("SHELL") in callPartis (line 29 of PartisFunctions.R) was fixed to bash by shell <- /bin/bash/ instead, which is what eventually fixed this issue (though the exact same error report still persisted, but this time, due to an issue with --locus parameter; posted separately in #24 ).

Beyond the specific issues reported here, given the fact that despite completely different problems resulting in the same error report, i.e. the .csv file does not exist, the more important thing do in the end is, going back to what we discussed in #20. I think, relaying back a modified version of the errors reported by the shell, partis or any other programme would solve a lot of the issues. Alternatively, raw errors saved in a .txt file and a line informing the user of a complaint from the corresponding programme and pointing them to the .txt file.

sumrep/R/PartisFunctions.R

Lines 19 to 72 in 275c698

	callPartis <- function(action,
	input_filename,
	output_filename,
	output_path,
	partis_path,
	num_procs,
	locus,
	germline_dir,
	extra_columns
	) {
	shell <- Sys.getenv("SHELL")
	script.file <- system.file("run_partis.sh", package="sumrep")
	command <- paste(shell, script.file,
	"-p", partis_path,
	"-a", action,
	"-i", input_filename,
	"-o", output_filename,
	"-n", num_procs,
	"-l", locus,
	"-h", file.path(output_path, "params"))
	if(!missing(germline_dir) && !is.null(germline_dir)) {
	command <- paste(command,
	"-g",
	germline_dir)
	}
	if(!missing(extra_columns) && !is.null(extra_columns)) {
	command <- paste(command,
	"-e",
	extra_columns)
	}
	command %>%
	system
	# Manually add query sequences to dataset from fasta file.
	# partis by default alters these, and it's not straightforward to
	# get both the raw and altered sequences (we need both).
	annotation_filename <- output_filename %>%
	ifelse(action == "partition",
	gsub(.,
	pattern='.csv',
	replace='-cluster-annotations.csv'
	),
	.
	)
	partis_dataset <- appendQuerySequencesToPartisAnnotationsFile(
	input_filename,
	annotation_filename
	)
	return(partis_dataset)
	}

sumrep/inst/run_partis.sh

Lines 1 to 41 in 275c698

	while getopts p:a:i:o:h:n:l:g:e: option
	do
	case "${option}" in
	p) PARTISPATH=${OPTARG};;
	a) ACTION=${OPTARG};;
	i) INFNAME=${OPTARG};;
	o) OUTFNAME=${OPTARG};;
	h) PARAMPATH=${OPTARG};;
	n) NPROCS=${OPTARG};;
	l) LOCUS=${OPTARG};;
	g) GERMLINE_DIR=${OPTARG};;
	e) EXTRA_COLUMNS=${OPTARG};;
	s) SEED=${OPTARG};;
	esac
	done
	COMMAND="$PARTISPATH $ACTION \
	--infname $INFNAME \
	--outfname $OUTFNAME \
	--n-procs $NPROCS \
	--locus $LOCUS \
	--parameter-dir $PARAMPATH "
	# Add the germline directory to the command only if passed
	if [ ! -z $GERMLINE_DIR ]
	then
	COMMAND="$COMMAND --initial-germline-dir $GERMLINE_DIR"
	fi
	if [ ! -z $EXTRA_COLUMNS ]
	then
	COMMAND="$COMMAND --extra-annotation-columns $EXTRA_COLUMNS"
	fi
	if [ ! -z $SEED ]
	then
	COMMAND="$COMMAND --seed $SEED"
	fi
	echo $COMMAND
	$COMMAND

Cheers,
Peji.

[BrandenOlson]

Ah, yeah, that must have been a nasty one to debug. My apologies!

The reason run_partis.sh exists is because I couldn't find a way to execute partis in R without providing the full PARTIS_PATH. However, at some point I must have needed this full path regardless as it's a required argument to sumrep::callPartis. Now it seems that run_partis.sh is obsolete and I should just be calling partis from R.

Thus, I think the best solution is to rework the partis call logic into sumrep::callPartis, and let system take care of any sort of shell particulars.

[BrandenOlson]

Fixed by 6e9def3.