Nextflow error

[yabeykoon]

Hi, I'm trying to run the Nextflow pipeline for trans analysis for my high MOI perturb-seq data. I've run steps on R console to create .odm files and the .rds object and now at the step of running the nextflow pipeline. I created a .sh script in the terminal of our university's computing cluster, but I'm getting the follwing error:

`Checking timothy-barry/sceptre-pipeline ...
WARN: Cannot read project manifest -- Cause: api.github.com
Cannot find `timothy-barry/sceptre-pipeline` -- Make sure exists a GitHub repos$
curl: (6) Could not resolve host: www.nextflow.io

 N E X T F L O W   ~  version 24.04.1

Pulling timothy-barry/sceptre-pipeline ...
WARN: Cannot read project manifest -- Cause: api.github.com
Cannot find `timothy-barry/sceptre-pipeline` -- Make sure exists a GitHub repos$
`

Here's a part of the code I'm running:

`#!/bin/bash           # the shell language when run outside of the job scheduler
#                     # lines starting with #$ is an instruction to the job scheduler
#$ -S /bin/bash       # the shell language when run via the job scheduler [IMPORTANT]
#$ -pe smp 10         # request 10 slots in the SMP parallel environment
nextflow pull timothy-barry/sceptre-pipeline
export NXF_OPTS="-Xms500M -Xmx4G"

##########################
# REQUIRED INPUT ARGUMENTS
##########################
data_directory="wynton/scratch/yabeykoon/sceptre_data/"
# sceptre object
sceptre_object_fp=$data_directory"sceptre_object.rds"
# response ODM
response_odm_fp=$data_directory"gene.odm"
# grna ODM
grna_odm_fp=$data_directory"grna.odm"

###################
# OUTPUT DIRECTORY:
##################
output_directory="wynton/scratch/yabeykoon/sceptre_outputs_netflow1"

#################
# Invoke pipeline
#################
nextflow run timothy-barry/sceptre-pipeline -r main \`

[yabeykoon]

Sorry I might have posted this question here prematurely. I'll look into how to fix the curl: (6) Could not resolve host: www.nextflow.io Nextflow issue itself first.

[timothy-barry]

Hi, is this resolved yet? In particular, are you able to run other NF pipelines? For example, are you able to run any pipelines in nf-core?

[yabeykoon]

Yes. I was able to run SCEPTRE trans on Nextflow on our computing cluster. I ran it on a development node instead of submitting as a job on the computing nodes since it required internet connection. If you have a tip on how to run it offline with qsub please let me know.

I would also like to know if you have some insight on what to expect with regards to callibration for trans discovery. I didn't run the massive-scale. Here's the trans analysis summary of my data.

`An object of class sceptre_object.

Attributes of the data:
	• 217754 cells (161461 after cellwise QC)
	• 36385 responses
	• High multiplicity-of-infection 
	• 74 targeting gRNAs (distributed across 27 targets) 
	• 28 non-targeting gRNAs 
	• 12 covariates (batch, grna_n_nonzero, grna_n_umis, PC_1, PC_2, PC_3, PC_4, PC_5, response_n_nonzero, response_n_umis, response_p_mito, X)

Analysis status:
	✓ import_data()
	✓ set_analysis_parameters()
	✓ assign_grnas()
	✓ run_qc()
	✓ run_calibration_check()
	✗ run_power_check()
	✓ run_discovery_analysis()

Analysis parameters: 
	• Discovery pairs: data frame with 982395 pairs (464270 after pairwise QC)
	• Positive control pairs: data frame with 0 pairs
	• Sidedness of test: both
	• Resampling mechanism: permutations
	• gRNA integration strategy: union
	• Resampling approximation: skew normal
	• Multiple testing adjustment: BH at level 0.1
	• N nonzero treatment cells threshold: 100
	• N nonzero control cells threshold: 7
	• Formula object: log(response_n_nonzero) + log(response_n_umis) + log(grna_n_nonzero + 1) + log(grna_n_umis + 1) + batch + PC_1 + PC_2 + PC_3 + PC_4 + PC_5

gRNA-to-cell assignment information:
	• Assignment method: mixture
	• Mean N cells per gRNA: 4962.67
	• Mean N gRNAs per cell (MOI): 2.32 
	• gRNA assignment formula object: log(response_n_umis) + log(response_n_nonzero) + log(grna_n_umis + 1) + log(grna_n_nonzero + 1) + batch

Summary of results:
	• N negative control pairs called as significant: 14668/464270
	• Mean log-2 FC for negative control pairs: -0.0031
	• N discovery pairs called as significant: 39555/464270`

In your experience, is it possible to get good calibration for a trans analysis of this scale? This was my first pass at trans analysis; I will make more adjustments as proposed in SCEPTRE documentation.

[timothy-barry]

That is good to hear. I have never tried to run Nextflow on a system without an internet connection, sorry. Perhaps this page might be useful?

The setup of your analysis looks good. I do see that there are ~15,000 false discoveries. The difficulty of obtaining good calibration is not about cis versus trans per se, but rather the number of pairs in the discovery set. I see that you have ~450,000 pairs in your discovery set. IMO this is not a crazy number; I have seen good calibration out to ~450,000 pairs before. Fortunately, your mean log-2 fold change on the negative control pairs looks quite good. My initial recommendation would be to decrease N nonzero treatment cells threshold; 100 is quite high. Also, if possible, it might be worthwhile to decrease the total number of pairs tested.

[timothy-barry]

Would you have a better way to control for different cell clusters?

I see. I did not realize that your experiment was carried out in a heterogeneous cell culture. The best way to control for cell type would be to include cell type as a covariate in the model. sceptre does not currently impute cell type, so you would need to determine cell type using some other program (e.g., Seurat). However, I see that you have included the top PCs as covariates, which should capture much of the variation due to cell type.

And what number would you suggest in this case for N nonzero treatment cells threshold (I see that the default is 7)?

Yes, I'd go with the default as a starting point.

Additionally, what is the code for decreasing total number of pairs tested?

Well, there's not code to decrease the total number of pairs per se. I think you are doing the right thing in prioritizing gRNAs that showed promising results in the cis analysis.

Also I've been playing around with the parameters for the Nextflow trans analysis and now I keep getting a qc error which I didn't get with the minimal parameters I used in my successful run.

Would you mind sending me your launch.sh script? No need to send the data yet.

[yabeykoon]

Hi sorry for the late response! Here's the script that's giving the QC Error. I will also attach what the error says below.

`#Throwing a QC error

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -j y
#$ -l mem_free=4G
#$ -l scratch=10G
#$ -l h_rt=336:00:00
#$ -r y
#$ -pe smp 10

# Deactivate any active Conda environment
conda deactivate

# Source the bash profile
source ~/.bash_profile

module load CBI r

module load openjdk

# Limit NF driver to 4 GB memory
export NXF_OPTS="-Xms500M -Xmx4G"

##########################
# REQUIRED INPUT ARGUMENTS
##########################
data_directory=$HOME"/sceptre_data_nextflow2/"
# sceptre object
sceptre_object_fp=$data_directory"sceptre_object.rds"
# response ODM
response_odm_fp=$data_directory"gene.odm"
# grna ODM
grna_odm_fp=$data_directory"grna.odm"

###################
# OUTPUT DIRECTORY:
##################
output_directory="wynton/scratch/yabeykoon/trans_sceptre_data_nextflow2"

#################
# Invoke pipeline
#################
nextflow run timothy-barry/sceptre-pipeline -r main \
 --config /wynton/group/pollen/nfcore/nfcore_wynton.config \
 --config /wynton/home/pollenlab/yabeykoon/.nextflow/custom.config \
 --sceptre_object_fp $sceptre_object_fp \
 --response_odm_fp $response_odm_fp \
 --grna_odm_fp $grna_odm_fp \
 --output_directory /wynton/scratch/yabeykoon/trans_sceptre_data_nextflow2 \
 --grna_assignment_method mixture \
 --grna_integration_strategy bonferroni \
 --resampling_approximation no_approximation \
 --response_n_umis_range_lower 0.1 \
 --response_n_umis_range_uppper 0.95 \
 --response_n_nonzero_range_lower 0.2 \
 --response_n_nonzero_range_upper 0.95 \
 --p_mito_threshold 0.075 \
 --pair_pod_size 1000 \
 --grna_pod_size 10 \
 --n_em_rep 20 \
 --n_nonzero_cells_cutoff 100 \
 --probability_threshold 0.99 \
 --n_nonzero_trt_thresh 100 \
 --n_nonzero_cntrl_thresh 100 \
 --run_association_analysis_memory 4GB \
 --assign_grnas_time_per_grna 15s \
 --combine_assign_grnas_time 60m \
 --run_qc_time 2h \
 --prepare_association_analysis_time 60m \
 --run_qc_memory 8GB \
 --run_association_analysis_time_per_pair 5m \
 --combine_association_analysis_time 5h `

[yabeykoon]

And here's the error:

`#QC Error
ERROR ~ Error executing process > 'run_qc'

Caused by:
  Process `run_qc` terminated with an error exit status (1)


Command executed:

  run_qc.R sceptre_object_fp   response_odm_fp   grna_odm_fp   100   default   0.1   default   0.2   0.95   0.075

Command exit status:
  1

Command output:
  (empty)

Command error:
  Error in `[.data.frame`(x, i, j) : undefined columns selected
  Calls: do.call ... compute_pairwise_qc_information -> [ -> [.data.table -> [.data.frame
Execution halted
Work dir:
  /wynton/scratch/yabeykoon/82/48fab0c8efd948c70a33b1fbaf8d83

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details`

[timothy-barry]

Thanks. Unfortunately, off the top it's hard for me to tell what is going wrong. Might you be free for a ~1h Zoom call to debug?

[yabeykoon]

Sounds great! Thank you so much. I'll email you at [email protected] to set up a call.