PEPATAC with looper failed with the latest looper and eido

[xuebingjie1990]

PEPATAC with looper failed with the latest looper and eido

here is the error message:

Command: run
/home/bx2ur/.local/lib/python3.8/site-packages/divvy/compute.py:150: UserWarning: The '_file_path' property is deprecated and will be removed in a future release. Use ComputingConfiguration["__internal"]["_file_path"] instead.
  os.path.dirname(self._file_path),
/home/bx2ur/.local/lib/python3.8/site-packages/divvy/compute.py:58: UserWarning: The '_file_path' property is deprecated and will be removed in a future release. Use ComputingConfiguration["__internal"]["_file_path"] instead.
  self.config_file = self._file_path
Pipestat compatible: False
## [1 of 1] sample: test1; pipeline: PEPATAC
Calling pre-submit function: refgenconf.looper_refgenie_populate
ProjectContext pipeline_interfaces: '['/sfs/qumulo/qhome/bx2ur/code/pepatac/sample_pipeline_interface.yaml']'
Traceback (most recent call last):
  File "/home/bx2ur/.local/bin/looper", line 8, in <module>
    sys.exit(main())
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/looper.py", line 1115, in main
    run(args, rerun=(args.command == "rerun"), **compute_kwargs)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/looper.py", line 501, in __call__
    curr_pl_fails = cndtr.add_sample(sample, rerun=rerun)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/conductor.py", line 474, in add_sample
    self.submit()
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/conductor.py", line 514, in submit
    script = self.write_script(self._pool, self._curr_size)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/conductor.py", line 734, in write_script
    namespaces = _exec_pre_submit(pl_iface, namespaces)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/looper/conductor.py", line 848, in _exec_pre_submit
    _update_namespaces(namespaces, func(namespaces))
  File "/home/bx2ur/.local/lib/python3.8/site-packages/refgenconf/populator.py", line 45, in looper_refgenie_populate
    rgc = refgenconf.RefGenConf(rgc_path)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/refgenconf/refgenconf.py", line 96, in __init__
    super(RefGenConf, self).__init__(
  File "/home/bx2ur/.local/lib/python3.8/site-packages/yacman/yacman.py", line 125, in __init__
    file_contents = load_yaml(filepath)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/yacman/yacman.py", line 438, in load_yaml
    return read_yaml_file(filepath)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/yacman/yacman.py", line 418, in read_yaml_file
    with open(filepath, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '$REFGENIE'
Exception ignored in: <function YacAttMap.__del__ at 0x2ab7e64a4430>
Traceback (most recent call last):
  File "/home/bx2ur/.local/lib/python3.8/site-packages/yacman/yacman.py", line 159, in __del__
    if hasattr(self[IK], FILEPATH_KEY) and not getattr(self[IK], RO_KEY, True):
  File "/home/bx2ur/.local/lib/python3.8/site-packages/attmap/pathex_attmap.py", line 59, in __getitem__
    v = super(PathExAttMap, self).__getitem__(item)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/attmap/ordattmap.py", line 48, in __getitem__
    return AttMap.__getitem__(self, item)
  File "/home/bx2ur/.local/lib/python3.8/site-packages/attmap/attmap.py", line 32, in __getitem__
    return self.__dict__[item]
KeyError: '__internal'```

[nsheff]

FileNotFoundError: [Errno 2] No such file or directory: '$REFGENIE'

It looks like you don't have your $REFGENIE variable set, but you're passing that in...

[ayobi]

If you put in the path instead of the variable I get

Using default config. No config found in env var: ['DIVCFG']
Pipestat compatible: False
##[1 of 1] sample: test1; pipeline: PEPATAC
Calling pre-submit function: refgenconf.looper_refgenie_populate
Traceback (most recent call last):
  File "/home/aaronobrien/projects/pepatac/venv/bin/looper", line 8, in <module>
    sys.exit(main())
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/looper.py", line 1115, in main
    run(args, rerun=(args.command == "rerun"), **compute_kwargs)
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/looper.py", line 501, in __call__
    curr_pl_fails = cndtr.add_sample(sample, rerun=rerun)
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/conductor.py", line 474, in add_sample
    self.submit()
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/conductor.py", line 514, in submit
    script = self.write_script(self._pool, self._curr_size)
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/conductor.py", line 734, in write_script
    namespaces = _exec_pre_submit(pl_iface, namespaces)
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/looper/conductor.py", line 848, in _exec_pre_submit
    _update_namespaces(namespaces, func(namespaces))
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/refgenconf/populator.py", line 47, in looper_refgenie_populate
    complete_sk_dict = rgc.list_seek_keys_values()
  File "/home/aaronobrien/projects/pepatac/venv/lib/python3.10/site-packages/refgenconf/refgenconf.py", line 1130, in list_seek_keys_values
    for seek_key_name in get_tag_seek_keys(tag_mapping):
TypeError: 'NoneType' object is not iterable```

[donaldcampbelljr]

Replacing $REFGENIE in the sample_pipeline_interface.yaml with the absolute path to the refgenie config allows me to continue and succeed with the tutorial. It appears that when looper calls render_var_templates it is not replacing with the path. I tried different syntax to see if I could get it to work, but no luck yet.

pipeline_name: PEPATAC
pipeline_type: sample
path: pipelines/pepatac.py
input_schema: pepatac_input_schema.yaml
output_schema: pepatac_output_schema.yaml
command_template: >
  {pipeline.path}
  --output-parent { looper.results_subdir }
  --cores { compute.cores }
  --mem { compute.mem }
  --sample-name { sample.sample_name }
  --input { sample.read1 }
  {% if sample.read2 is defined %} --input2 { sample.read2 } {% endif %}
  --single-or-paired { sample.read_type }
  --genome { sample.genome }
  {% if sample.chrom_sizes is defined %} --chrom-sizes { sample.chrom_sizes } {% elif refgenie[sample.genome].fasta is defined %} --chrom-sizes { refgenie[sample.genome].fasta.chrom_sizes } {% endif %}
  {% if sample.TSS_name is defined %} --TSS-name { sample.TSS_name } {% elif refgenie[sample.genome].refgene_anno is defined %} --TSS-name { refgenie[sample.genome].refgene_anno.refgene_tss } {% endif %}
  {% if sample.blacklist is defined %} --blacklist { sample.blacklist } {% elif refgenie[sample.genome].blacklist is defined %} --blacklist { refgenie[sample.genome].blacklist.blacklist } {% endif %}
  {% if sample.anno_name is defined %} --anno-name { sample.anno_name } {% elif refgenie[sample.genome].feat_annotation is defined %} --anno-name { refgenie[sample.genome].feat_annotation.feat_annotation } {% endif %}
  {% if sample.trimmer is defined %} --trimmer { sample.trimmer } {% else %} --trimmer "skewer" {% endif %}
  {% if sample.aligner is defined %} --aligner { sample.aligner } {% set aligner = sample.aligner %} {% else %} --aligner "bowtie2" {% set aligner = "bowtie2" %} {% endif %}
  {% if aligner == "bowtie2" or sample.aligner == "bowtie2" %} {% if sample.genome_index is defined %} --genome-index { sample.genome_index } {% elif refgenie[sample.genome].bowtie2_index is defined %} --genome-index { refgenie[sample.genome].bowtie2_index.dir } {% endif %} {% else %} {% if sample.genome_index is defined %} --genome-index { sample.genome_index } {% elif refgenie[sample.genome].bwa_index is defined %} --genome-index { refgenie[sample.genome].bwa_index.dir } {% endif %} {% endif %}
  {% if sample.prealignment_index is defined %} --prealignment-index { sample.prealignment_index } {% endif %}
  {% if sample.prealignment_names is defined %} {% if aligner == "bowtie2" or sample.aligner == "bowtie2" %} --prealignment-index {% for p in sample.prealignment_names %} { p ~ '=' ~ refgenie[p].bowtie2_index.dir } {% endfor %} {% else %} --prealignment-index {% for p in sample.prealignment_names %} { p ~ '=' ~ refgenie[p].bwa_index.dir } {% endfor %} {% endif %} {% endif %}
  {% if sample.deduplicator is defined %} --deduplicator { sample.deduplicator } {% endif %}
  {% if sample.peak_caller is defined %} --peak-caller { sample.peak_caller } {% else %} --peak-caller "macs2" {% endif %}
  {% if sample.peak_type is defined %} --peak-type { sample.peak_type } {% else %} --peak-type "fixed" {% endif %}
  {% if sample.extend is defined %} --extend { sample.extend } {% else %} --extend 250 {% endif %}
  {% if sample.genome_size is defined %} --genome-size { sample.genome_size } {% else %} --genome-size "2.7e9" {% endif %}
  {% if sample.frip_ref_peaks is defined %} --frip-ref-peaks { sample.frip_ref_peaks } {% endif %}
  {% if sample.motif is defined %} --motif {% endif %}
  {% if sample.sob is defined %} --sob {% endif %}
  {% if sample.sob is defined %} {% if refgenie[sample.genome].tallymer_index is defined %} --search-file { refgenie[sample.genome].tallymer_index.search_file } {% endif %} {% endif %}
  {% if sample.sob is defined %} {% if refgenie[sample.genome].fasta is defined %} --fasta { refgenie[sample.genome].fasta.fasta } {% endif %} {% endif %}
  {% if sample.fasta is defined %} --fasta { sample.fasta } {% elif refgenie[sample.genome].fasta is defined %} --fasta { refgenie[sample.genome].fasta.fasta } {% endif %}
  {% if sample.no_scale is defined %} --no-scale {% endif %}
  {% if sample.prioritize is defined %} --prioritize {% endif %}
  {% if sample.keep is defined %} --keep {% endif %}
  {% if sample.no_fifo is defined %} --noFIFO {% endif %}
  {% if sample.lite is defined %} --lite {% endif %}
  {% if sample.skipqc is defined %} --skipqc {% endif %}

compute:
  singularity_image: ${SIMAGES}pepatac
  conda_env: pepatac
  docker_image:  databio/pepatac
  bulker_crate:  databio/pepatac:1.0.10
  size_dependent_variables: resources-sample.tsv
bioconductor:
  readFunName: runCOCOA
  readFunPath: BiocProject/runCOCOA.R
var_templates:
  refgenie_config: /home/drc/GITHUB/pepatac_tutorial/tools/pepatac/examples/tutorial/tutorial_refgenie2.yaml
pre_submit:
  python_functions:
  - refgenconf.looper_refgenie_populate

[donaldcampbelljr]

Looper calls regenconf.looper_refgenie_populate in refgenconf's populator.py and passes namespaces.

The refgenie config path is set:
rgc_path = namespaces["pipeline"]["var_templates"]["refgenie_config"]

In Looper 1.3.1 namespaces["pipeline"]["var_templates"] is actually a PathExAttMap and has a dunder function __getitem that expands the path such that the environment variable is appropriately replaced with a real path.

In Looper 1.4.1, __getitem__ is called from yacman1.py. This function does not expand the path and thus, the environment variable is left as '$REFGENIE`

[yqlan]

how to fix this issue if I DONT use refgenie, instead just using the manually downloaded genome assets?