Installation issues - invalid escape sequence

[karenhn]

Hi, I'd like to use your package but I'm having issues with installation. I've followed all the steps until:

snakemake --use-conda --conda-create-envs-only --cores 1

But I get the following error messages:

/scratch/karenhn/KOunt/Snakefile:203: SyntaxWarning: invalid escape sequence '\/                                                     '
  filtered_gff=f'{outdir}Prodigal_Outputs/filtered_gffs/{{id}}/{{id}}.gff'
/scratch/karenhn/KOunt/Snakefile:219: SyntaxWarning: invalid escape sequence '\/                                                                  '
  rule bwa:
/scratch/karenhn/KOunt/Snakefile:680: SyntaxWarning: invalid escape sequence '\*                                                                  '
  touch {output}
/scratch/karenhn/KOunt/Snakefile:703: SyntaxWarning: invalid escape sequence '\*                                                                  '
  touch=f"{outdir}Touch/nohit_fa_{{id}}"
/scratch/karenhn/KOunt/Snakefile:748: SyntaxWarning: invalid escape sequence '\*                                                                  '
  '''
/scratch/karenhn/KOunt/Snakefile:767: SyntaxWarning: invalid escape sequence '\*                                                                  '
  of="6 qseqid sseqid pident qlen slen qstart qend sstart send evalue",
/scratch/karenhn/KOunt/Snakefile:807: SyntaxWarning: invalid escape sequence '\*                                                                  '
  raw=f"{outdir}diamond/NoHit_{{id}}_norna",
/scratch/karenhn/KOunt/Snakefile:825: SyntaxWarning: invalid escape sequence '\*                                                                  '
  shell:
FileNotFoundError in file /scratch/karenhn/KOunt/Snakefile, line 7:
[Errno 2] No such file or directory: 'test_fastqs/'
  File "/scratch/karenhn/KOunt/Snakefile", line 7, in <module>

I did try downloading the test_fastqs from the next step and rerunning the create envs code, but had the same issue. I'm fairly new to python etc, so could you tell me where I'm going wrong please?

[jmattock5]

Hi, thank you for your feedback. I think these warnings are occurring since a recent Snakemake update. I've updated the installation instructions in the README to specify the snakemake and mamba versions to install. Please could you try and make the snakemake_mamba env again and see if that solves the problem?
Thanks,
Jenny

[karenhn]

Thanks. I've tried again and it seemed to work better, but I now get a different error. Following
snakemake --conda-create-envs-only --cores 1 --use-conda
[my system said Error: --use-conda must be set if --conda-prefix or --create-envs-only is set]
I get these messages:

Traceback (most recent call last):
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/weakref.py", line 666, in _exitfunc
    f()
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/tempfile.py", line 933, in _cleanup
    cls._rmtree(name, ignore_errors=ignore_errors)
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/tempfile.py", line 929, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 752, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 1 more time]
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 683, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/karenhn/.conda/envs/snakemake_mamba/lib/python3.11/shutil.py", line 681, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'envs'

Any help would be appreciated!
Karen

[jmattock5]

Hi Karen,

Please could you try running the test installation steps and let me know what error messages you get for that?

Thanks,
Jenny

[karenhn]

Hi Jenny,

There's quite a lot of messages, so I'll just summarise! It starts with:

rule trim:
    input: test_fastqs/ERR2027889_005/ERR2027889_005_R1.fastq.gz, test_fastqs/ERR2027889_005/ERR2027889_005_R2.fastq.gz
    output: out/Touch/trim_ERR2027889_005
    jobid: 1
    reason: Missing output files: out/Touch/trim_ERR2027889_005
    wildcards: id=ERR2027889_005
    resources: tmpdir=/tmp

And has this for outputs: BamDeal, Kofamscan, assemblies, Prodigal etc

And then after a page of these green messages, it has a list of counts and threads for the job stats and:

Reasons:
    (check individual jobs above for details)
    input files updated by another job:
        abundance_matrix_annotate_with_mapping, abundance_results, all, bwa, cdhit, coverage, diamond_search, kallisto, kallisto_unmapped, kegg_fastas, kofamscan, kofamscan_results, megahit, mmseq_cluster_count, mmseq_keggs, mmseq_nohit, nohit_annotate_reads, nohit_fastas, nohit_fastq, prodigal, split_keggs, unmapped_reads
    missing output files:
        abundance_matrix_annotate_with_mapping, abundance_results, bamdeal, bwa, cdhit, coverage, diamond_search, kallisto, kallisto_unmapped, kegg_db, kegg_fastas, kofamscan, kofamscan_results, megahit, mmseq_cluster_count, mmseq_keggs, mmseq_nohit, nohit_annotate_reads, nohit_fastas, nohit_fastq, prodigal, split_keggs, trim, unmapped_reads

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

There was no out directory created and no output files.

Karen

[jmattock5]

Hi Karen,

This is the standard output we want to see after a dry run with Snakemake - it tells you which jobs will be run and what outputs will be created before you submit it. Please try running the same command but without the '-n' option to get it to actually run the pipeline.

Thanks,
Jenny

[karenhn]

2024-10-06T184627.149112.snakemake.log

Hi Jenny,

I've attached the log of all the error messages, but it finished with
Exiting because a job execution failed. Look above for error message
I did get an out directory with Kofamscan and trimmed, but nothing else.

Karen

[jmattock5]

Hi Karen,
Ok, I would expect it to report that it was making the env before running the pipeline so I think Snakemake thinks the envs exist. Could you try removing the hidden snakemake folder rm -r .snakemake in your current directory and try again?
Jenny

[karenhn]

Hi Jenny,

Failed again :(
Exiting because a job execution failed. Look above for error message
2024-10-09T120249.868195.snakemake.log
I also tried adjusting the latency to 20 seconds, but either that wasn't enough or it wasn't the issue.

Karen

[jmattock5]

Hi Karen,

Just to double check, what OS are you trying to run KOunt on? I've just been trying to recreate your issues on a MacBook and discovered that one of the dependancies can only be run with Linux.

If you are on Linux then I think we should try making yet another snakemake env using this command:
conda env create -n snakemake_mamba_old -f envs/snakemake_mamba.yaml

Then activate this env, remove the old attempts and see if we can make the KOunt envs:

conda activate snakemake_mamba_old
rm -r .snakemake
rm -r out
snakemake --conda-create-envs-only --cores 1 --use-conda

Please can you send me all of the outputted text when you run this?
Thanks,
Jenny

[karenhn]

Hi Jenny,

I am on Linux (5.4.0.-189) with Ubuntu 20.04.6 LTS.

I got errors setting up the new environment about not having the right conda distribution:
conda env create -n snakemake_mamba_old3 -f envs/snakemake_mamba.yaml

Installing pip dependencies: - Ran pip subprocess with arguments:
['/home/karenhn/.conda/envs/snakemake_mamba_old/bin/python', '-m', 'pip', 'install', '-U', '-r', '/scratch/karenhn/KOunt/envs/condaenv.ylft8l17.requirements.txt', '--exists-action=b']
Pip subprocess output:
Collecting aioeasywebdav==2.4.0
  Using cached aioeasywebdav-2.4.0-py3-none-any.whl (10 kB)
Collecting aiohttp==3.8.4
  Using cached aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
Collecting aiosignal==1.3.1
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting amply==0.1.5
  Using cached amply-0.1.5-py3-none-any.whl (16 kB)
Requirement already satisfied: appdirs==1.4.4 in /home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages (from -r /scratch/karenhn/KOunt/envs/condaenv.ylft8l17.requirements.txt (line 5)) (1.4.4)
Collecting async-timeout==4.0.2
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting attmap==0.13.2
  Using cached attmap-0.13.2-py3-none-any.whl (12 kB)
Collecting attrs==22.2.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Requirement already satisfied: backports-functools-lru-cache==1.6.4 in /home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages (from -r /scratch/karenhn/KOunt/envs/condaenv.ylft8l17.requirements.txt (line 9)) (1.6.4)
Collecting bcrypt==3.2.2
  Using cached bcrypt-3.2.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (62 kB)
Collecting boto3==1.26.77
  Using cached boto3-1.26.77-py3-none-any.whl (132 kB)
Collecting botocore==1.29.77
  Using cached botocore-1.29.77-py3-none-any.whl (10.4 MB)
Collecting brotlipy==0.7.0
  Using cached brotlipy-0.7.0-cp35-abi3-manylinux2010_x86_64.whl (1.1 MB)
Collecting cachetools==5.3.0
  Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting certifi==2022.12.7
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting cffi==1.15.1
  Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)

Pip subprocess error:
**ERROR: Could not find a version that satisfies the requirement conda==23.1.0 (from versions: 3.0.6, 3.5.0, 3.7.0, 3.17.0, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, 4.0.5, 4.0.7, 4.0.8, 4.0.9, 4.1.2, 4.1.6, 4.2.6, 4.2.7, 4.3.13, 4.3.16)
ERROR: No matching distribution found for conda==23.1.0**
failed
CondaEnvException: Pip failed

I have conda 23.11.0 within mambaforge. I tried changing the yaml file to reference either 23.11.0 or a version listed in the error message (4.3.16) but still got the same error.

I tried the next steps anyway - and got these messages:

Traceback (most recent call last):
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/snakemake/__init__.py", line 760, in snakemake
    success = workflow.execute(
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/snakemake/workflow.py", line 1095, in execute
    raise e
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/snakemake/workflow.py", line 1091, in execute
    success = self.scheduler.schedule()
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/snakemake/scheduler.py", line 571, in schedule
    run = self.job_selector(needrun)
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/snakemake/scheduler.py", line 714, in job_selector_ilp
    from stopit import ThreadingTimeout as Timeout, TimeoutException
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/stopit/__init__.py", line 10, in <module>
    import pkg_resources
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/pkg_resources/__init__.py", line 95, in <module>
    from jaraco.text import drop_comment, join_continuation, yield_lines
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/home/karenhn/.conda/envs/snakemake_mamba_old/lib/python3.10/site-packages/backports/__init__.py)

No out directory created.

[jmattock5]

Hi Karen,

Please could to try to see if either of these solutions work?

Try making a new env, specifying python version 3.10.6, you may need to use another name:

conda create -n snakemake_mamba -c conda-forge -c bioconda mamba=1.0.0 python=3.10.6
conda activate snakemake_mamba
mamba install -c bioconda snakemake=7.22.0

rm -r .snakemake
rm -r out
snakemake --conda-create-envs-only --cores 1 --use-conda

Or make an env using this edited yml file (I couldn't upload with the right extension so you'll have to rename it)
snakemake_mamba.txt

mv snakemake_mamba.txt snakemake_mamba.yml
conda env create -n snakemake_mamba_old3 -f snakemake_mamba.yml
conda activate snakemake_mamba_old3
rm -r .snakemake
rm -r out
snakemake --conda-create-envs-only --cores 1 --use-conda

Thanks,
Jenny