Cannot install ANARCI

[rsneumann]

Hi, when running 'python setup.py install' in order to install ANARCI, I get a lot of errors and cannot install. I have tried to understand where things go wrong, and at one point the program is trying to download files from IMGT:
urls = { "HV": "https://www.imgt.org/genedb/GENElect?query=7.3+IGHV&species=%s" (...)
When pasting this web address into my browser, it got an error message. Maybe IMGT has reorganised their website?
Thanks for any help, the full stack trace of the install error is as follows:

root@62b3208ae455:/anarci/ANARCI# python setup.py install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
INFO: ANARCI lives in: /usr/local/lib/python3.11/dist-packages/anarci
INFO: Downloading germlines from IMGT and building HMMs...
INFO: running 'RUN_pipeline.sh', this will take a couple a minutes.
Parsed and saved Homo+sapiens HV
Parsed and saved Mus HV
Parsed and saved Rattus+norvegicus HV
Parsed and saved Oryctolagus+cuniculus HV
Parsed and saved Macaca+mulatta HV
Parsed and saved Sus+scrofa HV
Parsed and saved Vicugna+pacos HV
Parsed and saved Bos+taurus HV
Parsed and saved Homo+sapiens HJ
Parsed and saved Mus HJ
Parsed and saved Rattus+norvegicus HJ
Parsed and saved Oryctolagus+cuniculus HJ
Parsed and saved Macaca+mulatta HJ
Parsed and saved Sus+scrofa HJ
Parsed and saved Vicugna+pacos HJ
Parsed and saved Bos+taurus HJ
Parsed and saved Homo+sapiens KV
Parsed and saved Mus KV
Parsed and saved Rattus+norvegicus KV
Parsed and saved Oryctolagus+cuniculus KV
Parsed and saved Macaca+mulatta KV
Parsed and saved Sus+scrofa KV
Parsed and saved Bos+taurus KV
Parsed and saved Homo+sapiens KJ
Parsed and saved Mus KJ
Parsed and saved Rattus+norvegicus KJ
Parsed and saved Oryctolagus+cuniculus KJ
Parsed and saved Macaca+mulatta KJ
Parsed and saved Sus+scrofa KJ
Parsed and saved Bos+taurus KJ
Parsed and saved Homo+sapiens LV
Parsed and saved Mus LV
Parsed and saved Rattus+norvegicus LV
Parsed and saved Oryctolagus+cuniculus LV
Parsed and saved Macaca+mulatta LV
Parsed and saved Sus+scrofa LV
Parsed and saved Bos+taurus LV
Parsed and saved Homo+sapiens LJ
Parsed and saved Mus LJ
Parsed and saved Rattus+norvegicus LJ
Parsed and saved Oryctolagus+cuniculus LJ
Parsed and saved Macaca+mulatta LJ
Parsed and saved Sus+scrofa LJ
Parsed and saved Bos+taurus LJ
Parsed and saved Homo+sapiens AV
Parsed and saved Mus AV
Parsed and saved Homo+sapiens AJ
Parsed and saved Mus AJ
Parsed and saved Homo+sapiens BV
Parsed and saved Mus BV
Parsed and saved Homo+sapiens BJ
Parsed and saved Mus BJ

Formatting alignments

IGs
Homo_sapiens H
Homo_sapiens K
Homo_sapiens L
Mus H
Mus K
Mus L
Rattus_norvegicus H
Rattus_norvegicus K
Rattus_norvegicus L
Oryctolagus_cuniculus H
Oryctolagus_cuniculus K
Oryctolagus_cuniculus L
Macaca_mulatta H
Macaca_mulatta K
Macaca_mulatta L
Sus_scrofa H
Sus_scrofa K
Sus_scrofa L
Vicugna_pacos H
Bos_taurus H
Bos_taurus K
Bos_taurus L

TRs
Homo_sapiens A
Homo_sapiens B
Mus A
Mus B

Traceback (most recent call last):
File "/anarci/ANARCI/build_pipeline/./RipIMGT.py", line 183, in
main()
File "/anarci/ANARCI/build_pipeline/./RipIMGT.py", line 178, in main
if ripfasta(species, gene_type, force = False):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anarci/ANARCI/build_pipeline/./RipIMGT.py", line 157, in ripfasta
htmlfile = get_html(species, gene_type, force)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anarci/ANARCI/build_pipeline/./RipIMGT.py", line 138, in get_html
if urllib.request.urlretrieve( urls[gene_type]%species, filename ):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 241, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 557, in error
result = self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(args)
^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 749, in http_error_302
return self.parent.open(new, timeout=req.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 519, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(args)
^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 1352, in do_open
r = h.getresponse()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
response.begin()
File "/usr/lib/python3.11/http/client.py", line 318, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
Warning - this alignment doesn't feature CYS at position 23 and/or position 104.
EIVLTQS.PAVLSMAPKERVTITCQASQNI......NKWLAWYHQEPGRAPKLLIYEA.......SKLITGVP.SRFSGSG..SGTDFTLTISSLEPEDAGTYYCLQS,('Mus', 'K'),('Mustela putorius furo_JIRA1106', 'IGKV1-6301')
Warning - this alignment doesn't feature CYS at position 23 and/or position 104.
EVVMTQT.PLSLSITPGESASISCRASQSLVHS.NGNTYLSWYLQKPGQSPQLLIYKV.......SNRFTGVP.DRFTGSG..SGTDFTLRISRVEADDVGVYYCQQG,('Mus', 'K'),('Mustela putorius furo_JIRA1106', 'IGKV2-1901')
(...lots of similar errors...)

Traceback (most recent call last):
File "/anarci/ANARCI/build_pipeline/./FormatAlignments.py", line 480, in
main()
File "/anarci/ANARCI/build_pipeline/./FormatAlignments.py", line 454, in main
jalignments = format_j_genes(jalignments)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anarci/ANARCI/build_pipeline/./FormatAlignments.py", line 183, in format_j_genes
aligned = read_fasta( al_filename )
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/anarci/ANARCI/build_pipeline/./FormatAlignments.py", line 127, in read_fasta
handle = open(filename, "r")
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/anarci/ANARCI/build_pipeline/./muscle_alignments/all_js_aligned.fasta'
Alignment input open failed.
couldn't open ./curated_alignments/ALL.stockholm for reading

Error: File existence/permissions problem in trying to open HMM file ./HMMs/ALL.hmm.
HMM file ./HMMs/ALL.hmm not found (nor an .h3m binary of it)

error: [Errno 2] No such file or directory: 'curated_alignments/germlines.py'

[broncio123]

Dear @rsneumann
I have just cloned a fresh copy of ANARCI, created a conda environment (python=3.8), installed all dependencies listed in the README with conda, and ran python setup.py install within the ANARCI directory.
Everything works well from my end.

The error you report looks consistent with a missing dependency (e.g. hmmer). Please, try to install ANARCI again but using a conda environment making sure all dependencies are installed as in the README file, which we recommend.

If the problem persists, we'll look into it.

[WhalleyT]

Hi @broncio123, I have also had issues and I have installed hmmer et al. through mamba. I'll drop the exact command as well as my specs to see if that helps once I'm at my machine

[broncio123]

Hi @rsneumann and @WhalleyT , any updates in regards to the installation issue?
Please, let us know. If everything is OK, then just thumbs up react to my message and I'll close the issue.
Thanks!

[presleydc]

Interestingly, I saw this exact same error trying to create a spack package for anarci. I loaded biopython and hmmer as dependencies but hit this until I added muscle as a dependency.

[sunny-rain-63]

Have you solved this problem? I also meet the same problem.

[rsneumann]

Hi, I still cannot install ANARCI. However, I have found an ANARCI docker container, so I am using this one instead of installing my own copy of ANARCI. The docker container I am using is:
quay.io/biocontainers/anarci:2021.02.04--pyhdfd78af_0
Maybe this helps someone out!

[WhalleyT]

I have it working now on my WSL install (I should add it was not problematic on my Ubuntu machine/other clusters).

My conda install of muscle was silently segfaulting, I'm not sure if that was something specific to WSL or a wider issue, but I compiled muscle from source and it seems to be behaving itself.

Another thing to add is that others might get an error from calling muscle v5 because the flag for input has changed from --in <fasta> to --align <fasta>

[SanFran-Me]

FileNotFoundError: [Errno 2] No such file or directory: './muscle_alignments/all_js_aligned.fasta'
Alignment input open failed.
couldn't open ./curated_alignments/ALL.stockholm for reading

Error: File existence/permissions problem in trying to open HMM file ./HMMs/ALL.hmm.
HMM file ./HMMs/ALL.hmm not found (nor an .h3m binary of it)

[anan64]

I have the same error despite installing all of the conda dependencies and muscle and running python setup.py install in a conda env

[michaelpeer1]

Ran into the same issue - cannot install ANARCI in WSL. I followed the readme file steps: created a new virtual environment, installed biopython, hmmer=3.3.2, and then ran python setup.py install. Installation fails during format_j_genes:

Traceback (most recent call last):
File "/home/michaelp/ANARCI/build_pipeline/./FormatAlignments.py", line 480, in
main()
File "/home/michaelp/ANARCI/build_pipeline/./FormatAlignments.py", line 454, in main
jalignments = format_j_genes(jalignments)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/michaelp/ANARCI/build_pipeline/./FormatAlignments.py", line 183, in format_j_genes
aligned = read_fasta( al_filename )
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/michaelp/ANARCI/build_pipeline/./FormatAlignments.py", line 127, in read_fasta
handle = open(filename, "r")
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/michaelp/ANARCI/build_pipeline/./muscle_alignments/all_js_aligned.fasta'

Alignment input open failed.
couldn't open ./curated_alignments/ALL.stockholm for reading
Error: File existence/permissions problem in trying to open HMM file ./HMMs/ALL.hmm.
HMM file ./HMMs/ALL.hmm not found (nor an .h3m binary of it)
error: [Errno 2] No such file or directory: 'curated_alignments/germlines.py'

[WhalleyT]

@michaelpeer1 what does it look like if you run MUSCLE on it's own? My apt-get and conda installs of MUSCLE on WSL were buggy. I compiled it from source and it works

[DantonIvanochko]

If anybody is still having this issue, I was able to get ANARCI working on WSL (conda 24.5.0, python 3.12.2) by using muscle version3 instead of v5:

# get anarci ready
cd ~
git clone https://github.com/oxpig/ANARCI.git
conda install -c conda-forge biopython -y
conda install -c bioconda hmmer # i just took the newest release

# for muscle3
mkdir muscle
cd muscle
wget https://drive5.com/muscle/downloads3.8.31/muscle3.8.31_src.tar.gz
tar xzvf muscle3.8.31_src.tar.gz
cd muscle3.8.31/src/
make
./muscle # test the muscle runs
cp muscle ~/ANARCI/bin/muscle # this should overwrite the existing muscle file (check that this actually replaced the file)

#finish install
cd ~/ANARCI
python setup.py install

[lauratwomey]

The solution above worked for me with a slight modification - I just conda installed muscle. The versions of biopython and hmmer in the github's README file did not work for me, but if you git clone it pulls the following versions, so I just installed those too with conda:
This is after creating and activating a new conda env with miniconda3/23.5.2

# Install ANARCI dependencies first
conda install biopython=1.83
conda install hmmer=3.4
conda install muscle=3.8.1551-6
# Now go to the folder where you want ANARCI installed
 cd ~/Tools/omics_tools/
git clone https://github.com/oxpig/ANARCI
# commit 79f6c575056dedef86cb8f405ebb039197923eec
cd ANARCI 
# Replace muscle version
# whereis muscle # this will tell you where your bin/muscle is (replace it below with your own directories)
cp ~/condaenvs/kasearch_new/bin/muscle ~/Tools/omics_tools/ANARCI/bin/muscle
cd bin
# Need to make sure the new muscle version has execute permissions
chmod +x ANARCI muscle muscle_macOS
cd ../
python setup.py install
# Wohoo! it worked:)

[joerloeffler]

@DantonIvanochko this worked for me as well! thank you!

[wjs20]

The issue is likely caused by muscle silently failing to align the J segments. It seems like muscle 3.8 is required, as the API has completely changed in muscle 5.x. Here is the output of my debugging.

| python build_pipeline/FormatAlignments.py

Formatting alignments

IGs

TRs
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(174)format_j_genes()
-> reference = ("WFAYWGQGTLVTVSA", 4  , 19 )
(Pdb) l
169         return new_calignments
170
171     def format_j_genes(jalignments):
172
173         breakpoint()
174  ->     reference = ("WFAYWGQGTLVTVSA", 4  , 19 )
175         #                 seq           start  end
176
177         ffile = write_fasta(jalignments)
178         al_filename = os.path.join( file_path, "muscle_alignments", "all_js_aligned.fasta" )
179
(Pdb) n
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(177)format_j_genes()
-> ffile = write_fasta(jalignments)
(Pdb)
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(178)format_j_genes()
-> al_filename = os.path.join( file_path, "muscle_alignments", "all_js_aligned.fasta" )
(Pdb)
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(180)format_j_genes()
-> if sys.platform == "darwin":
(Pdb)
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(183)format_j_genes()
-> pr = Popen([f"muscle", "-in", ffile, "-gapopen", "-10", "-out", al_filename], stdout=PIPE, stderr=PIPE)
(Pdb)
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(184)format_j_genes()
-> o, e = pr.communicate()
(Pdb)
> /home/ws/git/ANARCI/build_pipeline/FormatAlignments.py(185)format_j_genes()
-> aligned = read_fasta( al_filename )
(Pdb) e
b'\n\nInvalid command line\nUnknown option in\n\n'
(Pdb) o
b''
(Pdb) import shutil, subprocess
(Pdb) subprocess.run(['muscle', '--version'], capture_output=True).stdout
b'muscle 5.1.linux64 [12f0e2]\nBuilt Jan 13 2022 23:17:13\n\n'
(Pdb)

The function failed to write the all_js_aligned.fasta file because it found my system muscle, which is the most up to date one (5.1) which has different parameter names. However this error is not propogated up the stack, so it only breaks when it doesn't find the all_js_aligned.fasta

However if I specify the bundled pre-compiled muscle binary I get a segfault which is also not reported.

(Pdb) result = subprocess.run([f"./bin/muscle", "-in", ffile, "-gapopen", "-10", "-out", al_filename], capture_output=
True)
(Pdb) result.stdout
b''
(Pdb) result.stderr
b''
(Pdb) ffile
'/home/ws/git/ANARCI/build_pipeline/muscle_alignments/all_js.fasta'
(Pdb)

the segfault is not apparent from the debugger, but if I run it from the command line I can see it.

ANARCI on  fix-install (f9b4a2c) [!?] via 🐍 v3.11.9 via 🅒 anarci-env
| ./bin/muscle -in build_pipeline/muscle_alignments/all_js.fasta -gapopen -10 -out -
[1]    71313 segmentation fault  ./bin/muscle -in build_pipeline/muscle_alignments/all_js.fasta -gapopen -10
ANARCI on  fix-install (f9b4a2c) [!?] via 🐍 v3.11.9 via 🅒 anarci-env
| ./bin/muscle -in build_pipeline/muscle_alignments/all_js.fasta -gapopen -10 -out -

Not sure why the bundled muscle binary is segfaulting? Maybe it was compiled for a different architecture? I'm on a x86_64 GNU/Linux.