faiss-cpu 1.10.0 update broke RAGatouille install

[Quinn-Meyer-Sustainment]

FYI new version of faiss-cpu 1.10.0 broke a previously working install with ragatouille.

Here is the error running "pip install ragatouille"

Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Building wheels for collected packages: faiss-cpu
  Building wheel for faiss-cpu (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for faiss-cpu (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'faiss._swigfaiss' extension
      swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
      swig -python -c++ -Doverride= -doxygen -Ifaiss -I/tmp/pip-build-env-nnb44ti2/overlay/lib/python3.10/site-packages/numpy/_core/include -Ifaiss -I/usr/local/include -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
      swig error : Unrecognized option -doxygen
      Use 'swig -help' for available options.
      error: command '/usr/bin/swig' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Failed to build installable wheels for some pyproject.toml based projects (faiss-cpu)

Confirmed that downgrading faiss-cpu==1.9.0 resulted in successful install
pip install faiss-cpu==1.9.0 # downgrade required for ragatouille, fais-cpu==1.10.0 broke it
pip install ragatouille

Edit:
SWIG Version 3.0.12
OS: Amazon Linux 2 ("centos rhel fedora")

[mnorris11]

Hi @Quinn-Meyer-Sustainment, you're very fast! We haven't completely released 1.10.0 yet. The tagging was done yesterday but the package is not yet released to Pytorch channel due to some Github Actions workflow file issue. After release to the Pytorch channel, we will make the announcement.

However I see you're also installing via pip which we don't officially support atm. Can you try with Conda? We did not see swig error : Unrecognized option -doxygen in any of our CIs.

[Quinn-Meyer-Sustainment]

Ah, that would make sense. Conda only has 1.9.0 available at the moment whereas pypi is using 1.10.0, so unfortunately I cannot try its compatibility using 1.10.0 with Conda right now.

I'm probably a super niche user. I'll monitor this issue again once it's fully released and provide a comment if Conda install works or not. I'm not sure if it makes sense to have an open issue with an unfinished release, so feel free to close this issue if you want!

[mnorris11]

@Quinn-Meyer-Sustainment v1.10.0 is released to Conda now. Let us know if you still see the issue when installing with conda.

[Quinn-Meyer-Sustainment]

I'm on AWS SageMaker Linux which uses Python 3.10 (Python 3.10.16), but the conda installer is saying Python 3.11 (Python 3.11.11 works) is required for faiss-cpu==1.10.0. Python 3.10.16 works with faiss-cpu==1.9.0.

Is this new dependency intended, and if so is it documented somewhere?

Error for "conda install pytorch::faiss-cpu==1.10.0"

Python 3.10.16
Channels:
 - conda-forge
 - nvidia
 - pytorch
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: | warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides _python_rc needed by python-3.13.0rc1-h3b4618f_0_cp313t

Could not solve for environment specs
The following packages are incompatible
├─ faiss-cpu 1.10.0  is installable with the potential options
│  ├─ faiss-cpu 1.10.0 would require
│  │  └─ libfaiss 1.10.0 py3.10_hf65b397_0_cpu, which requires
│  │     └─ mkl 2023.0.* , which can be installed;
│  ├─ faiss-cpu 1.10.0 would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ faiss-cpu 1.10.0 would require
│  │  └─ python >=3.12,<3.13.0a0 , which can be installed;
│  └─ faiss-cpu 1.10.0 would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
├─ mkl-service is installable with the potential options
│  ├─ mkl-service 2.4.0 would require
│  │  └─ mkl >=2021.4.0,<2022.0a0 , which conflicts with any installable versions previously reported;
│  ├─ mkl-service [2.4.0|2.4.1] would require
│  │  └─ mkl >=2023.2.0,<2024.0a0 , which conflicts with any installable versions previously reported;
│  ├─ mkl-service [2.0.1|2.0.2|2.1.0|2.2.0|2.3.0] would require
│  │  └─ python >=2.7,<2.8.0a0 , which can be installed;
│  ├─ mkl-service [2.0.1|2.0.2|...|2.4.0] would require
│  │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│  ├─ mkl-service [2.0.1|2.0.2|...|2.4.0] would require
│  │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│  ├─ mkl-service [2.3.0|2.4.0|2.4.1] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ mkl-service [2.3.0|2.4.0|2.4.1|2.4.2] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  ├─ mkl-service 2.4.0 would require
│  │  └─ mkl >=2022.1.0,<2023.0a0 , which conflicts with any installable versions previously reported;
│  ├─ mkl-service 2.4.0 would require
│  │  └─ mkl >=2022.0.1,<2023.0a0 , which conflicts with any installable versions previously reported;
│  ├─ mkl-service [2.4.0|2.4.1|2.4.2] would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ mkl-service [2.4.0|2.4.1|2.4.2] would require
│  │  └─ python >=3.12,<3.13.0a0 , which can be installed;
│  ├─ mkl-service 2.4.1 would require
│  │  └─ mkl >=2024.2.1,<2025.0a0 , which conflicts with any installable versions previously reported;
│  ├─ mkl-service 2.4.1 would require
│  │  └─ python [>=3.13.0rc1,<3.14.0a0 |>=3.13.0rc2,<3.14.0a0 ] with the potential options
│  │     ├─ python [3.13.0|3.13.1], which can be installed;
│  │     └─ python [3.13.0rc1|3.13.0rc2|3.13.0rc3] would require
│  │        └─ _python_rc, which does not exist (perhaps a missing channel);
│  ├─ mkl-service 2.4.2 would require
│  │  └─ mkl >=2024.2.2,<2025.0a0 , which conflicts with any installable versions previously reported;
│  └─ mkl-service 2.4.2 would require
│     └─ python >=3.13,<3.14.0a0 , which can be installed;
└─ pin-1 is not installable because it requires
   └─ python 3.10.* , which conflicts with any installable versions previously reported.

[mnorris11]

We had to switch from the defaults channel to conda-forge channel this release, so there may be more packaging issues to resolve early on. Hopefully this post will be useful to others too.

We do build python 3.9, 3.10, 3.11, and 3.12 packages in Conda, so it should be supported.

Question: Do you have mkl-service and/or pin-1 installed already in your conda environment?

From the nightly logs when testing faiss-cpu-1.10.0-py3.10_hf65b397_0_cpu_nightly, mkl 2023.0.0 is used. But mkl-service is not in the logs anywhere, so I'm not sure why conda install is requiring it.

You can also try a conda install which also installs mkl=2023.0.0.

pin-1 is not in our logs anywhere either.

2025-02-03T06:53:43.1060261Z     _libgcc_mutex:     0.1-conda_forge                      conda-forge
2025-02-03T06:53:43.1060918Z     _openmp_mutex:     4.5-2_kmp_llvm                       conda-forge
2025-02-03T06:53:43.1061501Z     blas:              1.0-mkl                              conda-forge
2025-02-03T06:53:43.1062059Z     bzip2:             1.0.8-h4bc722e_7                     conda-forge
2025-02-03T06:53:43.1063025Z     ca-certificates:   2025.1.31-hbcca054_0                 conda-forge
2025-02-03T06:53:43.1064135Z     cpython:           3.10.16-py310hd8ed1ab_1              conda-forge
2025-02-03T06:53:43.1064780Z     faiss-cpu:         1.10.0-py3.10_hf65b397_0_cpu_nightly local      
2025-02-03T06:53:43.1065397Z     filelock:          3.17.0-pyhd8ed1ab_0                  conda-forge
2025-02-03T06:53:43.1065982Z     gmp:               6.3.0-hac33072_2                     conda-forge
2025-02-03T06:53:43.1066537Z     gmpy2:             2.1.5-py310he8512ff_3                conda-forge
2025-02-03T06:53:43.1067094Z     jinja2:            3.1.5-pyhd8ed1ab_0                   conda-forge
2025-02-03T06:53:43.1067703Z     ld_impl_linux-64:  2.43-h712a8e2_2                      conda-forge
2025-02-03T06:53:43.1068315Z     libblas:           3.9.0-1_h86c2bf4_netlib              conda-forge
2025-02-03T06:53:43.1068926Z     libcblas:          3.9.0-8_h3b12eaf_netlib              conda-forge
2025-02-03T06:53:43.1069541Z     libfaiss:          1.10.0-py3.10_hf65b397_0_cpu_nightly local      
2025-02-03T06:53:43.1070125Z     libffi:            3.4.2-h7f98852_5                     conda-forge
2025-02-03T06:53:43.1070942Z     libgcc:            14.2.0-h77fa898_1                    conda-forge
2025-02-03T06:53:43.1071523Z     libgcc-ng:         14.2.0-h69a702a_1                    conda-forge
2025-02-03T06:53:43.1072122Z     libgfortran:       14.2.0-h69a702a_1                    conda-forge
2025-02-03T06:53:43.1072760Z     libgfortran-ng:    14.2.0-h69a702a_1                    conda-forge
2025-02-03T06:53:43.1073412Z     libgfortran5:      14.2.0-hd5240d6_1                    conda-forge
2025-02-03T06:53:43.1074224Z     libhwloc:          2.11.2-default_h0d58e46_1001         conda-forge
2025-02-03T06:53:43.1074818Z     libiconv:          1.17-hd590300_2                      conda-forge
2025-02-03T06:53:43.1075459Z     liblapack:         3.9.0-8_h3b12eaf_netlib              conda-forge
2025-02-03T06:53:43.1076067Z     liblief:           0.12.3-h27087fc_0                    conda-forge
2025-02-03T06:53:43.1076652Z     liblzma:           5.6.4-hb9d3cd8_0                     conda-forge
2025-02-03T06:53:43.1077210Z     libnsl:            2.0.1-hd590300_0                     conda-forge
2025-02-03T06:53:43.1077787Z     libsqlite:         3.48.0-hee588c1_1                    conda-forge
2025-02-03T06:53:43.1078378Z     libstdcxx:         14.2.0-hc0a3c3a_1                    conda-forge
2025-02-03T06:53:43.1078982Z     libstdcxx-ng:      14.2.0-h4852527_1                    conda-forge
2025-02-03T06:53:43.1079566Z     libuuid:           2.38.1-h0b41bf4_0                    conda-forge
2025-02-03T06:53:43.1080161Z     libxcrypt:         4.4.36-hd590300_1                    conda-forge
2025-02-03T06:53:43.1080738Z     libxml2:           2.13.5-h0d44e9d_1                    conda-forge
2025-02-03T06:53:43.1081310Z     libzlib:           1.3.1-hb9d3cd8_2                     conda-forge
2025-02-03T06:53:43.1081906Z     llvm-openmp:       19.1.7-h024ca30_0                    conda-forge
2025-02-03T06:53:43.1082540Z     markupsafe:        3.0.2-py310h89163eb_1                conda-forge
2025-02-03T06:53:43.1083144Z     mkl:               2023.0.0-h84fe81f_26648              conda-forge
2025-02-03T06:53:43.1083980Z     mpc:               1.3.1-h24ddda3_1                     conda-forge
2025-02-03T06:53:43.1084494Z     mpfr:              4.2.1-h90cbb55_3                     conda-forge
2025-02-03T06:53:43.1085052Z     mpmath:            1.3.0-pyhd8ed1ab_1                   conda-forge
2025-02-03T06:53:43.1085611Z     ncurses:           6.5-h2d0b736_3                       conda-forge
2025-02-03T06:53:43.1086188Z     networkx:          3.4.2-pyh267e887_2                   conda-forge
2025-02-03T06:53:43.1086743Z     numpy:             1.26.4-py310hb13e2d6_0               conda-forge
2025-02-03T06:53:43.1087308Z     openssl:           3.4.0-h7b32b05_1                     conda-forge
2025-02-03T06:53:43.1088116Z     packaging:         24.2-pyhd8ed1ab_2                    conda-forge
2025-02-03T06:53:43.1088715Z     pip:               25.0-pyh8b19718_0                    conda-forge
2025-02-03T06:53:43.1089260Z     python:            3.10.16-he725a3c_1_cpython           conda-forge
2025-02-03T06:53:43.1089863Z     python_abi:        3.10-5_cp310                         conda-forge
2025-02-03T06:53:43.1090455Z     pytorch:           2.0.1-py3.10_cpu_0                   pytorch    
2025-02-03T06:53:43.1091062Z     pytorch-mutex:     1.0-cpu                              pytorch    
2025-02-03T06:53:43.1091663Z     readline:          8.2-h8228510_1                       conda-forge
2025-02-03T06:53:43.1092235Z     scipy:             1.15.1-py310hfa6ec8c_0               conda-forge
2025-02-03T06:53:43.1092826Z     setuptools:        75.8.0-pyhff2d567_0                  conda-forge
2025-02-03T06:53:43.1093407Z     sympy:             1.13.3-pyh2585a3b_105                conda-forge
2025-02-03T06:53:43.1094108Z     tbb:               2021.13.0-hceb3a55_1                 conda-forge
2025-02-03T06:53:43.1094625Z     tk:                8.6.13-noxft_h4845f30_101            conda-forge
2025-02-03T06:53:43.1095238Z     typing_extensions: 4.12.2-pyha770c72_1                  conda-forge
2025-02-03T06:53:43.1096055Z     tzdata:            2025a-h78e105d_0                     conda-forge
2025-02-03T06:53:43.1096603Z     wheel:             0.45.1-pyhd8ed1ab_1                  conda-forge

[Quinn-Meyer-Sustainment]

Good call. AWS has the following MKL libraries preconfigured in their environment:

mkl                       2024.2.2            ha957f24_16    conda-forge
mkl-devel                 2024.2.2            ha770c72_16    conda-forge
mkl-include               2024.2.2            ha957f24_16    conda-forge
mkl-service               2.4.2           py310h22455d7_0    conda-forge
mkl_fft                   1.3.11          py310h5bcb89a_0    conda-forge

uninstalling prevoius mkl versions

conda uninstall mkl -y
conda install faiss-cpu==1.10.0 -y

fixed the issue.

On a side note, when installing via the AWS pytorch kernel it automatically updates pytorch which causes installation failures. I'm not a packaging expert, but it does not happen in the same kernel with faiss-cpu==1.9.0.

The following packages will be UPDATED:

  blas                                              1.0-mkl --> 2.128-openblas 
  libblas                              3.9.0-16_linux64_mkl --> 3.9.0-28_h59b9bed_openblas 
  libcblas                             3.9.0-16_linux64_mkl --> 3.9.0-28_he106b2a_openblas 
  liblapack                            3.9.0-16_linux64_mkl --> 3.9.0-28_h7ac8fdf_openblas 
  pytorch            pytorch::pytorch-2.2.2-py3.10_cuda12.~ --> conda-forge::pytorch-2.4.1-cpu_generic_py310hcbfaffa_0 
  torchaudio                              2.2.2-py310_cu121 --> 2.4.1-py310_cu121 
  torchvision                            0.17.2-py310_cu121 --> 0.19.1-py310_cu121 

[mnorris11]

Glad to see that uninstalling prior mkl versions worked!

I'm not as familiar with the dependencies for pytorch unfortunately but it looks like it is trying to update Pytorch 2.2.2 to 2.4.1. What happens if you install the lower Pytorch version? Our test logs were using pytorch: 2.0.1 as copied above.