pip's dependency resolver error

[Ana06]

Details

Both locally and in daily CI, I see the following error in the log:

2024/09/03 09:29:54 [installer.vm] chocolateyinstall.ps1 [+] ERROR : Failed to install: magika.vm
2024/09/03 09:29:54 [installer.vm] chocolateyinstall.ps1 [+] ERROR : Failed to install: malware-jail.vm

It seems to be caused by:

ERROR: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

The tools appear to be installed correctly. So this is very confusing and should investigate how to fix it.

[emtuls]

NOTE: This is actually 2 separate issues. One for magika and separate issue affecting malware-jail.

This specific pip issue is mostly due to our installation of Python tools being installed separately and thus, Python's dependency resolver is unable to properly satisfy any dependency issues, which is what is displaying the error.

The primary underlying issue is that magika version 0.5.1 had an error in their code that was recently fixed in 0.6.0rc1 (currently unreleased on pypi) which is what made numpy version >=1.26 be required even for Python version <3.12 when it should have only needed >=1.24: google/magika@ebac72a

And because we have other python tools installed via libraries.python3.vm, their dependencies are not checked against this and thus more specifically we run into an issue because StringSifter requires numpy==1.24.4: https://github.com/mandiant/stringsifter/blob/33c0cd5538bf4dc499505b63f89bc045b191a0df/requirements.txt#L110
So when numpy is updated to 1.26 because of magika, we get an error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
stringsifter 3.20230711 requires numpy==1.24.4, but you have numpy 1.26.4 which is incompatible.

This can be resolved immediately by just using magika version 0.5.0, and should hopefully be resolved when version 0.6.0 is officially released.

The second issue is that malware-jail has not been updated in a long time, and thus many packages it uses are deprecated, which in turn raises WARN issues within npm, causing the package to seem to "fail":