[BUG] pip.installed: handles incorrectly modules with special characters in the name #66988
Comments
|
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. |
Similarly to `pip.list`, normalize package names before processing them. In this case most of the work is already done - most functions invoke `_check_package_version_format` and use the processed package name that it returns. Thus, to use the same normalization algorithm everywhere, make that function use `pip.normalize`, too. This, among other things, deals with some "There was no error installing package... although it does not show..." warnings when the package name contains underscores, since now we look for the correctly normalized name and we can find it. Fixes saltstack#66988
Similarly to `pip.list`, normalize package names before processing them. In this case most of the work is already done - most functions invoke `_check_package_version_format` and use the processed package name that it returns. Thus, to use the same normalization algorithm everywhere, make that function use `pip.normalize`, too. This, among other things, deals with some "There was no error installing package... although it does not show..." warnings when the package name contains underscores, since now we look for the correctly normalized name and we can find it. Fixes saltstack#66988
Similarly to `pip.list`, normalize package names before processing them. In this case most of the work is already done - most functions invoke `_check_package_version_format` and use the processed package name that it returns. Thus, to use the same normalization algorithm everywhere, make that function use `pip.normalize`, too. This, among other things, deals with some "There was no error installing package... although it does not show..." warnings when the package name contains underscores, since now we look for the correctly normalized name and we can find it. Fixes saltstack#66988
Description
Hi,
First of all, thanks for writing and maintaining SaltStack!
The problem with Python package names in general
Historically, there have always been some issues regarding the several
different types of names for Python packages:
the Python Package Index
its installation and where the Python interpreter will find its modules
The main issue revolves around the need for at least the last kind of name -
the one used for files and directories on the filesystem - to only contain
characters valid for a Python identifier, so that an
importstatement willbe able to handle it. The only "special" characters allowed there are the dot and
the underscore, with even the dot posing problems in some OS environments.
Over the years, the various package installers (e.g. pip) and package
repositories (e.g. PyPI, devpi, etc.) have allowed the package metadata and
the package repositories to specify names with additional characters,
most notably dashes, and sometimes provide (more or less automatically)
some kind of correspondence or a mapping algorithm to "normalize" these
names to ones only containing alphanumeric characters, underscores, and
possibly dots.
PEP 503 - the recommended normalization algorithm
Back in 2015, a Python Enhancement Proposal -
PEP 503 - was accepted that
defines and strongly recommends what has become more or less the canonical
algorithm for the normalization of Python package names. That algorithm is
succintly explained in
the Python Packaging User Guide
which also provides the sample Python implementation from the PEP and
recommends that this normalization be performed on any package names
(specified by the user, fetched from a package index, extracted from
a package metadata, or listed as installed by a local package manager)
before any comparisons are made and any decisions are taken whether to
install or upgrade any of them.
The problem with Python package names in SaltStack
The
pip.installedstate allows Python modules to be installed froma variety of sources, including PyPI. If a
pip.installedstatespecifies in its list of packages a name that may be found in the source,
but is not the same as the name from the package metadata, then
the state will download the package, invoke the
piptool to install it,but will then be unable to verify the installation: a warning will be
returned saying that the package could not be found after the installation.
Even worse, in later runs the
pip.installedstate will not be able tofind the package among the ones installed, so it will needlessly download and
install it again and also report its own status as "changed", so later
states may also perform unnecessary work.
The disconnect comes from the fact that the user-supplied package name is
not the same as the one obtained from asking the
piptool for the list ofinstalled packages.
An example of that may be seen with the following Salt configuration:
The proposed solution - normalize package names in several places
One possible solution to this problem is to modify both the
pipSaltmodule and the
pip.installedSalt state as follows:normalizefunction to thepipmodule that converts the suppliedpackage name string to the canonical form as stated in PEP 503 and
in the Python Packaging User Guide
pip.listalways return normalized package names (this is a changeto the current behavior!) and also normalize the names before comparing them
if a prefix is specified
pip.installedstate normalize the supplied package names andlook for their canonical forms in the
pip.listoutput (that is alreadynormalized) so that the packages will be found if already installed and
will also be found after they are installed or upgraded as necessary
A minor drawback of this proposed solution is that any modules or states that
use the list of package names returned by
pip.listmay now see slightlydifferent output. However, I believe that this should not cause any real
problems, at least not any more than in the current situation - right now,
the handling of Python modules with discrepancies between the different kinds of
names is already suboptimal.
Setup
Steps to Reproduce the behavior
Comment: There was no error installing package 'requests_ntlm' although it does not show when calling 'pip.freeze'.
Expected behavior
The package is installed with no warnings.
Versions Report
salt --versions-report
The text was updated successfully, but these errors were encountered: