Fix for PIP Inspector not working when pkg_resources package is not available in newer Python versions (#1287)

* Fix for PIP Inspector not working when pkg_resources package is not available in newer Python versions

* Strip whitespace from package name before attempting lookup

* Uniquify requirement list per package

* Use pip internal search_packages_info to look up packages when possible

* Minor refactor

* Update docs to reflect the updated pip inspector implementation

* Update comments and minor refactor

* Simplify decision around which method of package lookup to use

* Update date

---------

Co-authored-by: Andrian Sevastyanov <[email protected]>

Author: andrian-sevastyanov

documentation/src/main/markdown/packagemgrs/python.md

@@ -21,7 +21,7 @@
 
 Setuptools detector attempts to run on your project if a pyproject.toml file containing a build section with `requires = ["setuptools"]` or equivalent line is located, and a pip installation is found. (Setuptools scans can be run in both build, if a pip installation is available, and buildless mode, if not.)
 
-<note type="note">Setuptools build detector should be run in a virtual environment, or environment with a clean global pip cache, where a pip install has only been performed for the project being scanned. If you are on Python v3.12 or later, run `pip install setuptools` (before executing [detect_product_short]) to ensure required tools are installed.</note>
+<note type="note">Setuptools build detector should be run in a virtual environment, or environment with a clean global pip cache, where a pip install has only been performed for the project being scanned.</note>
 
 [detect_product_short] parses the pyproject.toml file determining if the `[build-system]` section has been configured for Setuptools Pip via the `requries= ["setuptools"]` setting. If the setting is located and pip is installed in the environment, either in the default location or specified via the `--detect.pip.path` property, the detector will execute in a virtual environment, if configured as suggested, and analyze the pyproject.toml, setup.cfg, or setup.py files for dependencies. If the detector discovers a configured pyproject.toml file but not a pip executible, it will execute in buildless mode where it will parse dependencies from the pyproject.toml, setup.cfg, or setup.py files but may not be able to specify exact package versions. If no dependencies are located in the pyproject.toml, setup.cfg, or setup.py files, or if the detector fails, the BDIO file output will not be generated in build or buildless mode. [detect_product_short] will also attempt to run additional detectors if their execution requirements are met.
 
@@ -69,10 +69,10 @@ Pip Native Inspector requires Python and pip executables.
 
 Pip Native Inspector runs the [pip-inspector.py script](https://github.com/blackducksoftware/detect/blob/master/src/main/resources/pip-inspector.py), which uses Python/pip libraries to query the pip cache for the project, which may or may not be a virtual environment, for dependency information:
 
-1. pip-inspector.py queries for the project dependencies by project name which can be discovered using setup.py, or provided using the detect.pip.project.name property, using the [pkg_resources library](https://setuptools.readthedocs.io/en/latest/pkg_resources.html). If your project is installed into the pip cache, this discovers dependencies specified in setup.py.
-1. If one or more requirements files are found or provided, pip-inspector.py uses the Python API called parse_requirements to query each requirements file for possible additional dependencies, and uses the pkg_resources library to query for the details of each.
+1. pip-inspector.py queries for the project dependencies by project name which can be discovered using setup.py, or provided using the detect.pip.project.name property. If your project is installed into the pip cache, this discovers dependencies specified in setup.py.
+1. If one or more requirements files are found or provided, pip-inspector.py queries each requirements file for possible additional dependencies and details of each.
 
-<note type="tip">pip-inspector.py uses the pkg_resources library to discover dependencies, only those packages which have been installed; using, for example, `pip install`, into the pip cache and appearing in the output of `pip list`, are included in the output. There must be a match between the package version on which your project depends and the package version installed in the pip cache. Additional details are available in the [pkg_resources library documentation](https://setuptools.readthedocs.io/en/latest/pkg_resources.html).</note>   
+<note type="tip">Only those packages which have been installed; using, for example, `pip install`, into the pip cache and appearing in the output of `pip list`, are included in the output of pip-inspector.py. There must be a match between the package version on which your project depends and the package version installed in the pip cache.</note>
 <note type="note">If the packages are installed into a virtual environment for your project, you must run [detect_product_short] from within that virtual environment.</note>
 
 ### Recommendations for Pip Detector

src/main/resources/pip-inspector.py

@@ -1,6 +1,6 @@
 # pylint: disable=fixme, line-too-long, import-error, no-name-in-module
 #
-# Copyright (c) 2024 Black Duck Software, Inc.
+# Copyright (c) 2025 Black Duck Software Inc.
 #
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements. See the NOTICE file
@@ -35,7 +35,6 @@
 from os import path
 import sys
 from re import split
-from pkg_resources import working_set, Requirement
 
 import pip
 pip_major_version = int(pip.__version__.split(".")[0])
@@ -97,9 +96,6 @@ def resolve_project_node(project_name):
 def populate_dependency_tree(project_root_node, requirements_path):
     """Resolves the dependencies of the user-provided requirements.txt and appends them to the dependency tree"""
     try:
-        # This line is pretty much the only reason why we call the internal pip APIs anymore. We should consider if we
-        # can do this with a more generalized approach.
-        # --rotte DEC 2020
         parsed_requirements = parse_requirements(requirements_path, session=PipSession())
         for parsed_requirement in parsed_requirements:
             package_name = None
@@ -128,43 +124,66 @@ def populate_dependency_tree(project_root_node, requirements_path):
 
 def recursively_resolve_dependencies(package_name, history):
     """Forms a DependencyNode by recursively resolving its dependencies. Tracks history for cyclic dependencies."""
-    package = get_package_by_name(package_name)
+    dependency_node, child_names = get_package_by_name(package_name)
 
-    if package is None:
+    if dependency_node is None:
         return None
 
-    dependency_node = DependencyNode(package.project_name, package.version)
-
-    if package_name.lower() not in history:
-        history.append(package_name.lower())
-        for package_dependency in package.requires():
-            child_node = recursively_resolve_dependencies(package_dependency.key, history)
+    if dependency_node.name not in history:
+        history.append(dependency_node.name)
+        for child_name in child_names:
+            child_node = recursively_resolve_dependencies(child_name, history)
             if child_node is not None:
                 dependency_node.children = dependency_node.children + [child_node]
 
     return dependency_node
 
+use_pip_internal_to_search_packages = False
+if sys.version_info.major > 3 or sys.version_info.major == 3 and sys.version_info.minor >= 12:
+    # Python version 3.12 is used as a cutoff for the following reasons:
+    #  * it is the version in which pkg_resources is no longer included by default
+    #  * a test confirmed that this version of python is incompatible with PIP versions below 21.2,
+    #    which was the most recent version in which PIP search_packages_info interface changed
+    use_pip_internal_to_search_packages = True
 
-def get_package_by_name(package_name):
-    """Looks up a package from the pip cache"""
-    if package_name is None:
-        return None
+if use_pip_internal_to_search_packages:
+    from pip._internal.commands.show import search_packages_info
 
-    package_dict = working_set.by_key
-    try:
-        # TODO: By using pkg_resources.Requirement.parse to get the correct key, we may not need to attempt the other
-        #     methods. Robust tests are needed to confirm.
-        return package_dict[Requirement.parse(package_name).key]
-    except:
-        pass
+    def get_package_by_name(package_name):
+        """Looks up a package from the pip cache using internal pip API.
+        This should not be used when PIP is older than 21.2 since the internal API was slightly different then.
+        """
+        if package_name is None:
+            return None, None
 
-    name_variants = (package_name, package_name.lower(), package_name.replace('-', '_'), package_name.replace('_', '-'))
-    for name_variant in name_variants:
-        if name_variant in package_dict:
-            return package_dict[name_variant]
+        package_info = next(search_packages_info([package_name.strip()]), None)
 
-    return None
+        if package_info is None:
+            return None, None
 
+        return DependencyNode(package_info.name, package_info.version), package_info.requires
+else:
+    from pkg_resources import working_set, Requirement
+
+    def get_package_by_name(package_name):
+        """Looks up a package from the pip cache using pkg_resouces"""
+        if package_name is None:
+            return None, None
+
+        package = None
+
+        package_dict = working_set.by_key
+        try:
+            package = package_dict[Requirement.parse(package_name).key]
+        except:
+            name_variants = (package_name, package_name.lower(), package_name.replace('-', '_'), package_name.replace('_', '-'))
+            for name_variant in name_variants:
+                if name_variant in package_dict:
+                    return package_dict[name_variant]
+
+        if package is None:
+            return None, None
+        return DependencyNode(package.project_name, package.version), [requirement.key for requirement in package.requires()]
 
 class DependencyNode(object):
     """Represents a python dependency in a tree graph with a name, version, and array of children DependencyNodes"""