Perform editable package .pth and .egg-link path rewriting at runtime (#1252)

Currently the build system performs builds in a different directory
(`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`).
This means that any hardcoded paths in the slug must be rewritten by the
buildpack, so the app still works after being moved.

One such case of hardcoded paths, are the `.pth` and `.egg-link` files
that are created in the `site-packages` directory by Pip/setuptools for
editable package installs (aka develop mode). The most common way
someone might use editable mode is via `-e <package specifier>` entries
in their `requirements.txt` file.

Until now, the Python buildpack rewrote paths inside these files during
the compile itself, which meant the build-time paths were no longer
present when subsequent buildpacks ran. This happened to work due to
an interaction of legacy setuptools behaviour and a buildpack bug, but
stops working in setuptools 47.2.0 or later - as described in #1006.

Longer term we would like to stop building in one location and running
the app from another, so that the path rewriting isn't required at all.
However that change breaks some other buildpacks so requires a long-term
transition plan.

In the meantime, this change moves path rewriting to a `.profile.d/`
script, so that it occurs at runtime, and so after all other buildpacks
have run.

Additional test coverage of editable packages was added previously in #1251
and #1253, and has confirmed that this new `profile.d/` script approach will
prevent the issues in #1006 when setuptools is updated in a future PR.

There is one subtle implication of moving this path rewriting, in that
subsequent cached builds will no longer see the existing package as
being already installed, so won't uninstall if before reinstalling it (as
seen in the test log output change). However this is not believed to
have any significant impact.

Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.)
GUS-W-7828034.

Author: edmorley

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- Perform editable package `.pth` and `.egg-link` path rewriting at runtime ([#1252](https://github.com/heroku/heroku-buildpack-python/pull/1252)).
 
 ## v200 (2021-10-04)
 

Makefile

@@ -14,7 +14,7 @@ lint: lint-scripts lint-ruby
 
 lint-scripts:
 	@shellcheck -x bin/compile bin/detect bin/release bin/test-compile bin/utils bin/warnings bin/default_pythons
-	@shellcheck -x bin/steps/collectstatic bin/steps/eggpath-fix  bin/steps/eggpath-fix2 bin/steps/nltk bin/steps/pip-install bin/steps/pipenv bin/steps/pipenv-python-version bin/steps/python
+	@shellcheck -x bin/steps/collectstatic bin/steps/nltk bin/steps/pip-install bin/steps/pipenv bin/steps/pipenv-python-version bin/steps/python
 	@shellcheck -x bin/steps/hooks/*
 
 lint-ruby:

bin/compile

@@ -264,12 +264,6 @@ if [ ! -f requirements.txt ] && [ ! -f Pipfile ]; then
   echo "-e ." > requirements.txt
 fi
 
-# Fix egg-links.
-# Because we're installing things into a different path than we're running them (temp dir vs app dir),
-# We must re-write all of Python's eggpath links to target the proper directory.
-# shellcheck source=bin/steps/eggpath-fix
-source "$BIN_DIR/steps/eggpath-fix"
-
 # SQLite3 support.
 # This sets up and installs sqlite3 dev headers and the sqlite3 binary but not the
 # libsqlite3-0 library since that exists on the stack image.
@@ -345,6 +339,17 @@ if [[ \$HOME != "/app" ]]; then
 fi
 EOT
 
+# At runtime, rewrite paths in editable package .egg-link and .pth files from the build time paths
+# (such as `/tmp/build_<hash>`) back to `/app`. This is not done during the build itself, since later
+# buildpacks still need the build time paths. The `/lib*/*/` glob is to ensure it matches against both:
+# - lib/python3.NN/site-packages/ (CPython)
+# - lib-python/3/site-packages/ (PyPy)
+if [[ "${BUILD_DIR}" != "/app" ]]; then
+  cat <<EOT >> "$PROFILE_PATH"
+find .heroku/python/lib*/*/site-packages/ -type f -and \( -name '*.egg-link' -or -name '*.pth' \) -exec sed -i -e 's#${BUILD_DIR}#/app#' {} \+
+EOT
+fi
+
 # Install sane-default script for $WEB_CONCURRENCY and $FORWARDED_ALLOW_IPS.
 cp "$ROOT_DIR/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH"
 cp "$ROOT_DIR/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH"
@@ -353,10 +358,6 @@ cp "$ROOT_DIR/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH"
 # shellcheck source=bin/steps/hooks/post_compile
 source "$BIN_DIR/steps/hooks/post_compile"
 
-# Fix egg-links, again.
-# shellcheck source=bin/steps/eggpath-fix2
-source "$BIN_DIR/steps/eggpath-fix2"
-
 # Store new artifacts in the cache.
 rm -rf "$CACHE_DIR/.heroku/python"
 rm -rf "$CACHE_DIR/.heroku/python-version"

bin/steps/eggpath-fix

@@ -118,7 +118,7 @@
           remote: -----> Inline app detected
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
           remote: /app/.heroku/src/gunicorn
-          remote: /app/local_package
+          remote: /tmp/build_.*/local_package
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           remote: /app/.heroku/src/gunicorn
@@ -140,7 +140,7 @@
           /app/.heroku/src/gunicorn
           .
           ==> .heroku/python/lib/python.*/site-packages/local-package.egg-link <==
-          /tmp/build_.*/local_package
+          /app/local_package
           .
           Running entrypoint for the local package: Hello!
           Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
@@ -159,17 +159,13 @@
           remote:        Obtaining gunicorn from git\\+https://github.com/benoitc/[email protected]#egg=gunicorn \\(from -r /tmp/build_.*/requirements.txt \\(line 2\\)\\)
           remote:          Cloning https://github.com/benoitc/gunicorn \\(to revision 20.1.0\\) to /app/.heroku/src/gunicorn
           remote:        Installing collected packages: gunicorn, local-package
-          remote:          Attempting uninstall: gunicorn
-          remote:            Found existing installation: gunicorn 20.1.0
-          remote:            Uninstalling gunicorn-20.1.0:
-          remote:              Successfully uninstalled gunicorn-20.1.0
           remote:          Running setup.py develop for gunicorn
           remote:          Running setup.py develop for local-package
           remote:        Successfully installed gunicorn local-package
           remote: -----> Running post-compile hook
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
-          remote: /tmp/build_.*/local_package
           remote: /app/.heroku/src/gunicorn
+          remote: /tmp/build_.*/local_package
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           remote: /app/.heroku/src/gunicorn
@@ -181,8 +177,8 @@
           remote: Running entrypoint for the VCS package: gunicorn \\(version 20.1.0\\)
           remote: -----> Inline app detected
           remote: ==> .heroku/python/lib/python.*/site-packages/easy-install.pth <==
-          remote: /app/local_package
           remote: /app/.heroku/src/gunicorn
+          remote: /tmp/build_.*/local_package
           remote: 
           remote: ==> .heroku/python/lib/python.*/site-packages/gunicorn.egg-link <==
           remote: /app/.heroku/src/gunicorn