Tests: Add additional log output to the editable package tests #1253
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The editable package tests now output the contents of all of the `.pth` and `.egg-link` files in `site-packages` to aid debugging, as well as help demonstrate the changes that will be taking place in #1253. GUS-W-10047026. [skip changelog]
doriskwan
approved these changes
Oct 19, 2021
edmorley
added a commit
that referenced
this pull request
Oct 19, 2021
Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.)
edmorley
added a commit
that referenced
this pull request
Oct 19, 2021
…#1252) Currently the build system performs builds in a different directory (`/tmp/build_<hash>`) to which the app will be run at runtime (`/app`). This means that any hardcoded paths in the slug must be rewritten by the buildpack, so the app still works after being moved. One such case of hardcoded paths, are the `.pth` and `.egg-link` files that are created in the `site-packages` directory by Pip/setuptools for editable package installs (aka develop mode). The most common way someone might use editable mode is via `-e <package specifier>` entries in their `requirements.txt` file. Until now, the Python buildpack rewrote paths inside these files during the compile itself, which meant the build-time paths were no longer present when subsequent buildpacks ran. This happened to work due to an interaction of legacy setuptools behaviour and a buildpack bug, but stops working in setuptools 47.2.0 or later - as described in #1006. Longer term we would like to stop building in one location and running the app from another, so that the path rewriting isn't required at all. However that change breaks some other buildpacks so requires a long-term transition plan. In the meantime, this change moves path rewriting to a `.profile.d/` script, so that it occurs at runtime, and so after all other buildpacks have run. Additional test coverage of editable packages was added previously in #1251 and #1253, and has confirmed that this new `profile.d/` script approach will prevent the issues in #1006 when setuptools is updated in a future PR. There is one subtle implication of moving this path rewriting, in that subsequent cached builds will no longer see the existing package as being already installed, so won't uninstall if before reinstalling it (as seen in the test log output change). However this is not believed to have any significant impact. Fixes #1006. (And so unblocks updating to the newer setuptools required for #1248.) GUS-W-7828034.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This expands on the editable package tests added in #1251, making them now output the contents of all of the
.pthand.egg-linkfiles insite-packagesto aid debugging, as well as help demonstrate the changes that will be taking place in #1252.GUS-W-10047026.
[skip changelog]