-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ability to use relative python executable paths in scripts exe files #13162
Comments
I don't have almost any experience here, but pip uses distlib to make the entry points / executables, you may want to look at the docs / logic there. There has been some recent talk about making virtual environments relocatable. And I think that has been a goal for uv / python-build-standalone projects, but I don't know much that transfers over to entry points / executables. |
Entry point wrappers by design use the absolute path to the interpreter. It allows symlinking or copying the wrapper without needing to copy the whole venv - this is critical to how pipx works, to give one example. Having a non-default opt-in that uses relative paths might be possible, or alternatively having a utility to make wrappers relocatable (the latter having the benefit that anyone could write such a thing - as the OP said, it’s possible, just messy). But changing the existing default behaviour would, IMO, be too disruptive for too little benefit. |
@aarismendi you can serve your use case today IIUC in many ways. One that uses Pip, works offline and is re-distributable on Windows machines utilizes:
I'm on Linux, but this also works for Windows. Just adjust some of the lift.toml paths for those found in a Windows venv ( :; python3.12 -mvenv tools.venv
# Prepare the resolve needed for offline use:
:; tools.venv/bin/python -mpip -q wheel MapProxy --wheel-dir wheels
# Install science to help build a scie with:
# N.B.: The `science` executable `insta-science` forwards to is a Python app written using itself; so dogfoods all this technology (although it uses shiv instead of building a venv at runtime) for both its windows-x86_64 and windows-aarch64 releases. Do note though that the windows-aarch64 `science` app as well as any of your own you create this way will run an x86_64 interpreter via PRISM until PBS can get out a native Windows arm interpreter distribution release.
:; tools.venv/bin/python -mpip -q install insta-science
# Write up our app scie lift manifest. This encodes the 2-step install process (create a venv, install wheels from wheel house into it) as well as the default entry-point - the un-named command that launches the mapproxy-util console script:
:; cat lift.toml
[lift]
name = "mapproxy-util"
[[lift.files]]
name = "wheels_dir"
[[lift.interpreters]]
id = "cpython"
provider = "PythonBuildStandalone"
version = "3.12.8"
[[lift.commands]]
exe = "{scie.bindings.install}/venv/bin/mapproxy-util"
[[lift.bindings]]
name = "install"
exe = "{scie.bindings.venv}/venv/bin/python"
args = [
"-mpip",
"--isolated",
"--no-cache-dir",
"install",
"--only-binary=:all:",
"--no-index",
"--find-links={wheels_dir}",
"MapProxy"
]
[[lift.bindings]]
name = "venv"
exe = "#{cpython:python}"
args = ["-mvenv", "{scie.bindings}/venv"]
# Build the scie:
:; insta-science lift --file wheels_dir=wheels/ build lift.toml
Downloading https://github.com/a-scie/lift/releases/latest/download/science-fat-linux-x86_64 ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29.8M/29.8M [00:01<00:00, 21.6MB/s]
warning: Failed to gather git state for provenance.
Downloading https://github.com/a-scie/jump/releases/latest/download/scie-jump-linux-x86_64 ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.75M/1.75M [00:00<00:00, 18.4MB/s]
Downloading https://github.com/astral-sh/python-build-standalone/releases/download/20250106/cpython-3.12.8%2B20250106-x86_64-unknown-linux-gnu-install_only.tar.gz ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 64.0M/64.0M [00:01<00:00, 47.3MB/s]
/tmp/example/mapproxy-util
# N.B.: A single file native executable:
:; file /tmp/example/mapproxy-util
/tmp/example/mapproxy-util: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, stripped
# With a zip trailer that contains a PythonBuildStandalone re-distributable Python as well as a wheel house.
# Note the `75329426 extra bytes` at the head of the zip include the `scie-jump` launcher and the PBS re-distributable Python archive:
:; zipinfo /tmp/example/mapproxy-util
Archive: /tmp/example/mapproxy-util
Zip file size: 93029676 bytes, number of entries: 13
warning [/tmp/example/mapproxy-util]: 75329426 extra bytes at beginning or within zipfile
(attempting to process anyway)
-rw-rw-r-- 4.6 unx 1737428 b- defN 80-Jan-01 00:00 MapProxy-3.1.3-py3-none-any.whl
-rw-rw-r-- 4.6 unx 88462 b- defN 80-Jan-01 00:00 jsonschema-4.23.0-py3-none-any.whl
-rw-rw-r-- 4.6 unx 4496081 b- defN 80-Jan-01 00:00 pillow-11.1.0-cp312-cp312-manylinux_2_28_x86_64.whl
-rw-rw-r-- 4.6 unx 9525831 b- defN 80-Jan-01 00:00 pyproj-3.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 4.6 unx 767542 b- defN 80-Jan-01 00:00 PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 4.6 unx 224498 b- defN 80-Jan-01 00:00 werkzeug-3.1.3-py3-none-any.whl
-rw-rw-r-- 4.6 unx 491326 b- defN 80-Jan-01 00:00 future-1.0.0-py3-none-any.whl
-rw-rw-r-- 4.6 unx 63397 b- defN 80-Jan-01 00:00 attrs-24.3.0-py3-none-any.whl
-rw-rw-r-- 4.6 unx 18459 b- defN 80-Jan-01 00:00 jsonschema_specifications-2024.10.1-py3-none-any.whl
-rw-rw-r-- 4.6 unx 23118 b- defN 80-Jan-01 00:00 MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 4.6 unx 26684 b- defN 80-Jan-01 00:00 referencing-0.35.1-py3-none-any.whl
-rw-rw-r-- 4.6 unx 385721 b- defN 80-Jan-01 00:00 rpds_py-0.22.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 4.6 unx 164927 b- defN 80-Jan-01 00:00 certifi-2024.12.14-py3-none-any.whl
13 files, 18013474 bytes uncompressed, 17697053 bytes compressed: 1.8%
# Now ship the single scie file to some other Windows machine and run it.
# It installs itself (offline) on the 1st run as evidenced by Pip install output here I did not quiet in the lift manifest install binding:
:; /tmp/example/mapproxy-util --help
Looking in links: /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/MapProxy-3.1.3-py3-none-any.whl
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/pillow-11.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/future-1.0.0-py3-none-any.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/jsonschema-4.23.0-py3-none-any.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/pyproj-3.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/werkzeug-3.1.3-py3-none-any.whl (from MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/attrs-24.3.0-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/jsonschema_specifications-2024.10.1-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/referencing-0.35.1-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/rpds_py-0.22.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from jsonschema>=4->MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/certifi-2024.12.14-py3-none-any.whl (from pyproj>=2->MapProxy)
Processing /home/jsirois/.cache/nce/aa3c071f11ea0a63dac93d424586947365e5aa5861a624186a417c00f3c899ac/wheels_dir/MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from werkzeug<4->MapProxy)
Installing collected packages: rpds-py, PyYAML, Pillow, MarkupSafe, future, certifi, attrs, werkzeug, referencing, pyproj, jsonschema-specifications, jsonschema, MapProxy
Successfully installed MapProxy-3.1.3 MarkupSafe-3.0.2 Pillow-11.1.0 PyYAML-6.0.2 attrs-24.3.0 certifi-2024.12.14 future-1.0.0 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 pyproj-3.7.0 referencing-0.35.1 rpds-py-0.22.3 werkzeug-3.1.3
Usage: mapproxy-util COMMAND [options]
Commands:
serve-develop Run MapProxy development server.
serve-multiapp-develop Run MultiMapProxy development server.
create Create example configurations.
scales Convert between scales and resolutions.
wms-capabilities Display WMS capabilites.
grids Display detailed informations for configured grids.
export Export existing caches.
autoconfig Create config from WMS capabilities or a Geopackage file.
defrag-compact-cache De-fragmentate compact caches.
# Run the scie again, install is skipped - you have a native venv Python app with a few ms of extra overhead to run the `scie-jump` dispatcher at the head of your binary and have it re-direct to this venv:
:; /tmp/example/mapproxy-util --help
Usage: mapproxy-util COMMAND [options]
Commands:
serve-develop Run MapProxy development server.
serve-multiapp-develop Run MultiMapProxy development server.
create Create example configurations.
scales Convert between scales and resolutions.
wms-capabilities Display WMS capabilites.
grids Display detailed informations for configured grids.
export Export existing caches.
autoconfig Create config from WMS capabilities or a Geopackage file.
defrag-compact-cache De-fragmentate compact caches.
And, as @pfmoore pointed out - an absolute path by design, but in this case supporting an app that is both re-locatable on the current machine or across networks to other machines (via the higher-layer :; cat /home/jsirois/.cache/nce/48befbbe619cebf69f17c57090fbd52f7bbd30b9589dcafaa332bb73f6abc041/bindings/venv/bin/mapproxy-util
#!/home/jsirois/.cache/nce/48befbbe619cebf69f17c57090fbd52f7bbd30b9589dcafaa332bb73f6abc041/bindings/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from mapproxy.script.util import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main()) |
And the Windows demo: # N.B.: I'm on Windows ARM; so uv makes it easy to get an x86_64 venv to avoid wheel bulding issues irrelevant to this demonstration.
:; uv venv --python cpython-3.12.8-windows-x86_64-none --seed tools.venv
Using CPython 3.12.8
Creating virtual environment with seed packages at: tools.venv
+ pip==24.3.1
Activate with: source tools.venv/Scripts/activate
# Resolve wheel house:
:; tools.venv/Scripts/python.exe -mpip -q wheel MapProxy --wheel-dir wheels
# Install insta-science:
:; tools.venv/Scripts/python.exe -mpip -q install insta-science
# Write same lift manifest as before with small Windows venv path tweaks:
:; cat lift.toml
[lift]
name = "mapproxy-util"
[[lift.files]]
name = "wheels_dir"
[[lift.interpreters]]
id = "cpython"
provider = "PythonBuildStandalone"
version = "3.12.8"
[[lift.commands]]
exe = "{scie.bindings.install}/venv/Scripts/mapproxy-util.exe"
[[lift.bindings]]
name = "install"
exe = "{scie.bindings.venv}/venv/Scripts/python.exe"
args = [
"-mpip",
"--isolated",
"--no-cache-dir",
"install",
"--only-binary=:all:",
"--no-index",
"--find-links={wheels_dir}",
"MapProxy"
]
[[lift.bindings]]
name = "venv"
exe = "#{cpython:python}"
args = ["-mvenv", "{scie.bindings}/venv"]
# Build the scie:
:; tools.venv/Scripts/insta-science lift --file wheels_dir=wheels/ build lift.toml
warning: Failed to gather git state for provenance.
Downloading https://github.com/a-scie/jump/releases/latest/download/scie-jump-windows-aarch64.exe ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.58M/1.58M [00:00<00:00, 7.84MB/s]
Downloading https://github.com/astral-sh/python-build-standalone/releases/download/20250106/cpython-3.12.8%2B20250106-x86_64-pc-windows-msvc-install_only.tar.gz ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39.9M/39.9M [00:10<00:00, 4.03MB/s]
C:\Users\jsirois\mapproxy-util.exe
# A native Windows single file executable:
# N.B.: In this case the scie-jump head of the exe is naive arm. The embedded PBS Python distribution is x86_64 though and runs under PRISM.
:; file C:/Users/jsirois/mapproxy-util.exe
C:/Users/jsirois/mapproxy-util.exe: PE32+ executable (console) Aarch64, for MS Windows, 5 sections
# With wheel house zip trailer again:
:; unzip -l C:/Users/jsirois/mapproxy-util.exe
Archive: C:/Users/jsirois/mapproxy-util.exe
warning [C:/Users/jsirois/mapproxy-util.exe]: 43531182 extra bytes at beginning or within zipfile
(attempting to process anyway)
Length Date Time Name
--------- ---------- ----- ----
63397 1980-01-01 00:00 attrs-24.3.0-py3-none-any.whl
164927 1980-01-01 00:00 certifi-2024.12.14-py3-none-any.whl
491326 1980-01-01 00:00 future-1.0.0-py3-none-any.whl
88462 1980-01-01 00:00 jsonschema-4.23.0-py3-none-any.whl
18459 1980-01-01 00:00 jsonschema_specifications-2024.10.1-py3-none-any.whl
1737428 1980-01-01 00:00 MapProxy-3.1.3-py3-none-any.whl
15601 1980-01-01 00:00 MarkupSafe-3.0.2-cp312-cp312-win_amd64.whl
2626369 1980-01-01 00:00 pillow-11.1.0-cp312-cp312-win_amd64.whl
6224720 1980-01-01 00:00 pyproj-3.7.0-cp312-cp312-win_amd64.whl
156338 1980-01-01 00:00 PyYAML-6.0.2-cp312-cp312-win_amd64.whl
26684 1980-01-01 00:00 referencing-0.35.1-py3-none-any.whl
235786 1980-01-01 00:00 rpds_py-0.22.3-cp312-cp312-win_amd64.whl
224498 1980-01-01 00:00 werkzeug-3.1.3-py3-none-any.whl
--------- -------
12073995 13 files
# And it works the same as the Linux example - 1st run install:
:; C:/Users/jsirois/mapproxy-util.exe --help
Looking in links: c:\Users\jsirois\AppData\Local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\mapproxy-3.1.3-py3-none-any.whl
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\pillow-11.1.0-cp312-cp312-win_amd64.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\pyyaml-6.0.2-cp312-cp312-win_amd64.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\future-1.0.0-py3-none-any.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\jsonschema-4.23.0-py3-none-any.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\pyproj-3.7.0-cp312-cp312-win_amd64.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\werkzeug-3.1.3-py3-none-any.whl (from MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\attrs-24.3.0-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\jsonschema_specifications-2024.10.1-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\referencing-0.35.1-py3-none-any.whl (from jsonschema>=4->MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\rpds_py-0.22.3-cp312-cp312-win_amd64.whl (from jsonschema>=4->MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\certifi-2024.12.14-py3-none-any.whl (from pyproj>=2->MapProxy)
Processing c:\users\jsirois\appdata\local\nce\2edbcda49b7a0df4177cf1cbd529b3c15577d71a0ff054b41d23ff1e7e03702a\wheels_dir\markupsafe-3.0.2-cp312-cp312-win_amd64.whl (from werkzeug<4->MapProxy)
Installing collected packages: rpds-py, PyYAML, Pillow, MarkupSafe, future, certifi, attrs, werkzeug, referencing, pyproj, jsonschema-specifications, jsonschema, MapProxy
Successfully installed MapProxy-3.1.3 MarkupSafe-3.0.2 Pillow-11.1.0 PyYAML-6.0.2 attrs-24.3.0 certifi-2024.12.14 future-1.0.0 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 pyproj-3.7.0 referencing-0.35.1 rpds-py-0.22.3 werkzeug-3.1.3
Usage: mapproxy-util COMMAND [options]
Commands:
serve-develop Run MapProxy development server.
serve-multiapp-develop Run MultiMapProxy development server.
create Create example configurations.
scales Convert between scales and resolutions.
wms-capabilities Display WMS capabilites.
grids Display detailed informations for configured grids.
export Export existing caches.
autoconfig Create config from WMS capabilities or a Geopackage file.
defrag-compact-cache De-fragmentate compact caches.
Error:
# 2nd run zippy:
:; C:/Users/jsirois/mapproxy-util.exe --help
Usage: mapproxy-util COMMAND [options]
Commands:
serve-develop Run MapProxy development server.
serve-multiapp-develop Run MultiMapProxy development server.
create Create example configurations.
scales Convert between scales and resolutions.
wms-capabilities Display WMS capabilites.
grids Display detailed informations for configured grids.
export Export existing caches.
autoconfig Create config from WMS capabilities or a Geopackage file.
defrag-compact-cache De-fragmentate compact caches.
Error:
|
What's the problem this feature will solve?
Use case (build portable python packages to be usable on other machines with no internet access):
There are exes e.g. mapproxy-util.exe pip installs to Scripts which have the original full path to python.exe hardcoded in the exe binary. This allows them to only work from that path and prevent ability to make portable.
Describe the solution you'd like
Some solution to make the exes in Scripts portable with pip install. Perhaps a flag to create exes with relative paths to python.exe.
Alternative Solutions
The work around I implemented was to extract the script the exe runs (used a hex editor to view) and used as a python.exe script file argument. This is problematic though, for example in the case of MapProxy's mapproxy-util.exe, the code automatically appends ".exe" to the file path e.g. https://github.com/mapproxy/mapproxy/blob/master/mapproxy/util/ext/serving.py#L386C12-L386C20. As a solution I named the python script mapproxy-util.py.exe to "trick" it. I also deleted all the exe files pip installed in Scripts since they are not usable outside of the original directory.
Additional context
I asked chat GPT how this can be addressed and it recommended a few things I have not tried such as use pyinstaller or shiv. I haven't tried these yet so am not sure if they work as an alternative way to create a portable app package. Ideally I'd like to just address the root problem of having hard coded paths in the exe files in Scripts.
Code of Conduct
The text was updated successfully, but these errors were encountered: