python: remove LD_LIBRARY_PATH hack from running python environment
#1562
Conversation
Flake lock file updates:
• Updated input 'devenv':
'github:vlaci/devenv/5186fecbe2835ac45018ea4940388f8523bf1624' (2024-10-04)
→ 'github:vlaci/devenv/953a526754c5ff5e64ab6925c4388b26ae2c45c6' (2024-10-30)
From the related PR[^1] in `devenv`:
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great
to bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via `subprocess`.
To work this around, `devenv` will inject a `pth`[^2] file to the virtual
environment it creates, which mangles the `LD_LIBRARY_PATH` variable,
undoing any changes to it made by `devenv` but preserving changes from
other sources.
[^1]: cachix/devenv#1562
[^2]: https://docs.python.org/3/library/site.html
Flake lock file updates:
• Updated input 'devenv':
'github:vlaci/devenv/5186fecbe2835ac45018ea4940388f8523bf1624' (2024-10-04)
→ 'github:vlaci/devenv/385d38cfb6872e7f95cae30d94f7fc4afd23fd76' (2024-10-30)
From the related PR[^1] in `devenv`:
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great
to bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via `subprocess`.
To work this around, `devenv` will inject a `pth`[^2] file to the virtual
environment it creates, which mangles the `LD_LIBRARY_PATH` variable,
undoing any changes to it made by `devenv` but preserving changes from
other sources.
[^1]: cachix/devenv#1562
[^2]: https://docs.python.org/3/library/site.html
src/modules/languages/python.nix
Outdated
| pth_file = | ||
| let | ||
| exec_content = '' | ||
| ld_library_path_var = "DYLD_LIBRARY_PATH" if sys.platform == "darwin" else "LD_LIBRARY_PATH" |
There was a problem hiding this comment.
I fairly sure DYLD_LIBRARY_PATH doesn't propagate to subprocesses. So it is probably not needed to support it here.
Maybe the pth file only needs to be added on Linux machines?
There was a problem hiding this comment.
I have no idea about it. I am okay to remove it
| os.environ[ld_library_path_var] = ld_library_path.removeprefix(ld_library_path_prefix + ":") | ||
| ''; | ||
| in | ||
| pkgs.writeText "devenv.pth" ''import os; exec("""${builtins.replaceStrings [ "\n" ] [ "\\n" ] exec_content}""")''; |
There was a problem hiding this comment.
Does this need exec? I thought it might be possible to be executed in pth as-is.
There was a problem hiding this comment.
Pth is executed line by line, so this is the only way to use multi-line constructs like if conditions
|
I was thinking, since this will probably be needed just for Linux, that it might be nice to eventually have an LD_PRELOAD that removed the path from LD_LIBRARY_PATH. That way this exact same trick can be applied to ruby, nodejs and more. Still useful to have this pth to see how it fares in the python world. |
|
Can we get some tests for this? |
Injecting `LD_LIBRARY_PATH` to the Python runtime environment is great to bypass the need of having to patch non-nix binaries loaded into that environment, however it breaks down, when Python executes any other program not compiled for the given Nix system, e.g. a shell script via `subprocess`. To work this around, `devenv` will inject a `pth`[^1] file to the virtual environment it creates, which mangles the `LD_LIBRARY_PATH` variable, undoing any changes to it made by `devenv` but preserving changes from other sources. Fixes cachix#1111 [^1]: https://docs.python.org/3/library/site.html
|
I've been testing this change in a few projects, and I have mixed results with it:
I am also testing a completely different approach, by using It has its issues, like It may also be implemented as an add-on, so python packages providing binaries, like |
|
Hi, I think this PR relates to #1699. I was not aware that What I noticed is that when using |
Injecting
LD_LIBRARY_PATHto the Python runtime environment is greatto bypass the need of having to patch non-nix binaries loaded into
that environment, however it breaks down, when Python executes any
other program not compiled for the given Nix system, e.g. a shell
script via
subprocess.To work this around,
devenvwill inject apth1 file to the virtualenvironment it creates, which mangles the
LD_LIBRARY_PATHvariable,undoing any changes to it made by
devenvbut preserving changes fromother sources.
I am unsure what the best way would be to integrate this, bit I think the approach in general is sane. I am tested only the poetry case currently.
Fixes #1111
Footnotes
https://docs.python.org/3/library/site.html ↩