Why does Pip documentation refer Windows to use an executable named "py"?

[notatallshaw]

Description

I'm trying to follow user guides on getting pip development done and one of the things I notice is that it refers to executing python through an executable called py.

I run Python almost exclusively on Windows and I don't know what py is, I can't google it because "py" or "python py" or "py executable" are all too generic for Google so I can't find any documentation on it, instead Google thinks I'm talking about the ".py" extension.

I think it's something that ships optionally with the Python installer from the Python Foundation? But if so it's optional as I've never had it installed. But even so I don't think it ships with the Python from the Windows store, or the Python in Anaconda, Miniconda, Miniforge, Mambaforge, etc...

So why is pip referring to py which is this optional thing that's impossible to find documentation on rather than just python which is what works on both Windows and other platforms?

Expected behavior

python

pip version

all

Python version

all

OS

Windows

How to Reproduce

Read Documentation

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

py is the Python launcher, and it's covered in the documentation.

It's the recommended way of running Python on Windows, not just by pip, but in general, precisely because the python command does not work consistently (it's not installed on PATH by default in the python.org installers, it doesn't behave the same as on Unix, where python runs Python2, and you need python3 to run Python 3, etc).

The pip documentation is written for users of "standard" Python, so what Anaconda does is not really relevant (apart from anything else, if you're using Anaconda you should be using conda, not pip, anyway). The Windows Store Python is an oddball, which doesn't work the "standard" way. I don't know why they chose to work differently, but frankly it's just made things more complicated than they already were 🙁

Basically the answer is "it's all a complicated mess, and nothing works properly across all possible distributions/environments". The most likely thing to work by far on Windows has always been py, so that's what we document.

If someone wants to improve the documentation to explain all of this complexity, then I'd be fine with someone having a go. But you should look at the history fist - this has all been thrashed out many times in the past. Circumstances do change, so that's not saying we won't change the docs, but in my personal opinion, things haven't changed enough yet to make that worthwhile.

[notatallshaw]

py is the Python launcher, and it's covered in the documentation.

Thanks for the docs. I uninstalled all versions of Python on my computer and ran the Python Foundation installer and it does indeed default to "Install Launcher for all Users (recommended)". I guess crappy Windows installers have taught me to untick all options that say they are going to install something that isn't the exact thing I am wanting to install.

It's the recommended way of running Python on Windows, not just by pip, but in general, precisely because the python command does not work consistently (it's not installed on PATH by default in the python.org installers, it doesn't behave the same as on Unix, where python runs Python2, and you need python3 to run Python 3, etc).

Unfortunately py does not work consistently either it seems, with either the Windows Store package or conda based installers 😞

apart from anything else, if you're using Anaconda you should be using conda, not pip, anyway

Conda is multi-functional, it's not just a pip replacement. A perfectly valid use case is to manage the binary dependencies in your enivonrment with conda such as Python, NodeJS, OpenSSL, Rust Compiler, etc. and then use pip to manage your python packages. For a number of reasons this is one of the way I advise teams develop Python applications inside the company I work at.

The Windows Store Python is an oddball, which doesn't work the "standard" way. I don't know why they chose to work differently, but frankly it's just made things more complicated than they already were 🙁

I wonder if we can ask @zooba on this?

Basically the answer is "it's all a complicated mess, and nothing works properly across all possible distributions/environments". The most likely thing to work by far on Windows has always been py, so that's what we document.

If someone wants to improve the documentation to explain all of this complexity, then I'd be fine with someone having a go. But you should look at the history fist - this has all been thrashed out many times in the past. Circumstances do change, so that's not saying we won't change the docs, but in my personal opinion, things haven't changed enough yet to make that worthwhile.

Fair enough, I guess to start off with it might be nice to have the above link in pips docs. First I will try and get pip's development environment working on my machine and build up some experience with py and any other tools.

[zooba]

The Windows Store Python is different because it can't manage the py.exe installed by the traditional installer. So it was going to make things even messier if it tried to install it into a different location that may/may not be overruled by a traditional install, and all it would have the ability to do is to launch the Python that it was installed with.

The best way to launch pip if you've installed from the store is pip3.10 (or whatever your version is). It's not the usual launcher, and so none of the issues associated with a regular pip install ... apply. If you manage the execution aliases carefully, for which there is standard Windows UI (though it's not very good...), you can confidently use pip3 or just pip as well (and python3.x, python3 and python also work).

In most cases it's expected to be an alternative to the traditional install. If you get it from the Store, you wouldn't install it from python.org as well (and I certainly haven't installed from python.org since 3.8, because I've got the Store install around).

So given that it wasn't going to be reliably consistent with the past Windows experience, that the past Windows experience was woefully inconsistent with everyone's POSIX-by-default instructions, and we didn't have PATH concerns, we went with a more POSIX-like design. (I'm sure I've had this conversation before on this tracker, but guess it didn't turn into doc updates...)

[pfmoore]

I'm sure I've had this conversation before on this tracker, but guess it didn't turn into doc updates...

If you've any suggestions for how the docs can cover this without being even more confusing than they already are, PRs would be appreciated 😉 Personally, I find the inconsistencies that Store python introduced sufficiently annoying¹ (in terms of "it was hard enough to explain before, this just made it worse") that I'm unlikely to tackle such a change myself.

¹ I understand that Store apps have constraints that mean they can't work like the python.org installers, and I have no doubt that the Store install is the most sensible design it can be given those constraints. And maybe even better overall, if we ignore compatibility as a constraint. But it still annoys me that Store python has to be an exception to the "classic" behaviour on Windows.

[notatallshaw]

The best way to launch pip if you've installed from the store is pip3.10 (or whatever your version is). It's not the usual launcher, and so none of the issues associated with a regular pip install ... apply. If you manage the execution aliases carefully, for which there is standard Windows UI (though it's not very good...), you can confidently use pip3 or just pip as well (and python3.x, python3 and python also work).

This information is very useful thanks, I will try to propagate it further when I update the Python guides for my company or answer Python questions in Reddit.

I'm sure I've had this conversation before on this tracker, but guess it didn't turn into doc updates...

If you've any suggestions for how the docs can cover this without being even more confusing than they already are, PRs would be appreciated 😉

I can't speak for others but I know I don't have a suggestion right now, I will continue to think on it. But if you want to close this issue feel free to, can always open another or a PR if I feel convinced of a better solution.

[pradyunsg]

Well, given that I've been fiddling with the docs a whole lot lately, I'd be happy to pick up the documentation updates and make the PR necessary for it here; if someone could state more clearly what the changes need to be.

It sounds like we have a more Unix-y experience with the Windows store Python?

[pfmoore]

My main concern is that I suspect that many of the people who get confused over this sort of question may be equally confused if you ask them "are you using Python from the Windows Store or from the official installers?" (To say nothing of Anaconda, Chocolatey, cygwin, mingw64, or the various other places they could have got Python from). So before we branch off into whether to use py or python, we need to answer that question, otherwise we'll not actually be helping much...

[pradyunsg]

All of this is making me think that we should add a paragraph/section/page, talking about "how to run pip" somewhere; to discuss these nuances -- and use exactly one format in the rest of pip's documentation.

[uranusjr]

This is not really a pip issue, but I'm wondering if Windows Store apps allow for requesting dependencies. If it's somewhat possible, I'd rather much prefer CPython to officially distribute py in the Store that gets installed when any Python version is installed. The py command made calling Python on Windows much easier for newcomers, and it's very disappointing that CPython is now effectively breaking their own good idea and actively confusing everyone.

[pfmoore]

To be honest, I think that it would be better if we could simply link to a central document that says "how to run Python" and then say for pip you run python -m pip where people should read "python" as meaning "whatever incantation you need to run Python (see the linked document)". Unfortunately I don't think there is such a document.

The lack of clear, beginner-friendly instructions on how to run Python (in a way that allows you to add command line arguments like -m pip) is not pip's issue, but it does cause problems for us (and other projects).

[pradyunsg]

Yea, I'm saying we should write up such a document, put it in our documentation and update all our invocations to be just plain pip/python -m pip.

This conversation is making me wonder if quirks around pip being different from python -m pip would be something that we could improve as well (especially once we explicitly write down the quirks). Who knows, maybe the python -m pip vs pip invocation mismatch is a problem that we could come up with a reasonable solution for?

[pradyunsg]

While I'm spiralling, the thought I have right now is adding a check + warning to the pip entrypoint, such that it warns when python -m pip / py -m pip has a mismatch with it (adding a flag to force it to run anyway) and then having pip straight up fail on those mismatches in a future release.

This will need quirks, like path-bassd invocation in a virtualenv and stuff; but this seems surprisingly tractable. BUT combined with PEP 668 preventing users from breaking their system, I think we'd have everything needed to using plain pip being not-a-footgun?

[pfmoore]

quirks around pip being different from python -m pip

As far as I know, there are only two main differences:

1. On Windows, when upgrading pip the OS prevents you from replacing the pip.exe wrapper, so pip install -U pip gives an error where python -m pip doesn't.
2. People manage to get confused as to which Python interpreter pip is managing. That confusion isn't possible if you use python -m pip.

The first issue can't really be fixed. The best options we have (all of which have been discussed and discarded in the past) are:

1. Rename pip.exe before replacing and "delete it later". The problem here is that it's hard to reliably delete it later, and leaving junk executables around is nasty. We can't even reliably move the old exe off PATH as we might be moving cross filesystem in that case. We could (and should) rename the old exe to have a different extension, for some safety.
2. Ignore the error because pip.exe very rarely changes in practice. We could even check if the new one is the same as the old and skip replacing if it is. That would probably eliminate a lot of the problem cases. There may be problems doing such a check, as I can't recall why we've never tried this, but if it is possible, I think this would be a worthwhile thing to try.

The second difference (user confusion) is not something we can really do much about. People can do pip --version to check. But they don't, and they do get in a mess, so this is a genuine problem. But conversely, telling people "don't do that, it's bad for you" is a futile exercise in general...

My personal thoughts are:

1. Have a document that discusses all of the various possibilities for how to invoke Python, how to invoke pip, etc. Make it as complex as you like and label it as an "in depth guide".
2. Settle on a relatively simple "this is how you invoke Python" for the summary in "how to invoke pip". Probably
   1. In a virtual environment, use python
   2. On Windows, use py unless you're using Store Python, when you use python
   3. On Unix, use python3 (or python if your Unix distro no longer points that at Python 2)
   4. If you are using Python from an independent distributor (Anaconda, Chocolatey, cygwin, ...) do whatever they say you should do
   5. If you know what you're doing, feel free to ignore the above and invoke your Python interpreter however you want
3. Standardise on python -m pip as the method for running pip. Include a note saying that a pip command may well exist, and people can use it, but they should confirm any issues they have still occur if they use python -m pip before reporting a bug (direct them to the in-depth guide if the issue is only happening with the wrapper).

I don't know why @zooba suggests above that pip3.10 is a better way of invoking pip in the store Python than python3.10 -m pip, so I've ignored that. But if there's some subtlety there, and it's for some reason not fixable, I say we add that to the in-depth discussion, and either leave the general guidance unchanged, or just add a one-line note to say "ignore all this for store Python and use pipX.Y - see the in-depth guide for why".

[pradyunsg]

On Windows, when upgrading pip the OS prevents you from replacing the pip.exe wrapper, so pip install -U pip gives an error where python -m pip doesn't.

How about a separate pip-cli package, that provides the executable instead? That package would likely never need to be updated, at least not as often as pip itself. At least, that was the idea in #3164 (comment).

• Standardise on python -m pip as the method for running pip.

Honestly, I want to standardise on pip as the method for running pip, or at least how we document using it everywhere. It's much easier to read. As I said earlier, I think we should have pip enforce that it matches python/py. Same for pipX and pipX.Y <-> pythonX/py -X and pythonX.Y/py -X.Y. That's already the case for virtual environments, and realistically, that's what we want everyone to use anyway.

We should definitely keep python -m pip -- that is unambigous and clear; but removing the footgun-nature of pip vs python -m pip being different, definitely seems like a solvable problem to me.

The second difference (user confusion) is not something we can really do much about.

I disagree. My proposal for this is literally my previous comment and what I said in the last paragraph, so... not gonna repeat that again. :)

Have a document that discusses all of the various possibilities for how to invoke Python, how to invoke pip, etc. Make it as complex as you like and label it as an "in depth guide".

I like how this sounds. :)

[pfmoore]

How about a separate pip-cli package

Oh yes! I'd forgotten that idea. That would be a really good solution IMO.

Honestly, I want to standardise on pip as the method for running pip

It's certainly more natural for people. And if the pip command comes from a pip-cli package, so there's no self-upgrade issue, that makes it much better.

Transition will be a bit of a PITA, though. I predict many cases of people having old wrappers lying around that result in very confusing issues. And getting distros to replace their existing wrappers will be a challenge, I suspect (although maybe we don't need to do that, as Unix doesn't have the self-upgrade issue). We'll also need to get it into ensurepip.

And I still think the user confusion issue is relevant:

As I said earlier, I think we should have pip enforce that it matches python/py

Ah, I think I follow your reasoning now, you're saying that if the pip command validates that it's using the same interpreter as python -m pip would do, then there's no confusion. And yes, I agree that would be plausible. But doesn't that just leave you with a different problem? How do you know what python command the user wanted? Given that we can't even decide how to describe the "python" part of python -m pip in our documentation, what chance have we got of determining programmatically what the user would have written if they had used python -m pip instead of pip?

Ultimately, the problem here isn't at the pip level at all, it's that there is no canonical way of invoking the Python interpreter. I don't honestly know how we ended up in such a mess over this - the canonical way to invoke git is via the git command, you invoke the Julia language via julia. etc., etc. But between the python/python3 mess on Unix, the fact that the Windows installers didn't add Python to PATH and then the use of the launcher to get around that, the fact that Windows then hijacked the python command to open the Store so people could download Python, and probably a few other things, simply running the Python interpreter is way harder for users than it should be.

I guess it's fairly clear to me that if we can get the pip wrapper to check and say "hey, I'm not managing the same Python installation that python -m pip would, have you got something wrong?" then that would address the "user confusion" argument against making pip the caconical way to invoke pip. But I'm not convinced that's achievable in practice (I'm not even sure it's possible to define the behaviour we're talking about precisely while still being useful).

So in summary, I'm +1 on a pip-cli package, to address the self-update issue, although publicity and management for the change won't be easy. However, I still prefer making python -m pip the canonical "way to run pip" unless someone can demonstrate how a "do I do the same as python -m pip" check would actually work in real life.

[zooba]

The pip-cli idea sounds good, and I don't see why it couldn't execute python -m pip in the background so that no matter where you pick up the CLI command from, it's going to launch with whatever python would run. (It could even "inject" pip onto the path of that python to handle environments where it hasn't been installed, though now we're back to not being able to delete pip.exe.)

FWIW, it is possible through some obscure API calls to make pip.exe deleteable even if it's running. But it depends on some OS updates, and I don't know off the top of my head when they shipped, but you can't rely solely on that yet. Personally I'm not as offended as Paul is by leaving the file behind after renaming it - you could even print a message saying "please run del <path to file> to finish cleanup".

For the "am I using the global python" check, just include a script file that prints sys.executable and compare it to the one you're running in. Takes an extra process launch, but any pip command for most people is already taking so much longer than a process launch it won't matter.

Ultimately, the problem here isn't at the pip level at all, it's that there is no canonical way of invoking the Python interpreter. I don't honestly know how we ended up in such a mess over this

I do. It's because we support side-by-side versions, rather than one incremental version that just updates and is basically backwards compatible, along with separate LTS releases that don't overlap like we have today (which you may recognise as being my release schedule PEP that was rejected in favour of the current one).

From the Windows POV, I also looked into renaming py.exe to python.exe so that we could just have the one consistent entry point (think python-cli package, like the potential pip one ;) ), but ultimately the amount of churn is too big for me to do unilaterally. Perhaps for Python 4.x we can revamp the approach to releases drastically enough to make it a single, reliable command and easier for packages to handle forwards and backwards compatibility within those releases.

[notatallshaw]

For the "am I using the global python" check, just include a script file that prints sys.executable

But don't actually use sys.executable as it points to the wrong path in a virtual environment subprocess on Windows: https://bugs.python.org/issue38905

As I've recently discovered trying to run pip's test cases: #10443

Edit: See correction below

[zooba]

But don't actually use sys.executable as it points to the wrong path in a virtual environment subprocess on Windows

Read the rest of that issue, you've summarised it incorrectly.

Launching python on Windows (as opposed to a full path) is going to resolve to a python.exe adjacent to the current process first, because that's the behaviour. You'd have to manually resolve python as part of this, and make the decision about whether to emulate CMD's legacy behaviour (look in CWD first) or Powershell's behaviour (ignore CWD).

[notatallshaw]

But don't actually use sys.executable as it points to the wrong path in a virtual environment subprocess on Windows

Read the rest of that issue, you've summarised it incorrectly.

Launching python on Windows (as opposed to a full path) is going to resolve to a python.exe adjacent to the current process first, because that's the behaviour. You'd have to manually resolve python as part of this, and make the decision about whether to emulate CMD's legacy behaviour (look in CWD first) or Powershell's behaviour (ignore CWD).

Thanks for the info, I did some testing and I see I am mistaken. I don't understand what adjacent means in this context but I will do some more reading.

Edit: I see my confusion now, when I was testing the only version of Python on the PATH was the virtual environment (I have even disabled the Windows Store stubs), so I mistakenly assumed that subprocess could only be finding that version of Python. Reading through the issue now I see there is an undocumented(?) secret/hidden/pseudo env variable "__APPDIR__" that the API for creating a new process on Windows checks before checking the PATH.

[zooba]

Reading through the issue now I see there is an undocumented(?) secret/hidden/pseudo env variable "__APPDIR__" that the API for creating a new process on Windows checks before checking the PATH.

It's documented, but not an env variable. It's just part of how CreateProcess (and LoadLibrary, for that matter) resolves relative paths. The process image filename is almost certainly going to be read from internal data structures that are easily accessible to the API implementations, so the only way to skip it is to calculate the absolute path.