Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-202: Simplify Windows installation #203

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jkloth
Copy link

@jkloth jkloth commented May 17, 2022

PR for issue #202

Comment on lines -26 to -46
Windows notes
-------------

On Windows, to allow pyperformance to build dependencies from source
like ``greenlet``, ``dulwich`` or ``psutil``, if you want to use a
``python.exe`` built from source, you should not use the ``python.exe``
directly. Instead, you must run the little-known command ``PC\layout``
to create a filesystem layout that resembles an installed Python::

.\python.bat -m PC.layout --preset-default --copy installed -v

(Use the ``--help`` flag for more info about ``PC\layout``.)

Now you can use the "installed" Python executable::

installed\python.exe -m pip install pyperformance
installed\python.exe -m pyperformance run ...

Using an *actually* installed Python executable (e.g. via ``py``)
works fine too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this should remain for the sake of Python versions that don't have the fix. So a note should be added identifying the last Python version to which this applies.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going with the assumption that using a source build on Windows is such a rare occurrence that this would be in the realm of CPython developers already operating out of a GH clone. This would imply an up to date (-ish) checkout, therefore containing the bug fix.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense. The catch is that we run benchmarks against older CPython versions too.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do note, though, the removed verbiage only applies to source builds. Other means of running Python (nuget, winget, installer) work as is. I would say workflows that clone a released Python branch and build it to run pyperformance would be better served using the released binary for the comparison.

For example:

nuget.exe install python -Version 3.10 [-OutputDirectory <dir>]

will download and unpack the released distribution for 3.10.0 to <dir>\python as a full usable interpreter (<dir>\python\tools\python.exe). This already is done in the build scripts just to build the interpreter now[1].
nuget.exe is also available in the <srcdir>\externals directory (see [1]). This has the added benefit of guaranteeing isolation from previous builds.

[1] Unless there is an existing Python installation usable for building.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll admit, however, I have no idea how any current workflows using pyperformance on Windows are implemented. Although the current guidance seems to be quite heavy already.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very few have managed to successfully run PyPerformance on Windows, because it has been so broken for so long (and the test suite is still broken -- as soon as you start it, the process detaches and runs in the background, which I didn't even know was possible in DOS Windows :-).

I have a reason for building past versions from source though: You never know which aspects of your current setup (compiler version, Windows version, hardware, version of other tooling installed) might affect the performance of the binary. It wouldn't be the first time that someone reported a significant but mysterious/irreproducible speedup or slowdown between two releases only to find out that they were comparing binaries built with different compiler versions. If the goal is to measure the effect of the source code changes rather than simply the progress in compiler technology, it's important to try to keep as many variables unchanged as possible.

# Recent setuptools versions have a bug that prevents being
# used to compile from a source build virtual environment.
# [BUG](https://github.com/pypa/setuptools/issues/3325)
# The stdlib distutils does not have this problem.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when the stdlib distutils goes away?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably the bug would have to be fixed in setuptools?

@ericsnowcurrently
Copy link
Member

@gvanrossum, what do you think?

@gvanrossum gvanrossum changed the title gh-202: Simply Windows installation gh-202: Simplify Windows installation May 24, 2022
@gvanrossum
Copy link
Member

Removing that docs section seems premature, the root cause hasn't even been fixed (seems the OP is on a quest here :-).

I'm okay with adding the env var to work around the setuptools problem, we can just remove it again (and require a high enough setuptools version) once that is fixed in setuptools. I don't think stdlib distutils is going away in 3.12 yet, is it? If it removed before setuptools fixes the bug on their side, we'd have to lean on setuptools.

@jkloth
Copy link
Author

jkloth commented May 25, 2022

I wouldn't say quest but attempting to run pyperformance from a source checkout is what started this whole chain of changes. I just found it unreasonable to need to remember to do the additional setup for testing an in-progress build. So, I leveraged my Windows build system knowledge to hunt down the cause of this issue.

@ericsnowcurrently
Copy link
Member

I don't think stdlib distutils is going away in 3.12 yet, is it?

See python/cpython#92584 (and https://peps.python.org/pep-0632), distutils is going away in 3.12. That has not happened quite yet, but probably will relatively soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants