Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] 74.0.0 broke x64 to ARM64 cross-compilation for pywin32 #4648

Closed
Avasam opened this issue Sep 14, 2024 · 5 comments · Fixed by pypa/distutils#298
Closed

[BUG] 74.0.0 broke x64 to ARM64 cross-compilation for pywin32 #4648

Avasam opened this issue Sep 14, 2024 · 5 comments · Fixed by pypa/distutils#298
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.

Comments

@Avasam
Copy link
Contributor

Avasam commented Sep 14, 2024

setuptools version

setuptools>=74.0.0

Python version

Any (tested 3.10 to 3.13)

OS

Windows 10

Additional environment information

pywin32 is cross-compiling ARM64 builds from x64 machines because GitHub actions environments don't offer Windows-on-ARM machines on the public free tier.

Description

pywin32's ARM64 builds start failing as of setuptools==74.0.0 with the following error:

build\temp.win-arm64-cpython-310\Release\win32\src\PyACL.obj : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'ARM64'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1112

Here's the full log from a GitHub action: https://github.com/mhammond/pywin32/actions/runs/10853733760/job/30122635996?pr=2370

Expected behavior

A working build as before, or guidance to update the setup.py file.

I see a lot of related change in https://setuptools.pypa.io/en/latest/history.html#v74-0-0 but if this is intentional breakage, the migration path is not immediately clear to me.

How to Reproduce

I can try to provide a minimal reproduction if needed, but pywin32's setup.py is quite complex, there's still parts I don't fully understand, and it's still using some deprecated distutils API (some of which I'm working on upgrading as per #4070, mhammond/pywin32#2119 and mhammond/pywin32#2208)

Note that the following steps should be run on an x86 Windows machine on a version of Python listed here: https://www.nuget.org/packages/pythonarm64#versions-body-tab

  1. Update setuptools to 74.0.0 or above
  2. Clone https://github.com/mhammond/pywin32 and navigate to the root of the project
  3. In the "Individual Components" tab in "Visual Studio Installer", make sure you have all of the following components installed for your VS version (2019 or 2022). This ensures you don't see a cross-compilation issue with setuptools==73.0.1:
    • VS2019:
      • MSVC v142 - VS 2019 C++ ARM64 build tools
      • C++ MFC for latest v142 build tools (ARM64)
      • C++ ATL for latest v142 build tools (ARM64)
    • VS2022:
      • MSVC v143 - VS 2022 C++ ARM64/ARM64EC build tools (Latest)
      • C++ MFC for latest v143 build tools (ARM64/ARM64EC)
      • C++ ATL for latest v143 build tools (ARM64/ARM64EC)
  4. Run python setup.py -q build_ext --plat-name win-arm64 build --plat-name win-arm64 bdist_wheel --plat-name win-arm64
  5. Wait for a few seconds and see the error

Output

This is my local log using Python3.11 and setuptools==74

> python3.11 setup.py -q build_ext --plat-name win-arm64 build --plat-name win-arm64 bdist_wheel --plat-name win-arm64
Building pywin32 3.11.306.0
-- distutils hack to expose all include & lib dirs
-- orig compiler.include_dirs: ['c:\\python311\\include', 'c:\\python311\\Include']
-- orig compiler.library_dirs: ['c:\\python311\\libs', 'c:\\python311', 'c:\\python311\\PCbuild\\arm64', 'build\\temp.win-arm64-cpython-311\\Release']
PyACL.cpp
PyDEVMODE.cpp
PyHANDLE.cpp
PyIID.cpp
PyLARGE_INTEGER.cpp
PyOVERLAPPED.cpp
PySECURITY_ATTRIBUTES.cpp
PySECURITY_DESCRIPTOR.cpp
PySID.cpp
PyTime.cpp
PyUnicode.cpp
PyWAVEFORMATEX.cpp
PyWinTypesmodule.cpp
LINK : warning LNK4075: ignoring '/MANIFESTUAC' due to '/MANIFEST:NO' specification
build\temp.win-arm64-cpython-311\Release\win32\src\PyACL.obj : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'ARM64'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.41.34120\\bin\\HostX86\\x64\\link.exe' failed with exit code 1112

This is my local log using Python3.11 and setuptools==73.0.1 (different error, which is expected since I'm missing some externals on my machine)

> python3.11 setup.py -q build_ext --plat-name win-arm64 build --plat-name win-arm64 bdist_wheel --plat-name win-arm64
Building pywin32 3.11.306.0
-- distutils hack to expose all include & lib dirs
-- orig compiler.include_dirs: ['c:\\python311\\include', 'c:\\python311\\Include']
-- orig compiler.library_dirs: ['c:\\python311\\libs', 'c:\\python311', 'c:\\python311\\PCbuild\\arm64', 'build\\temp.win-arm64-cpython-311\\Release']
PyACL.cpp
PyDEVMODE.cpp
PyHANDLE.cpp
PyIID.cpp
PyLARGE_INTEGER.cpp
PyOVERLAPPED.cpp
PySECURITY_ATTRIBUTES.cpp
PySECURITY_DESCRIPTOR.cpp
PySID.cpp
PyTime.cpp
PyUnicode.cpp
PyWAVEFORMATEX.cpp
PyWinTypesmodule.cpp
LINK : warning LNK4075: ignoring '/MANIFESTUAC' due to '/MANIFEST:NO' specification
   Creating library build\temp.win-arm64-cpython-311\Release\win32/src\pywintypes311.lib and object build\temp.win-arm64-cpython-311\Release\win32/src\pywintypes311.exp
PyACL.obj : error LNK2001: unresolved external symbol __imp_PySequence_Size
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyMapping_Items
PyACL.obj : error LNK2001: unresolved external symbol __imp_PySequence_GetItem
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyObject_GenericGetAttr
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyLong_FromLong
PyACL.obj : error LNK2001: unresolved external symbol __imp_Py_BuildValue
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyExc_MemoryError
PyACL.obj : error LNK2001: unresolved external symbol __imp__Py_NoneStruct
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyTuple_New
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyErr_SetString
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyExc_ValueError
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyErr_Format
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyExc_OverflowError
PyACL.obj : error LNK2001: unresolved external symbol __imp__Py_Dealloc
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyTuple_GetItem
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyObject_GetAttrString
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyErr_Clear
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyMapping_Check
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyDict_SetItem
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyDict_New
PyACL.obj : error LNK2001: unresolved external symbol __imp_PySequence_Check
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyTuple_SetItem
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyExc_NotImplementedError
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyArg_ParseTupleAndKeywords
PyACL.obj : error LNK2001: unresolved external symbol __imp__Py_NewReference
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyExc_TypeError
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyBuffer_Release
PyACL.obj : error LNK2001: unresolved external symbol __imp_PyArg_ParseTuple
PyDEVMODE.obj : error LNK2001: unresolved external symbol __imp_PyBytes_AsStringAndSize
PyDEVMODE.obj : error LNK2001: unresolved external symbol __imp_PyExc_WindowsError
PyDEVMODE.obj : error LNK2001: unresolved external symbol __imp_PyExc_AttributeError
PyDEVMODE.obj : error LNK2001: unresolved external symbol __imp_PyBytes_FromStringAndSize
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyEval_SaveThread
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp__Py_FalseStruct
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyErr_Restore
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyErr_Fetch
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyObject_GenericSetAttr
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp__Py_HashPointer
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp__Py_NotImplementedStruct
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp__Py_TrueStruct
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyObject_Print
PyHANDLE.obj : error LNK2001: unresolved external symbol __imp_PyEval_RestoreThread
PyIID.obj : error LNK2001: unresolved external symbol __imp_PyBuffer_FillInfo
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyErr_WarnEx
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyErr_Occurred
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyLong_FromUnsignedLongLong
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyLong_AsUnsignedLongLong
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyLong_AsLongLong
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyLong_FromLongLong
PyLARGE_INTEGER.obj : error LNK2001: unresolved external symbol __imp_PyExc_PendingDeprecationWarning
PyOVERLAPPED.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_AsUTF8
PySID.obj : error LNK2001: unresolved external symbol __imp_PyBool_FromLong
PySID.obj : error LNK2001: unresolved external symbol __imp__Py_BuildValue_SizeT
PySID.obj : error LNK2001: unresolved external symbol __imp_PyBytes_FromString
PySID.obj : error LNK2001: unresolved external symbol __imp__PyArg_ParseTuple_SizeT
PySID.obj : error LNK2001: unresolved external symbol __imp_PyErr_NoMemory
PySID.obj : error LNK2001: unresolved external symbol __imp_PySequence_Tuple
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyImport_ImportModule
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyDict_SetItemString
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyType_IsSubtype
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyLong_AsLong
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyErr_BadArgument
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyType_Ready
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyTuple_Size
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyObject_CallMethod
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyCapsule_Import
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyObject_CallObject
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyObject_HasAttrString
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyNumber_Check
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyNumber_Long
PyTime.obj : error LNK2001: unresolved external symbol __imp_PyObject_Call
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyBytes_AsString
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_DecodeMBCS
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_FromWideChar
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_AsWideCharString
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyList_New
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyList_Append
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyBytes_Size
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyMem_Free
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyMem_Malloc
PyUnicode.obj : error LNK2001: unresolved external symbol __imp_PyUnicode_AsMBCSString
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyEval_ReleaseThread
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_Py_MakePendingCalls
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyExc_Exception
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyErr_SetObject
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyModule_AddIntConstant
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyRun_StringFlags
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyMemoryView_FromObject
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyLong_FromUnsignedLong
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyThreadState_New
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyLong_AsUnsignedLongMask
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyModule_GetDict
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyObject_GetBuffer
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyModule_Create2
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyByteArray_FromStringAndSize
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyDict_GetItemString
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyLong_AsVoidPtr
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyThreadState_Clear
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyCallable_Check
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyThreadState_Swap
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp__Py_FatalErrorFunc
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyEval_AcquireThread
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyMemoryView_FromBuffer
PyWinTypesmodule.obj : error LNK2001: unresolved external symbol __imp_PyThreadState_Delete
build\lib.win-arm64-cpython-311\pywin32_system32\pywintypes311.dll : fatal error LNK1120: 104 unresolved externals
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.41.34120\\bin\\HostX86\\ARM64\\link.exe' failed with exit code 1120
@Avasam Avasam added bug Needs Triage Issues that need to be evaluated for severity and status. labels Sep 14, 2024
@Avasam
Copy link
Contributor Author

Avasam commented Sep 14, 2024

There's something about the setuptools history and the way it's merged with distutils that makes it impossible to bissect. The first failing and installable commit I could find is 98ad794
Edit: Installing through python3.11 -m pip install setuptools@git+https://github.com/pypa/setuptools@... seems unhelpful. I checked out locally with local pip installs instead and found that c1a8d9d was the first installable commit to fail.
Edit 2: After a bit of guessing, I was able to identify this commit as a probable cause: dde3ad3
Running git revert dde3ad3dfa5c04e1bf85c907a42b2efc0cb088f3 then accepting incoming changes in my local setuptools repo (starting from the main branch), does resolve the cross-compilation error mentioned above.

In distutils repo, this links to this PR commit: pypa/distutils@dde3ad3 (#285)

@saschanaz
Copy link
Contributor

Commented: pypa/distutils#285 (comment)

@jaraco
Copy link
Member

jaraco commented Sep 15, 2024

The issue has been merged into distutils, but there's a bunch of stuff on main that's backward-incompatible, so I'm going to apply a selective bugfix first before merging the rest of distutils.

@jaraco jaraco reopened this Sep 15, 2024
@jaraco jaraco closed this as completed in 809a9a7 Sep 15, 2024
@jaraco
Copy link
Member

jaraco commented Sep 15, 2024

Oh, geez. I see there's another removal there. I'll want to cherry-pick this change to 74.1.2.

jaraco pushed a commit that referenced this issue Sep 15, 2024
…piler`.

Actually use the `plat_name` param in `MSVCCompiler.initialize`.

Selective cherry-pick of pypa/distutils@a0339c1.

Closes #4648.
@jaraco
Copy link
Member

jaraco commented Sep 15, 2024

Releasing as v74.1.3. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Needs Triage Issues that need to be evaluated for severity and status.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants