Enable use of pyinstaller #14

schlomo · 2020-09-23T07:39:23Z

This is a really amazing tool!

Could you enable the use of pyinstaller? The use case is creating a Windows EXE self-contained file from a Python tool and thereby foregoing the need for a Windows build machine.

I got it all to work but then pyinstaller runs into this problem:

Unable to find "\usr\local\share\wenv\win64\drive_c\python-3.8.4\Include\pyconfig.h" when adding binary and data files.This would mean your Python installation doesn't
come with proper library files. This usually happens by missing development
package, or unsuitable build parameters of Python installation.
* On Debian/Ubuntu, you would need to install Python development packages
  * apt-get install python3-dev
  * apt-get install python-dev
* If you're building Python by yourself, please rebuild your Python with
`--enable-shared` (or, `--enable-framework` on Darwin)

I was able to proceed after copying that file from the Python 3.8.4 sources but then I got stuck with this error:

43979 INFO: Processing pre-safe import module hook win32com from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\pre_safe_import_module\\hook-win32com.py'.
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'win32com'
44361 INFO: Processing pre-safe import module hook win32com from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\pre_safe_import_module\\hook-win32com.py'.
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'win32com'

Any ideas how to get pyinstaller to work in wenv? That would really help me a lot.

The text was updated successfully, but these errors were encountered:

s-m-e · 2020-09-23T09:27:38Z

pyinstaller vs wenv ... and I thought I have crazy ideas ;) Very interesting idea. Please keep me posted.

Getting the CPython interpreter to work on Wine is not trivial at all, unfortunately. The installers from python.org fail most of the time. The Anaconda installer has never worked on Wine. ActivePython sort of did ... until a few years ago. The list goes on. I am working around the problem by using the "Windows x86[-64] embeddable zip file", see here for instance. The embedded build is lacking some features. It certainly lacks header files etc as you have already noticed.

win32com is not part of the standard library. You can make it work by installing pywin32, i.e. wenv pip install pywin32.

If you could share a complete, reproducible example (configuration / list of steps), I could look deeper into it.

schlomo · 2020-09-24T21:26:30Z

https://gist.github.com/schlomo/7f3c65969ca06f384e85accb8c88bff4 is a test case that demonstrates the problem with the help of a little Dockerfile:

+ wenv pyinstaller --onefile /usr/local/bin/yamlreader
291 INFO: PyInstaller: 4.0
291 INFO: Python: 3.8.4
291 INFO: Platform: Windows-7-6.1.7601-SP1
293 INFO: wrote Z:\work\yamlreader.spec
305 INFO: UPX is not available.
312 INFO: Extending PYTHONPATH with paths
['Z:\\usr\\local\\bin', 'Z:\\work']
353 INFO: checking Analysis
358 INFO: Building because inputs changed
358 INFO: Initializing module dependency graph...
375 INFO: Caching module graph hooks...
430 INFO: Analyzing base_library.zip ...
3296 INFO: Processing pre-find module path hook site from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-site.py'.
3299 INFO: site: retargeting to fake-dir 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\fake-modules'
3831 INFO: Processing pre-find module path hook distutils from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
3836 INFO: distutils: retargeting to non-venv dir 'C:\\python-3.8.4\\lib'
7835 INFO: Caching module dependency graph...
7959 INFO: running Analysis Analysis-00.toc
7963 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by c:\python-3.8.4\python.exe
8224 INFO: Analyzing \usr\local\bin\yamlreader
8609 INFO: Processing module hooks...
8610 INFO: Loading module hook 'hook-distutils.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
8615 INFO: Loading module hook 'hook-encodings.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
9082 INFO: Loading module hook 'hook-lib2to3.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
9240 INFO: Loading module hook 'hook-pkg_resources.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
10716 INFO: Processing pre-safe import module hook win32com from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\pre_safe_import_module\\hook-win32com.py'.
10924 WARNING: Hidden import "pkg_resources.py2_warn" not found!
10924 WARNING: Hidden import "pkg_resources.markers" not found!
10926 INFO: Excluding import '__main__'
10927 INFO:   Removing import of __main__ from module pkg_resources
10928 INFO: Loading module hook 'hook-sysconfig.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
10932 INFO: Loading module hook 'hook-xml.dom.domreg.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
10934 INFO: Loading module hook 'hook-xml.py' from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks'...
10937 INFO: Loading module hook 'hook-pythoncom.py' from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
12120 INFO: Loading module hook 'hook-pywintypes.py' from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
13305 INFO: Loading module hook 'hook-win32com.py' from 'C:\\python-3.8.4\\lib\\site-packages\\_pyinstaller_hooks_contrib\\hooks\\stdhooks'...
14250 INFO: Looking for ctypes DLLs
14286 INFO: Analyzing run-time hooks ...
14292 INFO: Including run-time hook 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_multiprocessing.py'
14298 INFO: Including run-time hook 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_pkgres.py'
14308 INFO: Including run-time hook 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\rthooks\\pyi_rth_win32comgenpy.py'
14325 INFO: Looking for dynamic libraries
15157 INFO: Looking for eggs
15157 INFO: Using Python library c:\python-3.8.4\python38.dll
15157 INFO: Found binding redirects:
[]
15164 INFO: Warnings written to Z:\work\build\yamlreader\warn-yamlreader.txt
15262 INFO: Graph cross-reference written to Z:\work\build\yamlreader\xref-yamlreader.html
15300 INFO: checking PYZ
15303 INFO: Building because name changed
15304 INFO: Building PYZ (ZlibArchive) Z:\work\build\yamlreader\PYZ-00.pyz
15813 INFO: Building PYZ (ZlibArchive) Z:\work\build\yamlreader\PYZ-00.pyz completed successfully.
15834 INFO: checking PKG
15836 INFO: Building because name changed
15836 INFO: Building PKG (CArchive) PKG-00.pkg
19056 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
19060 INFO: Bootloader C:\python-3.8.4\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe
19060 INFO: checking EXE
19063 INFO: Rebuilding EXE-00.toc because yamlreader.exe missing
19063 INFO: Building EXE from EXE-00.toc
19064 INFO: Appending archive to EXE Z:\work\dist\yamlreader.exe
19082 INFO: Building EXE from EXE-00.toc completed successfully.
+ : 'Build OK'
+ ls -l dist/yamlreader.exe
-rwxr-xr-x 1 root root 10566303 Sep 24 21:20 dist/yamlreader.exe
+ :
+ : 'Testing Windows binary:'
+ :
+ wine dist/yamlreader.exe --help
it looks like wine32 is missing, you should install it.
multiarch needs to be enabled first.  as root, please
execute "dpkg --add-architecture i386 && apt-get update &&
apt-get install wine32"
wine: created the configuration directory '/root/.wine'
0012:err:ole:marshal_object couldn't get IPSFactory buffer for interface {00000131-0000-0000-c000-000000000046}
0012:err:ole:marshal_object couldn't get IPSFactory buffer for interface {6d5140c1-7436-11ce-8034-00aa006009fa}
0012:err:ole:StdMarshalImpl_MarshalInterface Failed to create ifstub, hres=0x80004002
0012:err:ole:CoMarshalInterface Failed to marshal the interface {6d5140c1-7436-11ce-8034-00aa006009fa}, 80004002
0012:err:ole:get_local_server_stream Failed: 80004002
000b:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
000b:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
0014:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0014:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
0014:err:ole:apartment_createwindowifneeded CreateWindow failed with error 0
0014:err:ole:apartment_createwindowifneeded CreateWindow failed with error 0
0014:err:ole:marshal_object couldn't get IPSFactory buffer for interface {00000131-0000-0000-c000-000000000046}
0014:err:ole:apartment_createwindowifneeded CreateWindow failed with error 14007
0014:err:ole:StdMarshalImpl_MarshalInterface Failed to create ifstub, hres=0x800736b7
0014:err:ole:CoMarshalInterface Failed to marshal the interface {6d5140c1-7436-11ce-8034-00aa006009fa}, 800736b7
0014:err:ole:get_local_server_stream Failed: 800736b7
0010:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0010:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
Could not find Wine Gecko. HTML rendering will be disabled.
it looks like wine32 is missing, you should install it.
multiarch needs to be enabled first.  as root, please
execute "dpkg --add-architecture i386 && apt-get update &&
apt-get install wine32"
0025:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0025:err:winediag:nodrv_CreateWindow Make sure that your X server is running and that $DISPLAY is set correctly.
Could not find Wine Gecko. HTML rendering will be disabled.
wine: configuration in L"/root/.wine" has been updated.
Python path configuration:
  PYTHONHOME = 'C:\users\root\Temp\_MEI82'
  PYTHONPATH = (not set)
  program name = 'Z:\work\dist\yamlreader.exe'
  isolated = 0
  environment = 0
  user site = 0
  import site = 0
  sys._base_executable = 'Z:\\work\\dist\\yamlreader.exe'
  sys.base_prefix = ''
  sys.base_exec_prefix = ''
  sys.executable = 'Z:\\work\\dist\\yamlreader.exe'
  sys.prefix = ''
  sys.exec_prefix = ''
  sys.path = [
    'C:\\users\\root\\Temp\\_MEI82\\base_library.zip',
    'C:\\users\\root\\Temp\\_MEI82',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
Traceback (most recent call last):
  File "<frozen zipimport>", line 259, in load_module
  File "encodings\__init__.py", line 31, in <module>
ModuleNotFoundError: No module named 'codecs'

(showing only the relevant bits of the docker build . run, run yourself for full output)

So what happens is that pybuiilder creates a Windows EXE file, but the file doesn't work.

Running the resulting file on Windows doesn't help either.

schlomo · 2020-09-24T21:47:56Z

Opened pyinstaller/pyinstaller#5198 for cross-reference

s-m-e · 2020-09-25T09:11:55Z

One small mistake that immediately sticks out: You're testing the build with the command wine, which initialized and runs a new 32bit environment. However, you configured wenv for 64bit, i.e. you should really test with wine64.

codecs is part of Python's standard library. It's actually one of the first modules to be loaded when Python launches. A failure to load it usually points to Python being unable to find its entire standard library. Do you get this exact same error with the resulting EXE-file on an actual Windows system?

I just tested the procedure (wine64 / win64). This is how far I get:

(env38) ernst@e9:~/Desktop/test_so> wenv pyinstaller --onefile "c:/python-3.8.4/Scripts/yamlreader.exe"
226 INFO: PyInstaller: 4.1.dev0
226 INFO: Python: 3.8.4
226 INFO: Platform: Windows-7-6.1.7601-SP1
227 INFO: wrote Z:\home\ernst\Desktop\test_so\yamlreader.spec
238 INFO: UPX is not available.
242 INFO: Extending PYTHONPATH with paths
['c:\\python-3.8.4\\Scripts', 'Z:\\home\\ernst\\Desktop\\test_so']
271 INFO: checking Analysis
271 INFO: Building Analysis because Analysis-00.toc is non existent
271 INFO: Initializing module dependency graph...
280 INFO: Caching module graph hooks...
320 INFO: Analyzing base_library.zip ...
2873 INFO: Processing pre-find module path hook site from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-site.py'.
2875 INFO: site: retargeting to fake-dir 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\fake-modules'
3277 INFO: Processing pre-find module path hook distutils from 'C:\\python-3.8.4\\lib\\site-packages\\PyInstaller\\hooks\\pre_find_module_path\\hook-distutils.py'.
3279 INFO: distutils: retargeting to non-venv dir 'C:\\python-3.8.4\\lib'
6795 INFO: Caching module dependency graph...
6905 INFO: running Analysis Analysis-00.toc
6910 INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable
  required by c:\python-3.8.4\python.exe
7358 INFO: Analyzing c:\python-3.8.4\Scripts\yamlreader.exe
Traceback (most recent call last):
  File "runpy.py", line 194, in _run_module_as_main
  File "runpy.py", line 87, in _run_code
  File "C:\python-3.8.4\Scripts\pyinstaller.exe\__main__.py", line 7, in <module>
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\__main__.py", line 114, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\building\build_main.py", line 720, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\building\build_main.py", line 667, in build
    exec(code, spec_namespace)
  File "Z:\home\ernst\Desktop\test_so\yamlreader.spec", line 6, in <module>
    a = Analysis(['c:/python-3.8.4/Scripts/yamlreader.exe'],
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\building\build_main.py", line 242, in __init__
    self.__postinit__()
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\building\datastruct.py", line 160, in __postinit__
    self.assemble()
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\building\build_main.py", line 413, in assemble
    priority_scripts.append(self.graph.run_script(script))
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\depend\analysis.py", line 301, in run_script
    self._top_script_node = super(PyiModuleGraph, self).run_script(
  File "C:\python-3.8.4\lib\site-packages\PyInstaller\lib\modulegraph\modulegraph.py", line 1390, in run_script
    contents = fp.read() + '\n'
  File "codecs.py", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

Looks like an encoding issue. This can probably fixed with a Wine setting if I had to guess. I look deeper into it in next couple of days.

s-m-e · 2020-09-25T09:29:04Z

Ok, just a guess - this appears to work:

wenv pyinstaller --onefile "c:/python-3.8.4/Lib/site-packages/yamlreader/yamlreader.py"

This is the actual entry point of yamlreader. The EXE-file in dist looks ok, though it fails to run on Wine (5.10). I do not have a Windows system right now for testing it there.

s-m-e · 2020-09-25T09:54:36Z

Ok, still can not launch in Wine 5.17. Root cause is a Wine bug. Binary could actually be ok ...

schlomo · 2020-09-25T12:17:07Z

Maybe https://github.com/spesmilo/electrum/blob/aae06116f9cff9e3540c20a8fb6e51011ca9b3ab/contrib/build-wine/prepare-wine.sh is an interesting way how to install a full featured Python from MSI instead of using the embedded distribution?

s-m-e · 2020-09-25T13:54:03Z

wine msiexec ... this appears to work, actually. Interesting.

schlomo · 2020-09-30T08:14:32Z

@s-m-e I checked and found out that wine and wine64 do the same thing as long as I install only the 64-bit version of wine:

Besides that I think that testing the resulting EXE in a clean wine environment is better / more correct than testing it in the build environment as that ensures that the resulting binary works without the build environment.

Wrong thought?

s-m-e · 2020-09-30T09:34:35Z

Besides that I think that testing the resulting EXE in a clean wine environment is better / more correct than testing it in the build environment as that ensures that the resulting binary works without the build environment.

There is nothing wrong with that, sure. However, I am not changing the configuration of the wine environment that I create for wenv. wenv simply temperately sets a few environment variables when it runs. Otherwise, its environment can be considered "clean".

Ok, still can not launch in Wine 5.17. Root cause is a Wine bug. Binary could actually be ok ...

If you want to test your EXE, you may need to apply the patch mentioned in this ticket and build Wine from sources. I attached my Wine-based built of yamlreader. Looks OK but does does not launch due this exact Wine bug. You should probably see the same thing.

@s-m-e I checked and found out that wine and wine64 do the same thing as long as I install only the 64-bit version of wine:

Also interesting. AFAIK this depends on how Wine is packaged, which differs between distributions and releases. On openSUSE for instance, wine is a binary (the 32bit version), not a script with a fall-back and also not a symlink like in your case.

s-m-e added the question Further information is requested label Sep 23, 2020

schlomo mentioned this issue Sep 24, 2020

Allow cross-building Windows EXE on Linux pyinstaller/pyinstaller#5198

Closed

schlomo changed the title ~~Enable use if pyinstaller~~ Enable use of pyinstaller Sep 24, 2020

milahu mentioned this issue Apr 16, 2024

make wenv work with a read-only prefix - fallback to $HOME/.cache/wenv #22

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable use of pyinstaller #14

Enable use of pyinstaller #14

schlomo commented Sep 23, 2020

s-m-e commented Sep 23, 2020

schlomo commented Sep 24, 2020

schlomo commented Sep 24, 2020

s-m-e commented Sep 25, 2020

s-m-e commented Sep 25, 2020

s-m-e commented Sep 25, 2020

schlomo commented Sep 25, 2020

s-m-e commented Sep 25, 2020

schlomo commented Sep 30, 2020

s-m-e commented Sep 30, 2020

Enable use of pyinstaller #14

Enable use of pyinstaller #14

Comments

schlomo commented Sep 23, 2020

s-m-e commented Sep 23, 2020

schlomo commented Sep 24, 2020

schlomo commented Sep 24, 2020

s-m-e commented Sep 25, 2020

s-m-e commented Sep 25, 2020

s-m-e commented Sep 25, 2020

schlomo commented Sep 25, 2020

s-m-e commented Sep 25, 2020

schlomo commented Sep 30, 2020

s-m-e commented Sep 30, 2020