Cannot map shared library installed by CMake subproject to wheel install location

[tobiasdiez]

In a pyproject-based project, an external dependency is built using cmake (and another one using autotools) as fallback. This works well using plain meson, however pip install --editable . fails with

meson-python: error: Could not map installation path to an equivalent wheel directory: '{prefix}/lib/.so'

(see https://github.com/sagemath/sage/actions/runs/8222294612/job/22483617409?pr=37077#step:5:54174)

I can workaround this error message by providing the skip-subprojects argument for install. Then the editable wheel is built successful, but on runtime it does not find the <external>.so file just built.

I've seen

• Best practice for optionally bundling a library dependency #556
• How can I bundle a shared library in a wheel with mesonpy? #410
• Building Windows wheels with libraries fails #525

which discuss similar issues. However, my main problem (at least for the moment) is to create an editable installation for (local) development, and not build a wheel for pypi. So cibuildwheel etc are not really a solution.

[dnicolodi]

Does it work if you build a regular wheel? I don't think there is a way for meson-python to support cmake or autotools subprojects: there is no way to map the files installed by these subprojects to wheel install locations.

[dnicolodi]

By the way, what's the purpose of building an editable wheel for running the tests in CI? The only advantage of editable build is to have the package automatic rebuilt on edit, but I think your CI jobs are not editing the source code.

[dnicolodi]

To clarify: meson-python assembles a Python wheel from the files installed by the Meson project. To do so it uses introspection data exported by Meson and maps installation locations into the host filesystem to wheel install locations. The wheel format is designed to install Python modules, Python extension modules, and little else. meson-python somehow extends this to also support shared libraries (but not on Windows), which are installed in a non-standard location. Consequently, meson-python is able to map files installed to a few locations:

meson-python/mesonpy/__init__.py

Lines 83 to 93 in 2daa991

	_INSTALLATION_PATH_MAP = {
	'{bindir}': 'scripts',
	'{py_purelib}': 'purelib',
	'{py_platlib}': 'platlib',
	'{moduledir_shared}': 'platlib',
	'{includedir}': 'headers',
	'{datadir}': 'data',
	# custom location
	'{libdir}': 'mesonpy-libs',
	'{libdir_shared}': 'mesonpy-libs',
	}

These locations are enough to cover all files that can be mapped into a wheel. However, when using the cmake subproject support, Meson does not map the files installed by the cmake project to these installation prefixes. I think it could be possible to extend the logic used to map filesystem locations to wheel install paths to also understand {prefix}/lib/.

Subprojects using the Meson's external project module to build and autotools project do not have a list of the installed files, thus are supported by meson-python.

[dnicolodi]

I must take back some of the considerations above. I just put together a simple test package that installs an extension module that links to a shared library compiled as part of a cmake subproject and all works as expected. Apparently Meson is smart enough to correctly assign the shared library to the {libdir_shared} install location.

I don't know why in your case the install location for the shared library is {prefix}/lib/. We need much more information that what has been provided so far to understand where the issue is. I suspect it depends on how the cmake project is setup. However, I know little to nothing about cmake and I'm not sure I can debug this. We would at least need to know which library causes the issue.

[dnicolodi]

See #599. Interestingly, the test works on macOS but fails on Linux. There is a difference in how the Meson cmake modules maps installations paths on macOS and Linux. Some more investigation is required.

[tobiasdiez]

Oh wow @dnicolodi. Thanks a lot for diving so quickly into this issue. Do I understand it correctly, that with #599 you are able to reproduce the problem? Let me know if there is something I can do to assist in fixing or if you want me to try a new build etc.

To respond to your questions:

By the way, what's the purpose of building an editable wheel for running the tests in CI? The only advantage of editable build is to have the package automatic rebuilt on edit, but I think your CI jobs are not editing the source code.

No particular reason. I had various problems with the ci build and couldn't reproduce them locally within a devcontainer. So I tried to minimize the difference in the environments. Migrating sagemath to meson is a huge undertaking, still quite at the beginning and there are a lot of moving parts. Of course, normally you don't want/need to use editable builds for CI.

Subprojects using the Meson's external project module to build and autotools project do not have a list of the installed files, thus are supported by meson-python.

You probably meant not supported, right? This partly makes sense. On the other hand, an external project is also just a custom target, or not? And meson somehow understands where it should install the files afterwards.

I don't know why in your case the install location for the shared library is {prefix}/lib/. We need much more information that what has been provided so far to understand where the issue is. I suspect it depends on how the cmake project is setup. However, I know little to nothing about cmake and I'm not sure I can debug this. We would at least need to know which library causes the issue.

The subproject in question is flint as configured in https://github.com/sagemath/sage/blob/042c42ecfadce27d85be05ded583487dc570fca6/subprojects/flint/meson.build (whole PR sagemath/sage#37077). But it's a quite hacky setup and it turns out there are even issues with plain meson (meson claims to "Installing subprojects/flint-3.0.1/libflint.so to /usr/local/lib" but afterwards it cannot be found/its shadowed by a older system version). So I don't think it serves as a good isolated testing environment.

[dnicolodi]

Thanks a lot for diving so quickly into this issue.

No problem. Some of my early comments are based on wrong assumptions about how Meson's support for CMake subprojects works. Sorry for the confusion.

Do I understand it correctly, that with #599 you are able to reproduce the problem?

Yes, this is the case. I don't understand yet the root cause of the issue but I have what I need to investigate.

You probably meant not supported, right?

Yes, I meant not supported. But the statement is based on a wrong assumption on my part.

This partly makes sense. On the other hand, an external project is also just a custom target, or not? And meson somehow understands where it should install the files afterwards.

Not really. For CMake subprojects, Meson uses a reimplementation of the CMake interpreter that translates the CMake build definition into a Meson build definition and runs that with the regular Meson interpreter, with some limitations. Consequently, Meson knows which build artifacts are installed where, and meson-python can use this information to build the Python wheel. Unfortunately, it seems that somewhere Meson gets confused and does not assign the correct installation location to shared libraries build via the CMake "emulation" on Linux. This is the source of the issue you observe.

For autotools subprojects included with the external_project Meson module, Meson runs the usual configure, make, make install steps specifying as $DESTDIR a directory inside the Meson build directory. All files installed there are copied into their final location when the Meson project is installed. For subprojects using the external_projects module, Meson does not know (and cannot easily know) which files are installed where. The only information available to meson-python is that there is a directory tree in the buildir that needs to be copied into {prefix}. This is currently not supported, but could be supported with some heuristic (and some limitations, for example assuming that the subproject does not install Python modules). However, we did some work to get rid of such heuristic, and I would like to avoid to bring it back, unless there are very important reasons. The support for including shared libraries in Python wheels does anyhow not currently work on Windows, thus if you need cross-platform support, the story is complicated anyway.

But it's a quite hacky setup

Indeed. In my experience, it is much less time consuming to write a proper Meson build definition for the external projects that to try to work around all quirks. Especially for mature projects, it is mostly a one time cost.