Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't import mitsuba in a conda environment with openexr installed [🐛 bug report] #411

Open
Microno95 opened this issue Apr 2, 2021 · 12 comments · May be fixed by #481
Open

Can't import mitsuba in a conda environment with openexr installed [🐛 bug report] #411

Microno95 opened this issue Apr 2, 2021 · 12 comments · May be fixed by #481

Comments

@Microno95
Copy link

Microno95 commented Apr 2, 2021

Summary

Unable to import mitsuba in a conda environment on windows if the environment has openexr and openexr-python installed as mitsuba cannot find its own openexr libraries.

System configuration

  • Platform: Windows
  • Compiler: MSVC 2019 (Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29337 for x64)
  • Python version: 3.8.5
  • Mitsuba 2 version: 2.2.1
  • Compiled variants:
    • scalar_rgb
    • scalar_mono
    • scalar_spectral

Description

When importing Mitsuba 2 in a conda environment on Windows where openexr and openexr-python (python bindings for OpenEXR) are installed from the conda repositories, the mitsuba modules do not correctly load due to the .dll search path prioritising the environment openexr libraries over the mitsuba compiled ones. For example, Half.dll in the environment is found before the Half.dll library in dist/. You can check this by running:

>>> import ctypes
>>> import ctypes.util
>>> print(ctypes.util.find_library('Half.dll'))
C:\Users\ekin4\.conda\envs\test_exr_mitsuba\Library\bin\Half.dll

I've tested this on a Linux conda environment where the issue does not occur so this is entirely a problem with how Windows handles loading of .dll files and where it searches for them.

Steps to reproduce

In a shell with conda installed

  1. conda create -n test_exr_mitsuba python=3.8 numpy scipy openexr openexr-python -c conda-forge
  2. python -m mitsuba (alternatively run python and call import mitsuba)
  3. The above should raise an error with the cause:
Traceback (most recent call last):
  File "G:\research_files\academic_work\git_repositories\mitsuba_renderer_folder\mitsuba2\build_windows\release\dist\python\mitsuba\__init__.py", line 16, in <module>
    _import('mitsuba.core_ext')
  File "C:\Users\ekin4\.conda\envs\test_exr_mitsuba\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing core_ext: The specified procedure could not be found.```
@Speierers
Copy link
Member

Hi @Microno95 ,

Just checking here that you are running setpath.bat before all this? Maybe it is enough to re-run that script after creating the conda environment?

@Microno95
Copy link
Author

Hi @Speierers,

I actually tried three configurations. setpath.bat before environment creation, setpath.bat before environment activation and setpath.bat after environment activation. All three results in failure to import mitsuba.

@Speierers
Copy link
Member

Could you print you path in Python? According to setpath.bat, the Mitsuba directory should have precedence:

set PATH=%MITSUBA_DIR%;%PATH%

@Microno95
Copy link
Author

Ah, I don't think that's the case in the master branch

set PATH=%PATH%;%MITSUBA_DIR%\dist;%MITSUBA_DIR%\build\dist

I'll try swapping to give the Mitsuba directory precedence and see if that fixes it.

@Microno95
Copy link
Author

Just tested that, and now OpenEXR fails to load instead with the error:

>>> import OpenEXR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing OpenEXR: The specified procedure could not be found.

It seems that the DLL path resolution correctly respects the PATH environment variable, but the same name for the library(s) between conda's OpenEXR and Mitsuba's own OpenEXR means that, depending on the order in PATH, one or the other will fail to load.

From what I've read, a Windows application cannot load multiple DLLs with the same name in the same process unless the DLLs are loaded via the absolute path to the library instead of relying on Windows to resolve the dependency.

@Microno95
Copy link
Author

This is the result of print(sys.path) in Python after running setpath.bat:

['', 'G:\\research_files\\academic_work\\git_repositories\\mitsuba_renderer_folder\\mitsuba2\\build_windows\\release\\dist\\python', 'G:\\research_files\\academic_work\\git_repositories\\mitsuba_renderer_folder\\mitsuba2', 'C:\\Users\\ekin4\\.conda\\envs\\test_exr_mitsuba2\\python38.zip', 'C:\\Users\\ekin4\\.conda\\envs\\test_exr_mitsuba2\\DLLs', 'C:\\Users\\ekin4\\.conda\\envs\\test_exr_mitsuba2\\lib', 'C:\\Users\\ekin4\\.conda\\envs\\test_exr_mitsuba2', 'C:\\Users\\ekin4\\.conda\\envs\\test_exr_mitsuba2\\lib\\site-packages']

@Speierers
Copy link
Member

And is it necessary for you to import both mitsuba and OpenEXR in your Python script? We would rather have this not being a limitation, but in the meantime, you could rely on the Mitsuba Bitmap class to manipulate EXR images maybe?

@Microno95
Copy link
Author

At the moment, not a big priority. I have set up a separate environment for manipulating .exr images, which, for now, is viable.

The convenience of using OpenEXR has been the documentation showing how to go from image data to numpy arrays and back. Other than that, there's no particular reason I could not use Bitmap as long as it lets me do such manipulations (wasn't sure how...).

@Speierers
Copy link
Member

Speierers commented Apr 15, 2021

You can definitely use the Bitmap class from mitsuba to do such manipulations (load file from disk, convert to numpy array, and back, write to file, ...)

Here is a little tuto:

# load image from disk
bmp = Bitmap("some_image_file.png")
# convert to numpy array
np_img = np.array(bmp)
# convert from numpy array
bmp2 = Bitmap(np_img)
# write down to file
bmp2.write("my_image.exr")

You can also specify the image format (e.g. RGB vs XYZ vs RGBA, ...) and the data type (float32 vs int32). Take a look at src/libcore/tests/test_bitmap.py for more examples.

@Microno95
Copy link
Author

Swapping out OpenEXR for Bitmap worked superbly, much easier interface too. Thanks!

Unfortunately, I ran into a similar issue involving pytorch (my script loads exr images into pytorch tensors). I believe it's due to the fact that pytorch comes with its own version of asmjit which conflicts with mitsuba's version. So in a given environment I can either load pytorch or mitsuba, but not both because the shared library for asmjit for one does not match the other.

I tried setting asmjit to be compiled as a static library instead of shared, but this then breaks mitsuba compilation with many unresolved external symbol errors in MSVC.

Once I figure out the minimal number of steps to reproduce the error, I'll create a proper bug report. But I thought I'd comment here in case you have some idea of what to look at.

@Microno95
Copy link
Author

Microno95 commented Jul 6, 2021

I figured out how to make asmjit compile as a static library. Essentially, the preprocessor variable ASMJIT_STATIC is not correctly defined when asmjit/asmjit.h is imported in include/mitsuba/core/jit.h, adding the following lines before the header is imported solves this issue:

#ifndef ASMJIT_STATIC
#define ASMJIT_STATIC
#endif

Now I can import pytorch and mitsuba simultaneously. I am still testing whether I can run them at the same time without issue.

@Speierers
Copy link
Member

Nice catch! If you think this should be pushed on the main repository, feel free to open a PR 🚀

@Microno95 Microno95 linked a pull request Aug 4, 2021 that will close this issue
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants