IW3: Runtime error when using MPS as processor on macOS

[josedavidrp]

If I try to choose MPS as processor this error will show up, if I choose CPU the 3D conversion will start so I assume I followed the install instructions correctly.

If more info is needed, let me know please.

  File "/Users/jdavid/nunif/iw3/gui.py", line 1217, in on_exit_worker
    args = result.get()
           ^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/wx/lib/delayedresult.py", line 232, in get
    raise self.__exception
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/wx/lib/delayedresult.py", line 267, in wrapper
    result = workerFn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/utils.py", line 1722, in iw3_main
    process_video(args.input, args.output, args, depth_model, side_model)
  File "/Users/jdavid/nunif/iw3/utils.py", line 802, in process_video
    process_video_full(input_filename, output_path, args, depth_model, side_model)
  File "/Users/jdavid/nunif/iw3/utils.py", line 751, in process_video_full
    VU.process_video(input_filename, output_filename,
  File "/Users/jdavid/nunif/nunif/utils/video.py", line 719, in process_video
    for new_frame in get_new_frames(frame_callback(frame)):
                                    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/nunif/utils/video.py", line 1155, in __call__
    frames = future.result()
             ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/utils.py", line 741, in _batch_callback
    return __batch_callback(x)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/utils.py", line 709, in __batch_callback
    depths = args.state["depth_utils"].batch_infer(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/depth_anything_model.py", line 195, in batch_infer
    out = _forward(model, x, enable_amp)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/depth_anything_model.py", line 164, in _forward
    out = model(x).unsqueeze(dim=1)
          ^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/pretrained_models/hub/nagadomi_Depth-Anything_iw3_main/depth_anything/dpt.py", line 179, in forward
    depth = self.depth_head(features, patch_h, patch_w)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/iw3/pretrained_models/hub/nagadomi_Depth-Anything_iw3_main/depth_anything/dpt.py", line 114, in forward
    x = self.projects[i](x)
        ^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 554, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jdavid/nunif/.venv/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^

[nagadomi]

Same as #248

For DepthAnything, it is already using reshape, so it is a problem on PyTorch MPS side and cannot be fixed.

            x = x.permute(0, 2, 1).reshape((x.shape[0], x.shape[-1], patch_h, patch_w))
            
            x = self.projects[i](x)

https://github.com/nagadomi/Depth-Anything_iw3/blob/82a94d6fb5fd68add51d9a8be20e43249e728c81/metric_depth/zoedepth_depth_anything/models/base_models/dpt_dinov2/dpt.py#L108-L110

If you are using an older version of pytorch, try building an env with the latest pytorch version using venv.

---

I changed it now to use contiguous() explicitly, to be sure. nagadomi/Depth-Anything_iw3@e712ebf

Try update with

python3 -m iw3.download_models

If it is not fixed, then it is a PyTorch MPS backend problem and cannot be fixed on my end.

[nagadomi]

This error is dependent on the depth estimation model.
When I checked previously, ZoeD_N and Any_S were working, if I recall correctly.

I don't have macOS, could you check which model doesn't work?

• ZoeD_N
• Any_S
• Any_V2_S
• DepthPro (This only exists in the dev branch, if not using dev branch, no need to check)

[josedavidrp]

Those seem to work, along Any_V2_N_B, but seems that they might fail to work in macOS 15.4 beta
/AppleInternal/Library/BuildRoots/79d43cf9-ebfd-11ef-8231-fe9e33ca05fa/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:3532: failed assertion Error: MLIR pass manager failed'
`

[nagadomi]

Do you mean the problem you originally posted has been fixed?
Is macOS 15.4 beta a different environment than the original post?

For beta OS released a few days ago, it would be natural to assume that it is not yet supported.