[Bug]: openvino 2024.5 NPU plugin fails to load convolution

[koush]

OpenVINO Version

2024.5

Operating System

Other (Please specify in description)

Device used for inference

NPU

Framework

None

Model used

https://github.com/koush/openvino-models/tree/main/scrypted_yolov6n_320/FP32

Issue description

Following error/crash is seen when using the NPU device to load the model:

loc(fused<{name = "__module.FeatureExtraction.ConvNet.0/aten::_convolution/Convolution", type = "Convolution"}>["__module.FeatureExtraction.ConvNet.0/aten::_convolution/Convolution"]): error: Channels count of input tensor shape and filter shape must be the same: -9223372036854775808 != 1
LLVM ERROR: Failed to infer result type(s).

AUTO works presumably because the error is ignored and GPU is used instead.

The values -9223372036854775808 != 1 seem to indicate some sort of bitmask/endian issue.

This error did not occur in 2024.4 or any of the 2024.5-nightly releases.

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[avitial]

@koush thanks for reaching out, tested the linked model and the benchmark_app executes normally on NPU (see below). Issue does not reproduce on NPU for Linux, can you please try using the latest NPU driver and see if the issue is fixed on your end? Find NPU driver for Linux and Windows.

If the issue persists, kindly share more details about your environment (CPU SKU, OS, Python version, etc.)

$ benchmark_app -m best.xml -d NPU -t 5
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.5.0-17288-7975fa5da0c-refs/pull/3856/head
[ INFO ]
[ INFO ] Device info:
[ INFO ] NPU
[ INFO ] Build ................................. 2024.5.0-17288-7975fa5da0c-refs/pull/3856/head
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(NPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 5.64 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     x (node: x) : f32 / [...] / [1,3,320,320]
[ INFO ] Model outputs:
[ INFO ]     ***NO_NAME*** (node: __module.model.28/aten::cat/Concat_5) : f32 / [...] / [1,7,2100]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     x (node: x) : u8 / [N,C,H,W] / [1,3,320,320]
[ INFO ] Model outputs:
[ INFO ]     ***NO_NAME*** (node: __module.model.28/aten::cat/Concat_5) : f32 / [...] / [1,7,2100]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 38.58 ms
[...]
[ INFO ] First inference took 7.17 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:NPU
[ INFO ] Count:            3100 iterations
[ INFO ] Duration:         5011.63 ms
[ INFO ] Latency:
[ INFO ]    Median:        6.32 ms
[ INFO ]    Average:       6.41 ms
[ INFO ]    Min:           5.68 ms
[ INFO ]    Max:           21.31 ms
[ INFO ] Throughput:   618.56 FPS

Environment:
OpenVINO 2024.5.0
Intel(R) Core(TM) Ultra 7 155H
Python 3.10.12
Ubuntu 22.04.5 LTS

[koush]

Yes, I can confirm that benchmark_app works. The issue occurs when using python openVINO bindings.

[koush]

I figured out what was wrong. The following mode fails with a crash:

mode=AUTO:NPU,GPU,CPU

but this works:

mode=NPU

[avitial]

@koush I am still unable to reproduce the issue, using AUTO plugin with device priority NPU,GPU,CPU works fine. I tried different combinations of device priority and still not able to see the issue. Can you provide a sample reproducer and more details about your environment (CPU SKU, Operating System, Python version, OpenVINO version, NPU driver version, GPU driver version, etc.). Do you know at what stage the crash occurs? i.e. model loading, or inference.

[koush]

It fails during model loading.

CPU is core ultra 125h
python is 3.10 or 3.12
openvno is 2024.5.0
NPU driver is latest (1.10.0 download from GitHub) and also tried the one you linked.
compute runtime is latest (24.45.31740.9 downloaded from GitHub) and also the tried apt version.

I'll try to get an isolated repro.

[avitial]

@koush do you have an update for us? Also can you share the operating system that you used in your configuration? Just curious as this detail was not listed in your original description of the issue.

[koush]

@koush do you have an update for us? Also can you share the operating system that you used in your configuration? Just curious as this detail was not listed in your original description of the issue.

Not yet, I will try to get to it soon. The operating system is an ubuntu docker container inside Ubuntu 24.04.