Running the same image (asynchronously) on different models

[eladoni1]

Running C++ Openvino version 2023.3.0, Ubuntu 20.04.6 LTS.
The model files are generated from openvino 2024.0, exported from YOLOv8

Will try to explain -
I would like to run the same input, asynchronously, on 2 or more models. Current problem I am facing -

I have been using the sample '/samples/cpp/classification_sample_async', and tried to read more than one model, and run it using the 'start_async' function.

I've seen no improvement from the sequential.

code - (We are trying to load the same model 3 times)

/* -------- Step 2. Compile the Model --------*/
ov::Core core;
auto compiled_model = core.compile_model("/home/%user%/Desktop/test_for_open_vino/best_openvino_model/best.xml", "CPU");
auto compiled_model2 = core.compile_model("/home/%user%/Desktop/test_for_open_vino/best_openvino_model/best.xml", "CPU");
auto compiled_model3 = core.compile_model("/home/%user%/Desktop/test_for_open_vino/best_openvino_model/best.xml", "CPU");


// -------- Step 3. Create an Inference Request --------

ov::InferRequest infer_request = compiled_model.create_infer_request();
ov::InferRequest infer_request2 = compiled_model2.create_infer_request();
ov::InferRequest infer_request3 = compiled_model3.create_infer_request();

// -------- Step 4.Read a picture file and do the preprocess --------
Mat img = cv::imread("/home/%user%/Desktop/test_for_open_vino/data_resized32/0.bmp");
// Preprocess the image
Mat blob = cv::dnn::blobFromImage(img, 1.0 / 255.0, Size(576, 576), Scalar(), true);

// -------- Step 5. Feed the blob into the input node of the Model -------
// Get input port for model with one input
auto input_port = compiled_model.input();
 //Create tensor from external memory
ov::Tensor input_tensor(input_port.get_element_type(), input_port.get_shape(), blob.ptr(0));
// Set input tensor for model with one input
infer_request.set_input_tensor(input_tensor);


Mat blob2 = blob.clone();
auto input_port2 = compiled_model2.input();
// Create tensor from external memory
ov::Tensor input_tensor2(input_port2.get_element_type(), input_port2.get_shape(), blob2.ptr(0));
// Set input tensor for model with one input
infer_request2.set_input_tensor(input_tensor2);


Mat blob3 = blob.clone();
auto input_port3 = compiled_model3.input();
// Create tensor from external memory
ov::Tensor input_tensor3(input_port3.get_element_type(), input_port3.get_shape(), blob3.ptr(0));
// Set input tensor for model with one input
infer_request3.set_input_tensor(input_tensor3);



// -------- Step 6. Start inference --------


auto t1 = chrono::high_resolution_clock::now();
infer_request.start_async();
infer_request2.start_async();
infer_request3.start_async();

cout << chrono::duration_cast<chrono::milliseconds>(chrono::high_resolution_clock::now() - t1).count() << endl;

infer_request.wait();
infer_request2.wait();
infer_request3.wait();

cout << chrono::duration_cast<chrono::milliseconds>(chrono::high_resolution_clock::now() - t1).count() << endl;

I then stumbled upon the following sample, using face detector, which then followed by multiple async models - emotion, age, etc.

https://github.com/openvinotoolkit/open_model_zoo/tree/master/demos/interactive_face_detection_demo/cpp

Although it is exactly the same thing we are aiming to achieve (running few models on SAME input - in this sample's case, the cropped face), I didn't see any relevant code changes from what we have already tried.

If you could reference me for the right documentation, or if you knew what was happening here behind the scenes to explain this situation, would be happy to know~

running lscpu -e :

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ MINMHZ
0 0 0 0 0:0:0:0 yes 4800.0000 800.0000
1 0 0 1 1:1:1:0 yes 4800.0000 800.0000
2 0 0 2 2:2:2:0 yes 4900.0000 800.0000
3 0 0 3 3:3:3:0 yes 4900.0000 800.0000
4 0 0 4 4:4:4:0 yes 4800.0000 800.0000
5 0 0 5 5:5:5:0 yes 4800.0000 800.0000
6 0 0 6 6:6:6:0 yes 4800.0000 800.0000
7 0 0 7 7:7:7:0 yes 4800.0000 800.0000
8 0 0 0 0:0:0:0 yes 4800.0000 800.0000
9 0 0 1 1:1:1:0 yes 4800.0000 800.0000
10 0 0 2 2:2:2:0 yes 4900.0000 800.0000
11 0 0 3 3:3:3:0 yes 4900.0000 800.0000
12 0 0 4 4:4:4:0 yes 4800.0000 800.0000
13 0 0 5 5:5:5:0 yes 4800.0000 800.0000
14 0 0 6 6:6:6:0 yes 4800.0000 800.0000
15 0 0 7 7:7:7:0 yes 4800.0000 800.0000

running lscpu :

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 167
Model name: 11th Gen Intel(R) Core(TM) i9-11900T @ 1.50G
Hz
Stepping: 1
CPU MHz: 1500.000
CPU max MHz: 4900.0000
CPU min MHz: 800.0000
BogoMIPS: 3024.00
Virtualization: VT-x
L1d cache: 384 KiB
L1i cache: 256 KiB
L2 cache: 4 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerabl
e
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disable
d via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and u
ser pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional,
RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmov pat pse36 clflush dts acpi
mmx fxsr sse sse2 ss ht tm pbe syscall nx p
dpe1gb rdtscp lm constant_tsc art arch_perfm
on pebs bts rep_good nopl xtopology nonstop
tsc cpuid aperfmperf tsc_known_freq pni pclm
ulqdq dtes64 monitor ds_cpl vmx smx est tm2
ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 ss
e4_2 x2apic movbe popcnt tsc_deadline_timer
aes xsave avx f16c rdrand lahf_lm abm 3dnowp
refetch cpuid_fault epb invpcid_single ssbd
ibrs ibpb stibp ibrs_enhanced tpr_shadow vnm
i flexpriority ept vpid ept_ad fsgsbase tsc
adjust bmi1 avx2 smep bmi2 erms invpcid mpx
avx512f avx512dq rdseed adx smap avx512ifma
clflushopt intel_pt avx512cd sha_ni avx512bw
avx512vl xsaveopt xsavec xgetbv1 xsaves dth
erm ida arat pln pts hwp hwp_notify hwp_act_
window hwp_epp hwp_pkg_req avx512vbmi umip p
ku ospke avx512_vbmi2 gfni vaes vpclmulqdq a
vx512_vnni avx512_bitalg avx512_vpopcntdq rd
pid fsrm md_clear flush_l1d arch_capabilitie
s

[hegdeadithyak]

I think this should solve your issue : https://gist.github.com/dkurt/59a7e7b5b45f9c46b31e01ce47d098a2

Thanks,
Adithya Hegde Kota