How well openpiv-python deals with the upcoming huge PIV cameras?

[alexlib]

Is your feature request related to a problem? Please describe.
Here is a link to the issue on PIVLab: Shrediquette/PIVlab#68 (comment)

The message is about differences in RAM and disk usage when 25 megapixel images are analysed for PIV. I wonder how does openpiv-python performs and where are the bottlenecks.

Describe the solution you'd like
A volunteer is needed to create a good test environment based on debuggers and memory profilers, create different increasing size PIV test images up to 100 Mp and test performance. The main goal is to find the bottlenecks and understand them, looking for possible future enhancements of the code.

[ErichZimmer]

FYI, scipy FFTs operate better with dtype of float32 compared to other dtypes because some reason scipy processes in float64 on everything except float32. Additionally, it is faster by 10-112% and takes half as much RAM (good for multiprocessing).

[alexlib]

Good point. We need to check everywhere what dtype is used.

[ErichZimmer]

On possible bottlenecks, scipy/numpy's use of vectorized FFT algorithms could be one due to its high memory usage. A general rule of thumb is 1GB of free RAM per core to be used mostly eliminates it, but it still could be an issue for images >3 MP with window sizes <32 pixels and using multi/parallel-processing. This seems to be a python/matlab issue as nearly all c++ softwares using FFTW + OpenMP use very little memory and are orders of magnitude faster on large images. Another possible bottleneck is local median validation and second peak searching algorithms as it can take some time when there is a large amount of vectors (>50,000).

[alexlib]

Probably you are right. We need some profiling tools to learn the issue and prepare some heavy load tests. Meanwhile it is only a wishful future extension.

[ErichZimmer]

Have we tried chunking the 3D stack of arrays and use the multiprocessing.Process module for parallel processing? This is what Fluere uses and it's as fast as some commercial softwares. It also lowers the RAM requirements.

[alexlib]

Windef has some multiprocessing use. Not sure that it is more than that. Please try

…

________________________________ From: Erich Zimmer ***@***.***> Sent: Monday, November 29, 2021 7:56:45 PM To: OpenPIV/openpiv-python ***@***.***> Cc: Alex Liberzon ***@***.***>; Author ***@***.***> Subject: Re: [OpenPIV/openpiv-python] How well openpiv-python deals with the upcoming huge PIV cameras? (#209) Have we tried chunking the 3D stack of arrays and use the multiprocessing.Process module for parallel processing? This is what Fluere uses and it's as fast as some commercial softwares. It also lowers the RAM requirements. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#209 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAFWMZU57FWIDCYGTIV5VN3UOO5F3ANCNFSM5CLKOV5Q>.

[ErichZimmer]

For the testing of large images, my personal GUI now has a synthetic image generator based on synimagegen and PIVlab. It should give us a good idea of the bottlenecks.
Additionally, if you like, I could make it a standalone GUI.