Slow when opening pictures on slow file systems (e.g., remote file systems)

[dschrempf]

When open one of many image in a directory of a slow file system (for example, remote file systems such as NFS, or SSHFS), vimiv takes a long time to start. I think this is because it caches many files in the directory, and not only the file to be opened. Is there a way to check if a file system is slow, and act accordingly? Many times, I only want to open a single file, and do not care about creating thumbnails for all other files in the directory.

Thanks for you help!

[karlch]

Thanks for your interest in vimiv!

Unfortunately there is currently no such option, and I don't quite see how this would be implemented unless we add a simple setting. But of course it would be nicer to actually improve the performance in general, so we don't need any workarounds. In principle the thumbnails are created in parallel, so this should not delay startup, but all files in the current (the one of the first image) directory are checked if they are images or not which could be the cause of the delay.

I did some quick checks and found the performance to be pretty bad on slow file systems in general. The checking of all of the files becomes a significant part of the loading time (> 30 %) once there are around 100 images in the working directory. But even without this there are some parts that are surprisingly slow, profile attached for reference.

Startup profile

         27687 function calls (27276 primitive calls) in 3.331 seconds

   Ordered by: cumulative time
   List reduced from 916 to 15 due to restriction <15>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    2.563    2.563 /home/christian/Coding/VimivQt/vimiv/startup.py:81(setup_post_app)
        1    0.000    0.000    1.167    1.167 /home/christian/Coding/VimivQt/vimiv/startup.py:113(init_paths)
        1    0.000    0.000    1.166    1.166 /home/christian/Coding/VimivQt/vimiv/api/__init__.py:57(open_paths)
        1    0.000    0.000    0.991    0.991 /home/christian/Coding/VimivQt/vimiv/api/working_directory.py:122(chdir)
        1    0.000    0.000    0.990    0.990 /home/christian/Coding/VimivQt/vimiv/api/working_directory.py:156(_load_directory)
        2    0.002    0.001    0.964    0.482 /home/christian/Coding/VimivQt/vimiv/utils/files.py:39(supported)
        1    0.000    0.000    0.958    0.958 /home/christian/Coding/VimivQt/vimiv/api/working_directory.py:218(_get_content)
      131    0.004    0.000    0.945    0.007 /usr/lib/python3.9/imghdr.py:11(what)
      130    0.002    0.000    0.939    0.007 /home/christian/Coding/VimivQt/vimiv/utils/files.py:117(is_image)
     15/7    0.000    0.000    0.809    0.116 /home/christian/Coding/VimivQt/vimiv/api/objreg.py:54(inside)
        1    0.760    0.760    0.761    0.761 /home/christian/Coding/VimivQt/vimiv/app.py:24(__init__)
       77    0.722    0.009    0.722    0.009 {method 'connect' of 'PyQt5.QtCore.pyqtBoundSignal' objects}
        1    0.000    0.000    0.721    0.721 /home/christian/Coding/VimivQt/vimiv/api/working_directory.py:233(init)
        1    0.000    0.000    0.721    0.721 /home/christian/Coding/VimivQt/vimiv/api/working_directory.py:105(__init__)
        1    0.000    0.000    0.637    0.637 /home/christian/Coding/VimivQt/vimiv/startup.py:125(init_ui)


         27687 function calls (27276 primitive calls) in 3.331 seconds

   Ordered by: internal time
   List reduced from 916 to 15 due to restriction <15>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.760    0.760    0.761    0.761 /home/christian/Coding/VimivQt/vimiv/app.py:24(__init__)
       77    0.722    0.009    0.722    0.009 {method 'connect' of 'PyQt5.QtCore.pyqtBoundSignal' objects}
        1    0.589    0.589    0.589    0.589 {function MainWindow.show at 0x7f5b63733310}
      147    0.533    0.004    0.533    0.004 {method 'read' of '_io.BufferedReader' objects}
      135    0.384    0.003    0.384    0.003 {built-in method io.open}
        1    0.134    0.134    0.134    0.134 {fromImageReader}
      556    0.039    0.000    0.039    0.000 {built-in method posix.stat}
        2    0.028    0.014    0.028    0.014 {built-in method _imp.create_dynamic}
        1    0.026    0.026    0.026    0.026 /home/christian/Coding/VimivQt/vimiv/gui/commandline.py:50(__init__)
      131    0.023    0.000    0.023    0.000 {method 'close' of '_io.BufferedReader' objects}
       12    0.006    0.001    0.006    0.001 {built-in method posix.listdir}
        1    0.006    0.006    0.006    0.006 {built-in method canRead}
        1    0.005    0.005    0.007    0.007 /home/christian/Coding/VimivQt/vimiv/gui/thumbnail.py:79(__init__)
        2    0.005    0.002    0.030    0.015 /home/christian/Coding/VimivQt/vimiv/gui/library.py:448(_add_rows)
        1    0.004    0.004    0.005    0.005 /home/christian/Coding/VimivQt/vimiv/gui/statusbar.py:47(__init__)

If you would like to test how much of an impact the checking of all files has in your case, and are willing to hack the code, you could change the _load_directory method in vimiv.api.working_directory around line 156 to

    def _load_directory(self, directory: str) -> None:
        """Load supported files for new directory."""
        self._dir = directory
        return  # Quick hack to avoid checking the files in the current directory
        self._images, self._directories = self._get_content(directory)
        self.loaded.emit(self._images, self._directories)

In case this already improves the performance "enough" for your use-case, we could think about adding a setting for this as a workaround. Obviously this is not beautiful, but very easy to do. Finding a proper way to speed vimiv up requires more thinking, as already mentioned in #124..

[dschrempf]

Thank you for your detailed reply! I tried your suggestion and it reduces the startup time from 1 minute to 3 seconds for a remote directory mounted with SSHFS containing 240 pictures. The question is, what is lost when skipping this check or can it maybe be done in the background?

BTW, also when closing vimiv-qt, there is a significant delay --- maybe for similar reasons?

EDIT: It seems that also the delay when closing the application is gone with your function including the quick hack.

[karlch]

Loading the content in the current directory:

• Allows us to have all images in the current directory in the filelist automatically when a single one is opened. This is especially useful when opening an image from the library.
• Loads the images and directories of the cwd into the library.
• Enables support for a changing filelist, e.g. reloading the library when files are deleted on disk.

A useful library is one of the main features for vimiv in my opinion, and is actually the reason why I started writing it in the first place. I wanted a proper "ranger-like" way to navigate my image library integrated into an image viewer with vim-like keybindings.

Now this is of course not needed for the use-case "open one image and open it fast" and this issue has also came up previously, but given this sets vimiv apart I am quite resilient to disabling it.

Loading the directory in the background could work, I just never felt this is necessary as I almost always run vimiv from a fast SSD and the overhead felt over-the-top compared to the possible benefits. Looking at your results I certainly have to reconsider 😆 I will take a look into this, but I can make no promises as it may be more complicated than I can think of now.

[woefe]

In #451 you can see that stat() calls are quite expensive and can be reduced.

From the call graph pictures you can also see that the imghdr.what() is slow on remote file systems. Maybe adding a config option to detect images purely based on file endings might be a solution, since that would remove the expensive open() from files.is_image(). I'm aware that this proposal probably causes other issues (e.g. what happens if vimiv tries to display an unsupported file; worse image detection; ...), but it would probably be a compromise I'd find acceptable.

[wedens]

Is there any way to disable thumbnails generation to alleviate this issue when only one specific image is being open on a slow FS?

[jcjgraf]

Hey @wedens, unfortunately there is no straight-forward way. Some efforts have been put in to improving the thumbnail generation, but they have not (yet?) landed in mainline (#665). The best I can suggest is to try out that PR ans see whether it helps in your case.