UnableToOpenBlob : Too many open files

[omaryangtw]

After load() a number of images, Julia throw a Fatal error says 'Too many open files' and cannot load() any file at all

To represent the issue, try to load some images

files = readdir()
imgs = Any[]
for i in 1:1000
    io = open(files[i])
    push!(imgs, copy(load(io)))
    close(io)
end

in my case, this issue occurs at 510th images loaded

---

Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: AMD Ryzen 7 5800X 8-Core Processor
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-11.0.1 (ORCJIT, generic)

[johnnychen94]

Could you provide more verbose information for diagnosing? For example, the complete error message, and the result of versioninfo(), what image io backends you've installed, and what Images/FileIO version you're using.

This is very likely not an Images.jl issue.

And, you can simply load an image file by using push!(imgs, load(files[i])).

[omaryangtw]

Error encountered while load Stream{DataFormat{:JPEG}, IOStream, Nothing}(IOStream(<file 10458_茂.jpg>), nothing).

Fatal error:
UnableToOpenBlob `C:/Users/omar/AppData/Local/Temp/magick-33928B3rO7l6jEVRv': Too many open files @ error/blob.c/OpenBlob/2873
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] error(wand::ImageMagick.MagickWand)
   @ ImageMagick C:\Users\omar\.julia\packages\ImageMagick\b8swT\src\libmagickwand.jl:187
 [3] readimage
   @ C:\Users\omar\.julia\packages\ImageMagick\b8swT\src\libmagickwand.jl:280 [inlined]
 [4] load_(file::IOStream, permute_horizontal::Bool; ImageType::Type, extraprop::String, extrapropertynames::Nothing, view::Bool)
   @ ImageMagick C:\Users\omar\.julia\packages\ImageMagick\b8swT\src\ImageMagick.jl:146
 [5] load_ (repeats 2 times)
   @ C:\Users\omar\.julia\packages\ImageMagick\b8swT\src\ImageMagick.jl:139 [inlined]
 [6] load(::Stream{var"#s10", IOtype, Name} where {var"#s10"<:DataFormat, IOtype<:IO, Name}; key_args::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ ImageMagick C:\Users\omar\.julia\packages\ImageMagick\b8swT\src\ImageMagick.jl:130

Stacktrace:
  [1] handle_error(e::ErrorException, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\error_handling.jl:61
  [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\error_handling.jl:56
  [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:225
  [4] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted)
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:196
  [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::IOStream; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:183
  [6] action
    @ C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:183 [inlined]
  [7] #load#14
    @ C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:113 [inlined]
  [8] load(::IOStream)
    @ FileIO C:\Users\omar\.julia\packages\FileIO\7CSs0\src\loadsave.jl:110
  [9] top-level scope
    @ .\In[4]:8
 [10] eval
    @ .\boot.jl:360 [inlined]
 [11] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base .\loading.jl:1094

And yes, normally I can just load file with load(file) , but my filename contains unicode, so I tried to load files in this way.
This was how I ran into this issue.

[johnnychen94]

This might be a windows specific issue related to the filesystems. There's nothing we can help fix on the Images.jl side so I'm closing it.

To help further diagnosing, you can try this even simpler example with only open/read and see if it still errors. Because if it still errors, it's certainly an upstream issue that should be handled by Julialang/julia and you can open a new issue there:

rst = []
for i in 1:1000
    open(files[i]) do io
        push!(rst, read(io, UInt8))
    end
end

If not, then it's perhaps an image io backend issue (in this case it's ImageMagick), you can try if ImageIO works by testing your original version on a dataset of png files. (Currently, jpeg files are passed to ImageMagick.jl, and png files are passed to ImageIO.jl, which calls libpng eventually.) If it's an ImageMagick issue, please open a new issue there. (Although we don't have any active maintainers taking care of ImageMagick...)

---

A possible workaround to this very specific issue is to call GC.gc() every a few iterations, even if it works, it is only a workaround:

map(enumerate(files)) do i, fn
    i%100 == 0 && Base.Sys.iswindows() && GC.gc()
    open(fn) do io
        load(io)
    end
end

[johnnychen94]

If it's an ImageMagick issue, please open a new issue there. (Although we don't have any active maintainers taking care of ImageMagick...)

If it's really an ImageMagick issue, another possible solution is to call for a Julia wrapper for libjpeg-turbo so that we can pass jpeg files to libjpeg-turbo instead of imagemagick #960.

[kimikage]

See also: JuliaIO/ImageMagick.jl#145

The handle leak problem has been fixed in JuliaIO/ImageMagick.jl#144, but I have not checked the path encoding problem. So, perhaps we should cherry-pick JuliaIO/ImageMagick.jl@69206af first.

[DhairyaLGandhi]

What was the resolution here? I don't think the related ImageMagick prs were merged? I run into too many files opening while trying to train ImageNet with Flux on multiple GPUs.

[johnnychen94]

What kind of files are you loading? PNG or JPEG, or others?

Maybe related to JuliaIO/FileIO.jl#352