Custom loader undefined when precompilation of a module fails.

[m-j-w]

While observing the current error with StaticArrays on current Julia master, I also observed an issue when using a custom module loader for modules to be precompiled but fail.

An MWE looks like the following: Create a module that should precompile, but fails (this simulates the StaticArrays problem):

# This goes into a file 'ThisOneFails.jl' in the current directory
__precompile__()
module ThisOneFails
    throw(ArgumentError("This failed."))
end

Then, creating a custom module loader in a new module, and trying to import the above failing module, also fails as shown below.

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-DEV.1723 (2017-09-08 14:47 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 87d8af7 (2 days old master)
|__/                   |  x86_64-linux-gnu

julia> push!(Base.LOAD_PATH, ".");

julia> module CustomModuleLoaders

       struct ModuleLoader end

       import Base: load_hook
       Base.load_hook(prefix::ModuleLoader, name::String, found) = begin
           info("ModuleLoader: Looking for a '$name'...")
           found
       end

       __init__() = push!(Base.LOAD_PATH, ModuleLoader())

       end # module
CustomModuleLoaders

julia> Base.LOAD_PATH
4-element Array{Any,1}:
 "/home/mjw/src/julia/usr/local/share/julia/site/v0.7"
 "/home/mjw/src/julia/usr/share/julia/site/v0.7"      
 "."                                                  
 CustomModuleLoaders.ModuleLoader()                   

julia> using ThisOneFails
INFO: ModuleLoader: Looking for a 'ThisOneFails'...
INFO: ModuleLoader: Looking for a 'ThisOneFails'...
INFO: Precompiling module ThisOneFails.
ERROR: UndefVarError: CustomModuleLoaders not defined
Stacktrace:
 [1] deserialize_module(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:785
 [2] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:726
 [3] deserialize(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:665
 [4] deserialize_datatype(::SerializationState{Base.PipeEndpoint}, ::Bool) at ./serialize.jl:1003
 [5] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:705
 [6] deserialize(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:665
 [7] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:712
 [8] deserialize_array(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:906
 [9] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:703
 [10] deserialize(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:665
 [11] deserialize_expr(::SerializationState{Base.PipeEndpoint}, ::Int64) at ./serialize.jl:917
 [12] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:724
 [13] deserialize(::SerializationState{Base.PipeEndpoint}) at ./serialize.jl:665
 [14] deserialize_expr(::SerializationState{Base.PipeEndpoint}, ::Int64) at ./serialize.jl:917
 [15] handle_deserialize(::SerializationState{Base.PipeEndpoint}, ::Int32) at ./serialize.jl:724
 [16] deserialize(::Base.PipeEndpoint) at ./serialize.jl:662
 [17] anonymous at ./<missing>:2
ERROR: Failed to precompile ThisOneFails to /home/mjw/.julia/lib/v0.7/ThisOneFails.ji.
Stacktrace:
 [1] compilecache(::String) at ./loading.jl:598
 [2] _require(::Symbol) at ./loading.jl:407
 [3] require(::Symbol) at ./loading.jl:318

Without __precompile__() in ThisOneFails.jl everything looks fine as it should:

julia> using ThisOneFails
INFO: ModuleLoader: Looking for a 'ThisOneFails'...
ERROR: LoadError: ArgumentError: This failed.
Stacktrace:
 [1] include_relative(::Module, ::String) at ./loading.jl:464
 [2] _require(::Symbol) at ./loading.jl:401
 [3] require(::Symbol) at ./loading.jl:318
while loading /home/mjw/tmp/ThisOneFails.jl, in expression starting on line 4

[m-j-w]

Well, I believe the error of the undefined custom loader to always occur when precompilation of a module happens. Regardless of whether successful or not. But I haven't properly confirmed, yet.

[m-j-w]

And this would be my use-case:

"""
The module `PkgAutoInstall` will automatically download and add
all registered packages with a name that corresponds to a module `m`
that is intended to be loaded, viz. with `using m`, `import m`, or
`require(:m)` statements, respectively.  This will also automatically
add packages that other package authors forgot to add to their `REQUIRE`
file.

Usage:
1) Type `Pkg.add("PkgAutoInstall")` -- for the last time; then,
2) Type `using PkgAutoInstall`, or add the same to your `~/.juliarc.jl`
3) Test it with a package that you haven't installed yet.

Note: If you have other custom directories in your `Base.LOAD_PATH`,
make sure to first add those, then load PkgAutoInstall afterwards.
"""
module PkgAutoInstall

struct PkgAutoInstaller end

import Base: load_hook
function Base.load_hook(prefix::PkgAutoInstaller, name::String, found)
    # drop out if there is already a match
    found !== nothing && return found
    # otherwise, figure out whether there is a package of that name
    try
        Pkg.available(name) # if this doesn't throw, then install it
    catch err
        warn("PkgAutoInstall: '$name' is not a registered package. " *
             "Cannot install automatically. Typo?")
        return nothing
    end
    info("PkgAutoInstall: Installing registered package '$name'...")
    try
        Pkg.add(name)
    catch err
        warn("PkgAutoInstall: Installation of registered package '$name' " *
             "failed. $err")
        return nothing
    end    
    info("PkgAutoInstall: Registered package '$name' is now available.")
    # Now try again, using the default Pkg loader
    Base.load_hook(string(Pkg.dir()), name, nothing)
end

__init__() = begin
    push!(Base.LOAD_PATH, PkgAutoInstaller())
    info("PkgAutoInstall: Automatic installation of registered packages " *
         "is now enabled.")
end

end # module

[vtjnash]

I don't know what Base.load_hook was, but it does not exist now