Can not call @sync from another macro in Julia 0.7

[mattcbro]

It appears to be impossible to use the @sync macro within another macro or conversely call @sync on a user defined macro.

Case in point:

using Distributed
macro parsync(args...)
    :(@sync @distributed $(args...))
end

Use of this macro gives:

julia> @parsync for n=1:3 println("$n") end
ERROR: syntax: invalid let syntax
Stacktrace:
 [1] eval at ./boot.jl:319 [inlined]
 [2] #337 at /home/matt/.julia/packages/Atom/jodeb/src/repl.jl:129 [inlined]
 [3] with_logstate(::getfield(Main, Symbol("##337#339")), ::Base.CoreLogging.LogState) at ./logging.jl:397
 [4] with_logger(::Function, ::Atom.Progress.JunoProgressLogger) at ./logging.jl:493
 [5] top-level scope at /home/matt/.julia/packages/Atom/jodeb/src/repl.jl:128

If I remove the @sync from my parsync macro, namely:

julia> macro parsync(args...)
           :(@distributed $(args...))
       end;

julia> @parsync for n=1:3 println("$n") end
1
2
Task (runnable) @0x00007fb59a78f4903

If I use @sync outside of my macro it works with the raw @distributed macro

julia> @sync @distributed for n=1:3 println("$n") end
1
2
3
Task (done) @0x00007fb599774fd0

If @parsync only calls @distributed without the @sync, then using @sync with @parsync appears to have no effect on @parsync namely,

julia> @sync @parsync for n=1:3 println("$n") end
1
2
Task (runnable) @0x00007fb5997792703

Number n=3 is never printed.

I'm trying to create some universal macros I can use that depend on the version number of Julia, but this issue is blocking me for my modules containing parallel tasks.

[mattcbro]

The error here seems to be how @sync creates a temporary gensym'd variable. When called from a macro it seems to generate some extra annotations that fail to compile. For example here is a normal use of @sync, not called from a macro:

@macroexpand @sync @distributed for n=1:3 println("$n") end

This generates the following:

quote
    #= task.jl:246 =#
    let ##sync#72 = (Base.Any)[]
        #= task.jl:247 =#
        #10#v = begin
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:337 =#
                local #11#ref = (Distributed.pfor)(begin
                                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:289 =#
                                function (#12#R, #13#lo::Distributed.Int, #14#hi::Distributed.Int)
                                    #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:290 =#
                                    for n = #12#R[#13#lo:#14#hi]
                                        #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:291 =#
                                        begin
                                            #= none:1 =#
                                            println("$(n)")
                                        end
                                    end
                                end
                            end, 1:3)
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:338 =#
                if $(Expr(:isdefined, Symbol("##sync#72")))
                    #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:339 =#
                    (Distributed.push!)(##sync#72, #11#ref)
                end
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:341 =#
                #11#ref
            end
        #= task.jl:248 =#
        (Base.sync_end)(##sync#72)
        #= task.jl:249 =#
        #10#v
    end
end

Here is what it looks like called through @parsync

julia> @macroexpand @parsync for n=1:3 println("$n") end
quote
    #= task.jl:246 =#
    let Main.:(##sync#72) = (Base.Any)[]
        #= task.jl:247 =#
        #18#v = begin
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:337 =#
                local #19#ref = (Distributed.pfor)(begin
                                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:289 =#
                                function (#20#R, #21#lo::Distributed.Int, #22#hi::Distributed.Int)
                                    #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:290 =#
                                    for Main.n = #20#R[#21#lo:#22#hi]
                                        #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:291 =#
                                        begin
                                            #= none:1 =#
                                            (Main.println)("$(Main.n)")
                                        end
                                    end
                                end
                            end, 1:3)
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:338 =#
                if $(Expr(:isdefined, :(Main.:(##sync#72))))
                    #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:339 =#
                    (Distributed.push!)(Main.:(##sync#72), #19#ref)
                end
                #= /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Distributed/src/macros.jl:341 =#
                #19#ref
            end
        #= task.jl:248 =#
        (Base.sync_end)(Main.:(##sync#72))
        #= task.jl:249 =#
        #18#v
    end
end

I think that the statement
let Main.:(##sync#72) = (Base.Any)[]
generates an invalid let syntax error, but I'm not entirely certain.

[bojeryd91]

I am having the same issue, running the exact code as OP except my macro was called @fortype. My version of Julia is 1.6.0

[abraemer]

I think I discovered a workaround (Julia 1.7.1)

Consider the (rather pointless) macro:

julia> macro defer(ex)
	:(@sync Threads.@spawn $(ex))
end

When you try to call it, you'll see the same error as above:

julia> @defer println(Threads.threadid())
ERROR: syntax: invalid let syntax around task.jl:398

To fix this, you have to disable the macro hygiene by wrapping the expression in the macro in esc.

julia> macro deferfix(ex)
	esc(:(@sync Threads.@spawn $(ex)))
end
julia> @deferfix println(Threads.threadid())
1
Task (done) @0x00007ff62771d5a0

This probably also tells us that somehow the macro sanitization procedure is to blame.

Sidenote: Interestingly doing the above in Pluto.jl shows a different error: UndefVarError: ##sync#41#984 not defined but that's probably not so relevant. It's probably because it evaluates expression in a different context.

[charleskawczynski]

@ViralBShah, what PR closed this? I think I may be running into this issue here: CliMA/ClimaComms.jl#64. The suggestion above:

macro deferfix(ex)
	esc(:(@sync Threads.@spawn $(ex)))
end

is not hygienic, so it's not really a safe recommendation.

[vtjnash]

The hygienic version of that is simply:

macro deferfix(ex)
	esc(:($Base.@sync $Threads.@spawn $(ex)))
end

Where each macro is explicitly qualified with a module name and the outer macro is wrapped in esc()

[vtjnash]

The alternative "proper" version of this is:

macro deferfix(ex)
    :(@sync @spawn $(esc(ex))))
end

but many macros (include macroexpand itself) handle this form of expression incorrectly, resulting in the invalid let syntax error above

[charleskawczynski]

The

macro deferfix(ex)
	esc(:($Base.@sync $Threads.@spawn $(ex)))
end

pattern worked, thanks!