Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add docstring for gradient #62

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

add docstring for gradient #62

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c94eb9c
add docstring for gradient
johnnychen94 Jun 4, 2021
bd6e134
add missing space to jldoctest output
johnnychen94 Jun 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions src/autodiff/gradfunc.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,41 @@ end
(~protectf(g).f)(args...; kwargs...)
end

"""
gradient(Val(iloss), f, args::Tuple; kwargs...)
gradient(f, args; iloss::Int, kwargs...)

Calculate the gradient of `f(args...; kwargs...)` for reversible function `f` with regard to
input `args`. The integer value `iloss` specifies the position of `loss` in `args`.

!!! note
`iloss=1` is specially optimized, so putting the loss as the first parameter can avoid potential overhead.

# Examples

```jldoctest; setup=:(using NiLang)
X = rand(2, 2)
grads = NiLang.AD.gradient(Val(1), i_norm2, (0.0, X))
grads[2] ≈ 2 .* X

#output
true
```

Note that `gradient` calculation is disabled for container with integers:
Copy link
Contributor Author

@johnnychen94 johnnychen94 Jun 4, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why this is disabled. Is it possible to return nothing for it? Otherwise it can very easily be a root of potential bugs.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NiLang assumes gradient type == value type. Integers can be differentiated manually by wrapping GVar. Also, I assume in most cases, people do not want to differentiate them.
Wondering what are the potential bug?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not very easy to detect and understand that NiLang.AD.gradient(Val(1), i_sum, (0, X))[2] is zeros (maybe throw a warning for this?)

Another example comes from the Zygote example, and it's quite confusing from the error stack:

julia> norm2'([1,2,3,4])
4-element Vector{Int64}:
 2
 4
 6
 8

julia> Zygote.@adjoint function norm2(x::AbstractArray{T}) where T
           out = norm2(x)
           out, δy -> (grad((~r_norm2)(GVar(out, δy), GVar(x))[2]),)
       end

julia> norm2'([1,2,3,4])
ERROR: MethodError: no method matching (::Inv{typeof(r_norm2)})(::GVar{Int64, Int64}, ::Vector{Int64})
Closest candidates are:
  (::Inv{typeof(r_norm2)})(::T, ::AbstractArray{T, N} where N) where T at REPL[8]:2
Stacktrace:
 [1] (::var"#10#11"{Vector{Int64}, Int64})(δy::Int64)
   @ Main ./REPL[11]:3
 [2] (::var"#138#back#12"{var"#10#11"{Vector{Int64}, Int64}})(Δ::Int64)
   @ Main ~/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
 [3] (::Zygote.var"#41#42"{var"#138#back#12"{var"#10#11"{Vector{Int64}, Int64}}})(Δ::Int64)
   @ Zygote ~/.julia/packages/Zygote/6HN9x/src/compiler/interface.jl:41

Copy link
Owner

@GiggleLiu GiggleLiu Jun 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is how you can circumvent this issue:

julia> Zygote.@adjoint function norm2(x::AbstractArray{T}) where T
               out = norm2(x)
               out, δy -> (grad((~r_norm2)(GVar(out, δy), GVar(x, zero(x)))[2]),)
          end

julia> norm2'([1,2,3,4])
4-element Vector{Int64}:
 2
 4
 6
 8

Does this make sense?


```jldoctest; setup=:(using NiLang)
X = rand(Int, 2, 2)
NiLang.AD.gradient(Val(1), i_sum, (0, X))[2]

# output
2×2 Matrix{Int64}:
0 0
0 0
```
"""
gradient

@generated function gradient(::Val{iloss}, f, args::NTuple{N,Any}; kwargs...) where {iloss,N}
newres = gensym()
newargs = Any[:(GVar($newres[$i])) for i=1:N]
Expand Down