Add ELU, PReLU, ThresholdedReLU layers #32
Conversation
src/nn/Modules/Activations.cpp
Outdated
| { | ||
| auto tmp = max(input, 0.0); | ||
| auto res = expandAs(m_parameters[0],tmp) * tmp; | ||
| //TODO: Determine if doing the max after the mul is preferable |
There was a problem hiding this comment.
That would be two different behaviors. max after multiply will have a different behavior when values of input and m_parameters are both negative. Can you double check what the exact behavior should be.
src/nn/Modules/Activations.cpp
Outdated
| { | ||
| auto mask = input > m_threshold; | ||
| return input * mask; | ||
| } |
There was a problem hiding this comment.
This should probably just be renamed to Threshold ?
There was a problem hiding this comment.
I believe Threshold may be misleading, since threshold usually denotes a function R -> {0,1} (e.g., return input > m_threshold;). ThresholdedReLU or ThresholdReLU seem a better fit to me.
There was a problem hiding this comment.
ThresholdReLU sounds better. Thanks @FloopCZ
src/autograd/Functions.cpp
Outdated
| }; | ||
| return Variable(result, {lhs, rhs, mask}, grad_func); | ||
| } | ||
|
|
There was a problem hiding this comment.
This function should be indented by four spaces.
src/nn/Modules/Activations.cpp
Outdated
| auto res = max(input, m_alpha * (exp(input) - 1)); | ||
| return res; | ||
| auto mask = input >= 0.0; | ||
| return (mask * input) + (!mask * m_alpha * exp(input)); |
There was a problem hiding this comment.
Shouldn't this be (!mask * m_alpha * (exp(input) - 1)) ?
https://keras.io/layers/advanced-activations/#elu
fixed PReLU, ELU forward methods, mix name changes
This is based off of the Keras documentation here The PReLU layer in Keras supports more features that we can discuss adding in the future. (e.g. weight initialization function, regularization).