First attempt at ReLU

[plavin]

I'm getting a segfault when I run this, haven't figured out why yet. I'll look more later.

Error:

libc++abi.dylib: terminating with uncaught exception of type af::exception: ArrayFire Exception (Input types are not the same:205):
In function af_err af_matmul(af_array *, const af_array, const af_array, const af_mat_prop, const af_mat_prop)
In file src/api/c/blas.cpp:133
Type mismatch inputs

In function af::array af::matmulNT(const af::array &, const af::array &)
In file src/api/cpp/blas.cpp:28
Abort trap: 6

[pavanky]

Can you send this in a different PR.

[pavanky]

Or edit perceptron.cpp to use ReLU instead of sigmoid.

[pavanky]

Is this necessary ? If it is use #include <cfloat>

[pavanky]

Indentation

[pavanky]

I don't think this is right. May be this function doesnt need to have a gradient?

[pavanky]

If you are removing grad_func, then return Variable(result, false); This tells the backward pass that this variable is not differentiable.

[pavanky]

I think this can be simplified like this:

Variable relu(const Variable &input)
          {
            auto mask = input > 0;
  	    auto result = mask.array() * input.array();
  	    auto grad_func = [](std::vector<Variable> &inputs, const Variable &grad_output) {
  	      inputs[0].addGrad(grad_output * inputs[1]);
              };
              return Variable(result, {input, mask}, grad_func);
          }

[plavin]

What is inputs[1]?

[pavanky]

You are sending mask as second input for backward pass. You can use that instead of comparing inputs[0] with 0 again

[pavanky]

You are sending mask as second input for backward pass. You can use that instead of comparing inputs[0] with 0 again

[pavanky]

👍

[pavanky]

@plavin Instead of having a autograd::relu function, it is better to implement an autograd::max function.

You can then call autograd::max(var, 0) from nn::Relu::forward