Feature/relu #18
Feature/relu #18
Conversation
# Conflicts: # Cargo.lock
# Conflicts: # Cargo.lock
There was a problem hiding this comment.
Thanks for the PR, I really like the idea, however I'm wondering if we shouldn't go further and allow the user to provide their own activation functions by creating a trait that the user will implement and chose their own methods?
Also I can't merge in its current state because tanh and sigmoid should be separated in the compute method. Otherwise it will break for users.
| @@ -95,20 +95,20 @@ where | |||
| #[replace_numeric_literals(T::from(literal).unwrap())] | |||
| #[inline] | |||
| pub fn get_value(&mut self) -> T { | |||
| let update_gate = fast_sigmoid(self.update); | |||
| let reset_gate = fast_sigmoid(self.reset); | |||
| let current_memory = fast_tanh(self.input + self.memory * reset_gate); | |||
There was a problem hiding this comment.
We should have options for tanh,sigmoid and relu for all possible activations as it wil bias the output
Yeah that sound like a better idea. Make a NN generic over its activation function. I will try that later |
I finally had time to implement this. This time i implemented it as a feature, which should be nicer to use, since it allows users this:
neat-gru-rust = { version="1.1.1", features="relu"}but without it the tanh function will be used.