Fully automated AA with PyTorch

[wangzcl]

Now we have softmax function to approximately "project" arrays on to the unit simplex, and there is also an entropic descent method which use softmax perform optimization on the unit simplex (https://ieeexplore.ieee.org/abstract/document/10213413).

However, I still feel there is a more straightforward way to use softmax, that is, integrating it into the loss function like what people do to many neural networks. Define the loss function as

$f=\lVert X - \sigma(A')\sigma(B')X\rVert$,

where $\sigma$ is the row-wise softmax function. So $A = \sigma(A'), B = \sigma(B')$, and we only need to optimize $A'$ and $B'$.

What we need to do is just defining the loss function in PyTorch, creating an optimizer object with $A$ and $B$, and PyTorch can do everything else. This would enable us to use some adavanced optimizers like SGD and adam to avoid local minima, and optimize both $A$ and $B$ simultaneously and automatically. It could be especially suitable for large datasets.

[aleixalcacer]

If I’m not mistaken, you’re suggesting introducing the softmax inside the loss function. Right?

Feel free to create a PR and see if it works :)

What do you think?

[wangzcl]

Yes. I would try that in the near future.

[aleixalcacer]

On the other hand, I should update the numpy biaa to the new API. Right?

If so, I’ll create a PR and would appreciate it if you could review it.

[wangzcl]

Happy to do so :)

[aleixalcacer]

Another thing: When the library is mature, I was thinking of writing an article about it and submitting it to a journal. Would you like to join?

[wangzcl]

That would be awesome. If you need anything that I can help with, just let me know