Learning Rate and Adaptive LR

[matte0fabr0]

Tweaking optimizer system prompt is tedious and not ideal if aiming for ease of use (particularly because various optimizer prompts are not immediately evident). Having some numeric learning rate parameter would be useful - either within the prompt, or multiple prompts, or other. This also allows for the development of an adaptive learning rate optimizer.

Analogy to traditional ML, currently only SGD and Momentum. Adaptive LR (AdaGrad) + Momentum = Adam

Just an idea! Thanks for this awesome framework.

[mertyg]

Thanks for the comment!! This is a very exciting idea we have also been thinking about -- currently we don't know what's the best way to do this, though. I'm sure in the near future we'll have analogs of learning rates / adaptive gradients!!

[den-run-ai]

I experimented with this a bit and came up with 2 possibilities:

• add instructions in the loss to limit the feedback to emulate gradual changes,
• limit the number of feedback / instructions generated in the loss/gradients.

There is probably an opportunity to define and encode a system prompt variable for this in the glossary.