Generative Meta-Learning for Large-Scale Non-Convex Optimization (RL)

[kayuksel]

Hello!

You can find it here: https://github.com/kayuksel/generative-opt

Just change the following lines for combinatorial optimization.

I also implemented Fast CMA-ES and Tabu Search in PyTorch.

Here is Fast CMA-ES: https://github.com/kayuksel/torch-tsp-es/

Let me know if Tabu Search would also be helpful, I can share.

Please to don't forget to contribute back, and cite when possible.

Sincerely,
Kamer

class Generator(nn.Module):
    def __init__(self, noise_dim = 0):
        super(Generator, self).__init__()
        def block(in_feat, out_feat):
            return [nn.Linear(in_feat, out_feat), nn.Tanh()]
        self.model = nn.Sequential(
            *block(noise_dim+args.cnndim, 512), *block(512, 1024), nn.Linear(1024, len(assets)))
        init_weights(self)
        self.extract = Extractor(args.cnndim)
    def forward(self, x):
        mu = self.model(self.extract(x))
        return torch.bernoulli(mu.sigmoid())

actor = Generator(args.noise).to(device)
opt = torch.optim.AdamW(filter(lambda p: p.requires_grad, actor.parameters()), lr=1e-3)

best_reward = None

for epoch in range(args.iter):
    torch.cuda.empty_cache()
    weights = actor(torch.randn((args.batch, args.noise)).to(device))
    weights = weights / weights.sum(dim=1).reshape(-1, 1)

    loss = calculate_reward(weights.clone(), valid_data[:-test_size], index[:-test_size], True)
    opt.zero_grad()
    loss.mean().backward()
    nn.utils.clip_grad_norm_(actor.parameters(), 1.0)
    opt.step()

    with torch.no_grad():
        weights = weights[rewards.argmin()]
        test_reward = calculate_reward(weights.unsqueeze(0), 
            valid_data[-test_size:], index[-test_size:])[0]

[shixun404]

@kayuksel Thank you for bringing this to our attention, Kamer. I appreciate you taking the time to share the suggestions and links to your implementation of Fast CMA-ES and Tabu Search in PyTorch.

We would be delighted to have your contributions in the form of a pull request. It would be great if you could submit a pull request for the suggested modifications, this way we can easily review your code and merge it into our repository.

Additionally, we will check out the links you shared for Fast CMA-ES and Tabu Search and consider integrating them into the project as well.

If you have any questions or need help with submitting a pull request, please don't hesitate to reach out.

I will make sure to credit your contributions, and let you know if any issues arise or if we have any questions.

Thank you again for your help.

Sincerely,
Shixun

[YangletLiu]

Portfolio optimization with integer constraint would be an interesting case, to be included in RLSolver.
We expect to collect good demos for community users.

[kayuksel]

Tabu Search can probably allow integer constraints. It works by increasing or decreasing integer stocks by one at each iteration.

[kayuksel]

Comparison of Optimizers

OnePlusOne and its variants are performing very well. This is similar to what I've implemented in parallel as Tabu Search.
Let me know if you would like to collaborate on that. We can also try integer constraints (to invest in non-fractional shares).

[YangletLiu]

Comparison of Optimizers

OnePlusOne and its variants are performing very well. This is similar to what I've implemented in parallel as Tabu Search. Let me know if you would like to collaborate on that. We can also try integer constraints (to invest in non-fractional shares).

Dear @kayuksel , your proposal is great! This "optimization solver" (RL as one tool) project would like to consider general optimization problems and search methods (including Tabu search and other optimizers).

Integer constraints (to invest in non-fractional shares) in portfolio allocation/management is one typical example/application to include. Would be happy to work with you, and learn from you.

[kayuksel]

Thanks!

Here is the Tabu Search stuff that I've done previously (incl. Probabilistic Sharpe Ratio in PyTorch): tabu-search.zip

I have also been working on a new loss function that considers eigenvector entropy or gini-index of the portfolio.

This increments and decrements stocks in discrete steps, so we can easily convert that to integer portfolio allocation.
One just needs to use the price vector to calculate the portfolio weights from those integer-stock vectors of portfolios.

We should probably also check how OnePlusOne variants are implemented in Nevergrad as they seem effective.
I also have an Augmented Random Search (continuous) implementation here: https://github.com/kayuksel/braxars

[YangletLiu]

Maybe @shixun404 would run your codes and see whether it is possible to borrow some ideas.

[kayuksel]

Sounds good. I've also seen one of the targeted use-case is compressive sensing. Here is an application of Gen-Meta to low-rank matrix factorization (500K parameters) for creating a recommendation system on MovieLens 1M dataset, which may be relevant.