Initial point samplers

[uri-granta]

Related issue(s)/PRs:

Summary

Add support for custom samplers for generating the optimization initial point candidates. This solves two problems:

1. It allows including pre-computed points among the candidates
2. It allows batching the random sampling to avoid running out of memory for high dimensional problems.

Fully backwards compatible: yes

Given how often generate_continuous_optimizer is used (and often with keyword arguments) I think we should make an extra effort to maintain backwards compatibility here. The current approach is simply to expand num_initial_samples so it can take either an int or a sampler. An alternate approach would be to add an additional optional initial_point_sampler parameter, but the downside of that is that it wouldn't be possible to catch cases where people accidentally pass in both a num_initial_samples and a initial_sampler.

PR checklist

• The quality checks are all passing
• The bug case / new feature is covered by tests
• Any new features are well-documented (in docstrings or notebooks)

[hstojic]

this can work as a short term solution, but we would want to make this more flexible - we could pass generators of initial samples to the optimizer, and provide a nice function/class for random sample generators - but this would then allow us to pass other types of generators when needed, e.g. qp generated by some other optimization process which would provide better starting points

[uri-granta]

this can work as a short term solution, but we would want to make this more flexible - we could pass generators of initial samples to the optimizer, and provide a nice function/class for random sample generators - but this would then allow us to pass other types of generators when needed, e.g. qp generated by some other optimization process which would provide better starting points

Is generators definitely the way to go here? Space.sample doesn't currently support this, and I worry that a generator giving 10,000 samples one at a time would be much less efficient than a single call to sample(10_000).

Another approach would be to provide an additional optional argument initial_sampler: Callable[[int], TensorType] that lets you specify a function that returns a given number of initial sample points. The default behaviour would be equivalent to initial_sampler = space.sample but we could provide a helper function so people could write something like initial_sampler = select_from_samples(generator) if they want to. This would work nicely with split_initial_samples, and (more importantly) not break the current generate_continuous_optimizer API.

[hstojic]

this can work as a short term solution, but we would want to make this more flexible - we could pass generators of initial samples to the optimizer, and provide a nice function/class for random sample generators - but this would then allow us to pass other types of generators when needed, e.g. qp generated by some other optimization process which would provide better starting points

Is generators definitely the way to go here? Space.sample doesn't currently support this, and I worry that a generator giving 10,000 samples one at a time would be much less efficient than a single call to sample(10_000).

Another approach would be to provide an additional optional argument initial_sampler: Callable[[int], TensorType] that lets you specify a function that returns a given number of initial sample points. The default behaviour would be equivalent to initial_sampler = space.sample but we could provide a helper function so people could write something like initial_sampler = select_from_samples(generator) if they want to. This would work nicely with split_initial_samples, and (more importantly) not break the current generate_continuous_optimizer API.

it doesn't have to be generators, that's just what seemed like what could allow us to pass a variety of initial samples, but
initial_sampler could do the trick - though, it would perhaps need to be a list of samplers, e.g. we would want to have some random samples and some pre-optimised initial points

@vpicheny any thoughts?

[uri-granta]

I've added something that I think is flexible enough but still easy to use. Opinions?

[hstojic]

left a few comments for improvement, but overall looks much better, I think this would fit the bill - with a sampler I can create a custom one where I mix random samples with points obtained in other ways to provide a better start to the optimiser

[hstojic]

should we also do a basic check that dimensionality is correct?

[hstojic]

I'm thinking that with initial samplers we can simplify arguments here, i.e. we could subsume split_initial_samples and num_initial_samples within the samplers, then each call of the sampler gives a splitted sample and we call them until they are exhausted, what do you think?
it would be a breaking change, but we are releasing 3.0 anyway :)

[uri-granta]

That is a much better idea! I've pushed a first attempt at this but will see what breaks and write some tests before asking for a review.

As an aside, it's actually possible to do this without breaking the interface by still letting people specify an int for num_initial_samples if they want to. There's an argument that we should rename the argument from num_initial_samples to initial_samples, but I'm worried that this would break a lot of code (and not just ours). Another option is to add a separate initial_samples argument and make people specify just one of this and num_initial_samples.

[hstojic]

great work! left a few minor comments, but I think this is it :)

it would still be good to pass it by @khurram-ghani as well before merging

p.s. this will make it way easier to test alternative generate_initial_points, e.g. with clustering to achieve better diversity of starting points between the runs

[hstojic]

you are using this space in multiple tests, perhaps create a constant at the top and reuse it? or a fixture?

[khurram-ghani]

One important comment I think (on -inf), and some minor ones.

[khurram-ghani]

It is a really good and flexible implementation!