(1) hoping you won't remove lmfit, and (2) seeking help re Geodesic acceleration

[donboyd5]

First, thanks for LsqFit and especially LsqFit.lmfit. I am finding them especially valuable.

I am not an expert in optimization or GitHub so if I am asking my questions the wrong way, please let me know.

I am using LsqFit.lmfit to solve an unconstrainted nonlinear least squares problem with the following key features in a typical instance:

• 1,000 parameters
• the problem involves several unchanging data items:
• ... a 40k x 20 Float64 matrix of data (xmat),
• ... a 40k x 1 Float64 vector of data (wh)
• ... a 1,000 element Float64 vector of what I call geotargets (the "y" data)
• the function f that produces the residuals (1) takes the parameters as an argument, (2) through a complicated highly nonlinear set of operations involving unchanging xmat, wh, and geotargets produces estimated values for geotargets (yhat, if you will), and (3) calculates the vector of residuals between these estimated y values (yhat) and the desired targets (geotargets) ; it is not terribly expensive to compute the function
• it is not practical to compute the analytic derivative of the function, but autodiff works well
• the jacobian is completely dense

The setup for lmfit is very simple:

LsqFit.lmfit(f, x0, Float64[]; autodiff=:forwarddiff, show_trace=true)

This seems to be faster and more robust than other implementations of the Levenberg-Marquardt algorithm that I have tried.

On my computer (AMD 5900X, 64gb RAM), this solves in about 80 minutes.

Question (comment) 1
While it is easy to set this up in lmfit, I am not sure it is easy (or desirable) to do so in curve_fit.

I saw here that at least at one point, you intended to remove lmfit. This is a plea not to do so, or if you do, I am hoping that you can post instructions on how to formulate a problem such as mine with curve_fit.

Question 2
Even though this is robust and acceptably fast, I would like to solve it more quickly if I can. From reading the documentation, Geodesic acceleration, if it can be used for this kind of problem, seems worth investigating. However, my multiple efforts to figure out how to define Avv! in a way that works all have failed. I have copied a failed version below, but it may not make much sense to look at because I think it involves a very incomplete understanding of how lmfit works, how Geodesic acceleration works, and of lmfit syntax. I am not sure what, in my problem, should correspond to xdata from the documentation example and I am not at all sure the approach can work in my implementation.

For example the following code generates an error suggesting that I have the wrong dimensions.

function Avv!(dir_deriv, p, v)
  for i=1:length(p)
      dir_deriv[i] = transpose(v) * Zygote.hessian(z -> f(z), p) * v
  end
end

LsqFit.lmfit(f, x0, Float64[]; avv! = Avv!, lambda=0, min_step_quality = 0, show_trace=true)

Would it be possible to tell me whether (1) Geodesic acceleration might be appropriate for a problem like mine, and if so, (2) how a function like Avv! should be defined for lmfit or, (3) alternatively, how I might set this up in curve_fit? I am happy to provide more details if needed.

Many thanks in advance.

[garrekstemo]

You might have a better chance of getting your question answered on the Julia Discourse. This is the main place for discussion about Julia and I've had my questions answered very quickly there.

[donboyd5]

Many thanks, @garrekstemo, I'll give it a go.