How was max_nfev = 2000 * (nvarys + 1) chosen?

[jagerber48]

Seems like it is ancient history now, but how was max_nfev = 2000 * (nvarys + 1) chosen? I'm most familiar with least_squares. Looking at the documentation it looks like least_squares uses 100 * n or 100 * n * (n+1) depending on some settings. The old leastsq had 200*n. I guess it was empirically determined that more fits could succeed if you just bump up max_nfev by ~10x?

[newville]

@jagerber48 yeah. The "200*(n_variables+1)" in leastsq is probably taken directly from lmdif1.f (https://netlib.org/minpack/lmdif1.f) the "simplified wrapper with some sane defaults" around the real lmdif.f from the MINPACK library. Which was written in 1980. I don't know what that means to you (I'm listening to Remain in Light), but that code was no doubt being tested on high-end PDP-11s with clock speeds well over 1 MHz and as much as 4 MB "core memory", that were meant to be time-shared with multiple users.

When max_nfev is reached, the fit stops and reports failure. So, "200*(n_variables+1)" is probably a reasonable estimate of "if it takes that many evaluations, something could be wrong", but that is not necessarily the same thing as saying "if it takes that many evaluations, there is an error". Yes, this is a default value that can be set by a user. We raised the default to avoid the infrequent-but-possible case where a fit converges only after many function evaluations.

Since the whole concept is "use Python instead of Fortran to do non-linear-least-squares fitting", we favor "easy/friendly for novices" over raw performance and don't "give up early to avoid using too many expensive CPU cycles and expect the programmer/analyst to fix initial values, recompile, and start over".

FWIW, I have a low opinion of scipy.optimize.least_squares. The observation that the default for max_nfev changes with different options is a good illustration of its problems. It also straight-up lies about the number of function evaluations.

[jagerber48]

Thank you very much for all of this information! This all makes sense.

Favoring "easy/friendly for novices" and then having user "opt in" to trying to improve their own performance by e.g. decreasing max_nfev makes sense.

The cause for my writing this question was that I was using curve_fit (wrapper for least_squares or leastsq I think) directly in some code on a certain model and it was surprisingly failing. However lmfit successfully fit it right away and I tracked the difference down to max_nfev. So in that case the large max_nfev definitely helped me. Bumping it up on curve_fit got it to fit there as well.

Interesting you have a low opinion of scipy.optimize.least_squares. It's the only optimization method I'm very familiar with, but that's just because it's the first one that comes up. I haven't dug into the others further. I'll have to do more research/comparisons of the various methods.

It also straight-up lies about the number of function evaluations.

Yikes.

[newville]

Setting max_nfev higher does not degrade performance. It just fails later. In your case, that increased performance ;).

scipy.optimize.least_squares is a case study of API "donts". The one-line description is

Solve a nonlinear least-squares problem with bounds on the variables.

1. If you use method='lm', bounds are ignored.
2. loss defaults to 'linear' (meaning "least-squares", which can certainly be confusing by itself) but can be set to 4 other values. Choosing any other value means that least_squares is then not solving a non-linear least-squares problem. Yep.

If you look at several of the other function arguments, the meaning and default value change when you change method. That inter-connected meaning of the function arguments makes least_squares very difficult to wrap in lmfit. It is really three different functions for three different solvers. We just ignore the deliberately incorrect value for "number of function evaluations" from least_squares and report the actual number of function evaluations.

Of course, lmfit supports bounds with all methods.