Multip Processing

[pnachtwey]

I have a function with 15 parameters that I am fitting 3040 points and sometimes even more. It takes about 0.5 seconds for each function evaluation. Python now makes it easy to do multi-processing so all the evaluations to compute the Jacobian can be spawned at the same time and then 'joined'. Most modern CPUs have 16 threads so this technique would speed up computing the Jacobian significantly.

I have implemented Levenberg-Marquardt in our auto tuning product. The original code was modified from Argonne Labs Fortran version. I am looking a the Jacobian now. It doesn't seem like it would be too much work for a "HUGE" speed up if Python's multi processing overhead is too bad.

BTW, I have tried all the other 'methods' and none work as well as lmfit.

[newville]

@pnachtwey I think that no one would object to having individual fits use multiple cores. I tried something like this many (~10) years ago, and it did not actually make much of an improvement, but I am completely willing to say that it could definitely give a very large improvement. I am not aware of anyone doing that with Python, though I can certainly believe someone has, and it sort of sounds like you have some information on this.

I think that using Python multiprocessing for fitting is Not Easy (TM), and doing it at the highest level from Python/Lmfit might be challenging. It may be that the latest "multiple interpreters" effort (maybe coming in 3.12 -- I am not sure) would help.

If you have a modified version of MINPACK that supports multi-core from the Fortran level, that would definitely be worth exploring. I might suggest that is most suitable for scipy. But we might be willing to try that out here too.

Anyway, if you are interested in pursuing this, I would definitely encourage that, and let us know how we might be able to help.

[pnachtwey]

Python 3.11 has made multiprocessing easier.
https://docs.python.org/3/library/multiprocessing.html
The new python3.11 has a join() function that waits until all the evaluations are complete.
To explain more clearly I download some code, lmfit-6.2.tgz, that we used only we used a much older version , lmfit-2.1.gz, 20+ years ago but this part of the code looks about the same. It is the part that calculated the Jacobian so the LM algorithm can compute which direction to step next. Notice there is an indirect call to evaluate. This is where LM takes a small step in each direction to calculate the Jacobian and then the direction down hill ( to a smaller sum of squared errors ).
I recently posted a problem where I was minimizing 15 parameters. Each evaluation required about 0.5 seconds on my old 2.5MHz computer. Why can't all the evaluations in this loop be done in parallel? Each evaluation is independent of the other.

The Jacobian part isn't big.
[code]
for (j = 0; j < n; j++) {
temp = x[j];
step = MAX(epseps, eps * fabs(temp));
x[j] += step; / replace temporarily /
(evaluate)( x, m, data, wf, &(S->userbreak) );
++(S->nfev);
if ( S->userbreak )
goto terminate;
for (i = 0; i < m; i++)
fjac[jm+i] = (wf[i] - fvec[i]) / step;
x[j] = temp; / restore /
}
if ( C->verbosity >6 ) {
/ print the entire matrix /
printf("\nlmmin Jacobian\n");
for (i = 0; i < m; i++) {
printf(" ");
for (j = 0; j < n; j++)
printf("%.5e ", fjac[jm+i]);
printf("\n");
}
}
[/code]

What would be better yet is to have a standardized call to the parallel processing Jacobian function so that other minimizing algorithms can use it.

[pnachtwey]

I don't believe it [code] does not format code.
If I click on edit the code is formatted.