Warnings generated in calculation of covariance matrix

[eendebakpt]

When fitting a method with lmfit there are some warnings generated about an invalid sqrt. This happens in minimizer.py at (around line 852)

par.stderr = np.sqrt(self.result.covar[ivar, ivar])

The reason is that the covariance matrix (which should have a positive diagonal) has some negative entries.

I traced this down to the calculation in _calculate_covariance_matrix which is:

            Hfun = ndt.Hessian(self.penalty, step=1.e-4)
            hessian_ndt = Hfun(fvars)
            cov_x = inv(hessian_ndt) * 2.0

By changing the value of step to 1e-6 the covariance matrix calculations gives no warnings.

How can I prevent the warnings and have a correct calculation of the covariance matrix? The model I am using seems reasonable (e.g. no extreme values in the input data, smooth function).
Could the default value of 1e-4 be changed into a lower value, a value determined automatically or perhaps a value that can be configured?

Minimal example reproducing the warning:

import numpy as np
import lmfit

def rabi_model(x, offset: float, amplitude: float, centre: float, Omega: float):
    Omega2 = Omega * Omega
    angle = np.pi
    t = angle / Omega
    t_half = 0.5 * t
    Delta = x - centre
    s = Omega2 + Delta * Delta
    sin_value = np.sin(np.sqrt(s) * t_half)
    return offset + amplitude * (Omega2 / s) * sin_value * sin_value


x, y, initial_parameters = (
    np.array(
        [
            0.0,
            0.18181818,
            0.36363636,
            0.54545455,
            0.72727273,
            0.90909091,
            1.09090909,
            1.27272727,
            1.45454545,
            1.63636364,
            1.81818182,
            2.0,
            2.18181818,
            2.36363636,
            2.54545455,
            2.72727273,
            2.90909091,
            3.09090909,
            3.27272727,
            3.45454545,
            3.63636364,
            3.81818182,
            4.0,
        ]
    ),
    np.array(
        [
            0.06666667,
            0.16666667,
            0.16666667,
            0.1,
            0.03333333,
            0.06666667,
            0.13333333,
            0.33333333,
            0.4,
            0.66666667,
            0.66666667,
            0.9,
            0.9,
            0.73333333,
            0.43333333,
            0.2,
            0.06666667,
            0.1,
            0.06666667,
            0.0,
            0.1,
            0.03333333,
            0.13333333,
        ]
    ),
    {
        "offset": 0.49880894230131395,
        "amplitude": -39.96107188727934,
        "centre": -17.179417670379504,
        "Omega": 11.002660957177593,
    },
)
weights = np.array(
    [
        62.0,
        37.81223367,
        16.98743568,
        13.66578457,
        12.32509234,
        11.6788976,
        11.35965545,
        11.21199197,
        11.15406427,
        11.1377845,
        11.13559361,
        11.13578601,
        11.14039807,
        11.16368501,
        11.23126958,
        11.38142275,
        11.67148055,
        12.1964309,
        13.14187886,
        14.96671734,
        19.37741088,
        57.29542891,
        62.0,
    ]
)


method = "lbfgsb"
lmfit_model = lmfit.Model(rabi_model)
result = lmfit_model.fit(y, **initial_parameters, weights=weights, x=x, method=method)

print(result.best_values)
print(result.covar)

result.plot_fit(show_init=True)

Output:

{'offset': 0.4146331808834299, 'amplitude': -38.22890735611283, 'centre': -22.359527189650727, 'Omega': 13.994545464983691}
[[ 6.56914461e-03 -1.13124865e+00 -1.13341821e-01  6.57476155e-02]
 [-1.13124865e+00 -7.73416425e+03 -2.53547909e+03  1.46975103e+03]
 [-1.13341821e-01 -2.53547909e+03 -8.20231375e+02  4.75475574e+02]
 [ 6.57476155e-02  1.46975103e+03  4.75475574e+02 -2.75623595e+02]]
C:\develop\env310\lib\site-packages\lmfit\minimizer.py:859: RuntimeWarning: invalid value encountered in sqrt
  (par.stderr * np.sqrt(self.result.covar[jvar, jvar])))
C:\develop\env310\lib\site-packages\lmfit\minimizer.py:852: RuntimeWarning: invalid value encountered in sqrt
  par.stderr = np.sqrt(self.result.covar[ivar, ivar])

[newville]

@eendebakpt I guess we could make that step size an option. Another option would be to check whether the diagonal elements are positive and return None as would happen for some other failures of estimating the covariance with numdifftools.

It turns out that method='leastsq' does not give that error in the covariance. But the fits with both lbfgsb or leastsq are just terrible. When the fit is that terrible, it's going to be hard to draw much of a useful conclusion about the best-fit values or uncertainties. I'm not sure that is really an answer but I also don't think we're really guaranteeing that uncertainties can be estimated well for all fits.

[eendebakpt]

@newville The fit is indeed terrible (that is ok, I know how to deal with that), so the actual value of the covariance is not important for me. It is more the warnings that I would like to get rid of.

I did check the effect of the step parameter in this particular example. The diagonal elements of the covariance matrix as a function of step are:

Since the values are varying wildly, the I do not expect to find a "good" value of the step that will work better than the current 10e-4.
For a good fit of this model, the covariance is much better behaved.

I like your suggestion of returning None when of the diagonal elements is negative. If there are no objections, I can make a PR.

Code to reproduce the figure

import numpy as np
import lmfit

def rabi_model(x, offset: float, amplitude: float, centre: float, Omega: float):
    Omega2 = Omega * Omega
    angle = np.pi
    t = angle / Omega
    t_half = 0.5 * t
    Delta = x - centre
    s = Omega2 + Delta * Delta
    sin_value = np.sin(np.sqrt(s) * t_half)
    return offset + amplitude * (Omega2 / s) * sin_value * sin_value


x, y, initial_parameters = (
    np.array(
        [
            0.0,
            0.18181818,
            0.36363636,
            0.54545455,
            0.72727273,
            0.90909091,
            1.09090909,
            1.27272727,
            1.45454545,
            1.63636364,
            1.81818182,
            2.0,
            2.18181818,
            2.36363636,
            2.54545455,
            2.72727273,
            2.90909091,
            3.09090909,
            3.27272727,
            3.45454545,
            3.63636364,
            3.81818182,
            4.0,
        ]
    ),
    np.array(
        [
            0.06666667,
            0.16666667,
            0.16666667,
            0.1,
            0.03333333,
            0.06666667,
            0.13333333,
            0.33333333,
            0.4,
            0.66666667,
            0.66666667,
            0.9,
            0.9,
            0.73333333,
            0.43333333,
            0.2,
            0.06666667,
            0.1,
            0.06666667,
            0.0,
            0.1,
            0.03333333,
            0.13333333,
        ]
    ),
    {
        "offset": 0.49880894230131395,
        "amplitude": -39.96107188727934,
        "centre": -17.179417670379504,
        "Omega": 11.002660957177593,
    },
)
weights = np.array(
    [
        62.0,
        37.81223367,
        16.98743568,
        13.66578457,
        12.32509234,
        11.6788976,
        11.35965545,
        11.21199197,
        11.15406427,
        11.1377845,
        11.13559361,
        11.13578601,
        11.14039807,
        11.16368501,
        11.23126958,
        11.38142275,
        11.67148055,
        12.1964309,
        13.14187886,
        14.96671734,
        19.37741088,
        57.29542891,
        62.0,
    ]
)


method = "lbfgsb"
lmfit_model = lmfit.Model(rabi_model)
result = lmfit_model.fit(y, **initial_parameters, weights=weights, x=x, method=method)

print(result.best_values)
print(result.covar)

plt.figure(10); plt.clf();
result.plot_fit(show_init=True)

self=result

#%%
# code from result._calculate_covariance_matrix
from copy import deepcopy
from lmfit.minimizer import LinAlgError, ndt, inv
import matplotlib.pyplot as plt

fvars = np.array(list(result.best_values.values()))
nfev = deepcopy(self.result.nfev)
best_vals = self.result.params.valuesdict()

steps=np.logspace(-8, -4, 20, endpoint=True)
hh=[]
dd=[]
ee=[]
for idx, step in enumerate(steps):
    try:
        Hfun = ndt.Hessian(self.penalty, step=step)
        hessian_ndt = Hfun(fvars)
        cov_x = inv(hessian_ndt) * 2.0
        hh.append(np.diag(hessian_ndt))
        dd.append(np.diag(cov_x))
        ee.append(np.linalg.eigvals(hessian_ndt))
    except (LinAlgError, ValueError):
        hh.append([np.NaN]*4)
        dd.append([np.NaN]*4)
        ee.append([np.NaN]*4)


dd=np.array(dd)        
hh=np.array(hh)
ee=np.array(ee)

plt.figure(20); plt.clf()

for idx, key in enumerate(lmfit_model.param_names):
    plt.semilogx(steps, dd[:, idx], 'd--', label=f'Covariance: {key}' )
plt.yscale('symlog')
plt.xlabel('Step value')
plt.ylabel('Parameter covariance')
plt.title('Covariance')
plt.legend()
    
plt.figure(21); plt.clf()
for idx, key in enumerate(lmfit_model.param_names):
    plt.semilogx(steps, hh[:, idx], '.-', label=f'Hessian: {key}' )
plt.yscale('symlog')
plt.xlabel('Step value')
plt.ylabel('Hessian')
plt.title('Hessian ')
plt.legend()


plt.figure(22); plt.clf()
for idx, key in enumerate(lmfit_model.param_names):
    plt.semilogx(steps, ee[:, idx], '.-', label=f'{key}' )
plt.yscale('symlog')
plt.xlabel('Step value')
plt.ylabel('Eigenvalue')
plt.title('Eigenvalues of Hessian ')
plt.legend()

[reneeotten]

Personally I don't think anything needs to be changed. Fits that are not converged are not expected to have well-behaved covariance matrices. The warnings about this are appropriate here and we shouldn't suppress them.

If you want to get rid of the warnings, feel free to do so in your own code. I don't remember the exact syntax, bit you'll want to look at the "warnings" module that is part of the Python core library.

[newville]

@eendebakpt

The fit is indeed terrible (that is ok, I know how to deal with that),

Well, you are kind of raising this issue because you are not dealing with it, right?

We really cannot prevent someone from getting a terrible fit if that is their goal, and it is sort of pointless for us to worry about the results of bad fits.

so the actual value of the covariance is not important for me. It is more the warnings that I would like to get rid of.

If getting rid of the warning is the goal, then perhaps

import warnings
warnings.simplefilter("ignore", RuntimeWarning)

is what you are looking for. I cannot recommend that.

Anyway, I would be ok with leaving _calculate_covariance_matrix() as it is or alerting to have something like:

        try:
            Hfun = ndt.Hessian(self.penalty, step=1.e-4)
            hessian_ndt = Hfun(fvars)
            cov_x = inv(hessian_ndt) * 2.0
        except (LinAlgError, ValueError):
            nvar = len(fvars)
            cov_x = -np.ones(nvar*nvar, dtype=np.float64).reshape(nvar, nvar)
        finally:
            self.result.nfev = nfev

        if cov_x.diagonal().min() < 0:
            cov_x = None

        # restore original values
        for name in self.result.var_names:
            self.result.params[name].value = best_vals[name]
        return cov_x

That would make cov_x be None for your case.

[peendebak]

I created #813 based on the suggestions above.