Bug in the flux-calibration variance calculation

[segasai]

Hi,

While preparing for #2419,
I've looked into fluxcalibration code and I believe the the current code that compute the ivar's of flux-calibrated frames is not quite correct:

desispec/py/desispec/fluxcalibration.py

Line 1478 in 255647f

frame.ivar[i,ok] = 1./( 1./(frame.ivar[i,ok]*C[i,ok]**2)+frame.flux[i,ok]**2/(fluxcalib.ivar[i,ok]*C[i,ok]**4) )

Specifically the current code uses this eqn:

$$Newivar = 1. / ( 1. /(iVar(F) C^2) + F^2 /(iVar(C) C^4))$$
where Ivar(C) is the inverse variance of fluxcalib vector, and ivar(F) is the inverse variance of spectra.
The problem here, is that it applies this eqn to already calibrated spectra (i..e original flux divided by C)
see

desispec/py/desispec/fluxcalibration.py

Line 1473 in 255647f

frame.flux = frame.flux * (C>0) / (C+(C==0))

While the equation only applies to original undivided flux.

Derivation:

$$F_1 = F/C$$

$$Var(F_1) = 1/C^2 Var(F) + F^2 Var(1/C) = 1/C^2 Var(F) + F^2 /C^4 Var(C)$$

$$iVar(F_1) = 1./(1/C^2 Var(F) + F^2 /C^4 Var(C)) = 1./(1/(C^2 iVar(F)) + F^2 /(C^4 iVar(C)) ) $$
This is exactly what the current code uses, but instead of F in the second term, it uses $F_1$.
Therefore this leads to incorrect C^2 factor in the second term.
Essentially the current code does:
$$iVar(F_1) = 1./(1/C^2 Var(F) + F^2 /C^4 Var(C)) = 1./(1/(C^2 iVar(F)) + F^2 /(C^6 IVar(C)) ) $$
A quick check shows the typical value of C is ~20-70, so that will lower the second term significantly, reducing the effect from flux-calibration error.

I am expecting that this will affect variances, where flux calibration uncertainty starts to become comparable to the spectrum uncertainty.
I don't know a typical S/N of flux calibration vectors a look at a few random files they can be pretty low, i.e. 20.

Am I missing something here ?
S

[segasai]

Assuming that I don't do something stupid here, the effect from this bug is massive I believe.
I.e. I compared the reduction with the #2422 fix and loa, and here's a S/N new vs S/N original

This is from /global/cfs/cdirs/desi/spectro/redux/koposov/fluxcalib_fix/exposures/

I.e. we are talking about factor of two decrease of S/N for high S/N pixels.

[segasai]

And here's the confirmation that the effect is real.
I took spectra- file (spectra-sv3-bright-15354.fits.gz from kibo ) and looked at the $\delta=\frac{S_1-S_2}{\sqrt{E_1^2+E_2^2}}$ for pairs of spectra vs S/N and one can see clearly that at high S/N the distributions broadens away from N(0,1), matching what's expected based on reasoning above.

[moustakas]

I.e. we are talking about factor of two decrease of S/N for high S/N pixels.

Just a quick comment that in FastSpecFit I add a 1% uncertainty floor in quadrature to the formal inverse variance because I was having trouble modeling super high signal-to-noise spectra. I wonder if this fix would make this step unnecessary.

[segasai]

For stars, our model spectra are "always" wrong, so I personally always ignored the increase in chi-square at high S/N, therefore I didn't notice this earlier.
But IMO the information content would change drastically with this and I believe it's relevant at S/N much lower than 100 (i.e. your 1% uncertainty floor). I.e. the example plot from above shows that S/N=20 could be S/N=10 (depending on the S/N of fluxcalib).

[julienguy]

I am afraid you are correct @segasai . The bug results in underestimating the contribution of the flux calibration statistical error to the calibrated flux variance. It affects the uncertainties of bright spectra. This is why we did not detect this so far in Lya P1D (where the noise was validated down to 1% of variance).

[araichoor]

kind of vaguely related: in Jan. 2023, I did look at sky fibers (in rosette-like observations of tertiary programs) and checked that the flux * sqrt(ivar) distributions where close to a N(0, 1) distributions; ie confirming that this bug shouldn t affect the low snr spectra.
here, and subsequent posts: https://desisurvey.slack.com/archives/C0351RV8CBE/p1673051607355669.

[segasai]

Yes, thanks, @araichoor , in MWS I also looked at low S/N often and I've been tracking the chi-square close to 1 there, so I think it's clear that low S/N is okay. But I'd expect to see differences from S/N=10-20. I don't have a good intuition how much higher the S/N in fluxcalib is in dark survey if that indeed leads to issues only at higher S/Ns -- that would explain the Lya results @julienguy mentioned.

[segasai]

BTW
I've checked the S/N of fluxcalib in the dark program, and across random set of exposures, I'm seeing median values of 20 (with tail to 40). That means dark program will equally experience this issue and a typical DESI pixel with S/N=20 has uncertainties overestimated by ~ 40%. That corresponds to roughly 5% error floor added in quadrature ( for a single exposure, but that will be reduced by sqrt(N) from stacking multiple exposures).