discrepancy between specsim.simulator and CMX spectra

[changhoonhahn]

I noticed that the CMX spectra from mini-SV2 and SV0 were noisier than the simulated spectra I was getting from a BGS version of specsim. As a test, in this jupyter notebook I compare a bright BGS galaxy spectrum from CMX to a simulated spectrum constructed using specsim.simulator.Simulator.

To reduce the number of moving parts, I smooth the CMX spectrum and use it as the source flux (black dashed). I also use the actual sky surface brightness from CMX for the surface brightness [^1]. The simulated spectrum is substantially less noisy and has much lower inverse variances.

tl;dr: The spectra I get from specsim.simulator.Simulator is substantially less noisy than observed spectra from CMX.

[^1] @dkirkby pointed out that the way I correct for flux calibration in the raw sky data overestimates the sky flux because it corrects for fiberloss. I didn't correct this since a fainter sky would make the problem worse.

[julienguy]

Interesting. A possibility is that the particular fiber you are looking at is on average not well positioned on its target. Can you compare the value of FIBERFLUX_R in the fibermap with the integral of the calibrated spectrum?

[dkirkby]

Thanks for documenting this nice test @changhoonhahn. I am looking into this now...

[changhoonhahn]

@julienguy @dkirkby I found the issue for all the galaxies spectra that I compared to so I don't think it's caused by a particular fiber.

I've updated the notebook to print out the specific fiber number and also compared a second galaxy spectrum.

[dkirkby]

I think I have finally gotten to the bottom of this. The discrepancy you found is because specsim assumes a nominal transparency and fiber acceptance fraction that is significantly higher than actually achieved in this exposure. As a result, the relative contribution of the signal shot noise is much larger than specsim predicts.

Here is a summary plot for the TARGETID 35185736613364265 on EXPID 52113 of TILEID 70502 from 20200225 that you studied:

The cyan curve shows the predicted signal (in flatfielded electrons / bin) for each camera, assuming 100% transparency and fiber fraction. The actual (smoothed) signal + sky is shown as the blue / red / black curves. Comparing these, the actual r-band TRANSP x FFRAC is only 4 %. The PSF-like sources in this exposure have r-band TRANSP x FFRAC ~ 13%, so the large size of this galaxy is playing a role here too.

[changhoonhahn]

Hmm, I didn't realize transparency would be such a large effect!

Regarding the fiber acceptance fraction, in my example, I used the smoothed spectra as the source. In that case, FFRAC should not play a role since the source signal is already scaled down --- right?

To account for transparency, should I be scaling my exposure times by a factor of (transp/1)^2 as you did for your depth calculations?

[dkirkby]

FFRAC still plays a role since the CFRAME spectrum is corrected for the actual fiber acceptance measured with stars in this exposure. Specsim then applies the nominal fiber acceptance for 1.1" seeing, which could be more or less than the actual value. The bigger effect here is probably the large source size, which decreases the FFRAC for his target relative to a PSF-like source.

[changhoonhahn]

I updated the comparison to include TRANSP and FFRAC, estimated from the GFA data, in the fiber_acceptance_fraction of specsim.Simulator.simulate. Also, instead of the incorrectly flux calibrated sky, I included sky brightness derived from converting the throughput corrected sky data.

Now specsim produces spectra much closer to the actual CMX observations:

Still, the inverse variance of the specsim spectrum doesn't agree very well for r and z spectrograph:

I suspect the throughput corrected sky brightness (blue), which by eye looks like it is overestimated in r and z:

Nominal dark sky included for comparison.

@dkirkby I derived the sky brightnesses as (see notebook for details):

(sky surface brightness) = (sky data)/(throughput)/(photons per bin)/(exp. time)/(fiber area)

Is this consistent with how you got the sky brightness in the comparison you had in your survey planning talk during the sci-com telecon?

[dkirkby]

I don't see anything with units of energy in your formula. Here is the code I use:

def convert_sky_rate(inc):
    # The incident rate is in phot/sec per 0.8A wavelength bin.
    # Convert to surface brightness units (sbunit) suitable as input to specsim.
    wave = inc.wave * u.Angstrom
    dwave = np.gradient(wave)
    ephot = (h * c / wave / u.ph).to(u.erg / u.ph)
    etendue = (DESI.instrument.effective_area * DESI.fiber_area.mean()).to(u.cm ** 2 * u.arcsec ** 2)
    # Calculate surface brightness on the 0.8A reduction grid.
    surfbrightness = ((inc.flux * u.ph / u.s) / dwave * ephot / etendue).to(sbunit)
    # Upsample to 0.1A specsim simulation grid. This means that resolution will be reapplied by specsim, but
    # should at least give the correct normalization and smoothed shape. Because we are upsampling a density,
    # no rescaling for the new bin size is required. The result will be padded to the wider simulation grid
    # by repeating the first and last values.
    return np.interp(simwave, inc.wave, surfbrightness.data) * sbunit

The input inc has attributes wave and flux with throughput corrections already applied (and coadded over the cameras). This gives the following sky surface brightness suitable for input to specsim:

[changhoonhahn]

@dkirkby Thanks! Turns out it was a bug in my code with the wavelength binning.

I reran the test with the updated sky surface brightness and now there's better agreement between the specsim output and the CMX data overall:

There's still some discrepancy, specsim slightly underestimates the noise for the b spectrograph and overestimates the noise for the z spectrograph. I'm not sure if discrepancies on these levels are worth the scrutiny.

[dkirkby]

Thanks for the update @changhoonhahn. We should still be able to achieve better agreement than this, at least for the FRAME ivar (which should dominate and is what specsim actually calculates), but is this good enough for your studies?

[changhoonhahn]

Here's a comparison with the FRAME ivar:

There are still some significant discrepancies.

I'm currently quantifying how much this will impact redshift success rates, which is what I'm ultimately interested in estimating. Do you have any suggestions on what other factors may be causing this discrepancy?

[dkirkby]

Is this for the same bright TARGETID 35185736613364265? What are you assuming for the throughput (transparency x fiberfrac) for this target? This doesn't look right.