Step 3.3: Test of xRFI

[steven-murray]

This test is the first explicit test of xRFI. The idea is to use hera_sim to add RFI to a fiducial simulation including EoR and FG, and then use xRFI to flag it, and then use pspec to determine a power spectrum after flagging.

• Simulation Component: GSM, GLEAM, EoR (powerlaw), gains, RFI
• Simulators: hera_sim, RIMEz
• Pipeline Components: redcal, smoothcal, xRFI, pspec`
• Depends on: Step 2.1 #16

Why this test is required

#21 and #22 check the performance of pspec with realistic flags (more realistic than this will generate), thus this is not a particularly good test of how pspec deals with flags. What it adds is two things: 1) how well xRFI does in terms of accuracy of removing RFI, and 2) how much the inaccuracy propagates through to the power spectrum.

Summary

A brief step-by-step description of the proposed test follows:

• Obtain FG and EoR visibilities in the same way as Step 2.1 #16.
• Add gains in the same way as Step 2.1 #16.
• Use hera_sim.add_rfi to add RFI visibilities (and save pure-RFI visibilities as well)
• Run redcal and smoothcal.
• Run xRFI
• Look at accuracy of xRFI
• Run @nkern's post-processing and in-painting.
• Run pspec

Simulation Details

• Freq. range:
• Channel width:
• Baseline/antenna configuration:
• Total integration time:
• Number of realisations:

Criteria for Success

• P(k) matches known input to within 1%

[r-pascua]

@lwhitler to chat with @adampbeardsley to figure out how to fix bug with xrfi (xrfi has no check attribute) and to determine proper use of noise--does xrfi expect a realistic level of noise, or does it just need enough noise to have finite chi-squared values?

[adampbeardsley]

What do you mean check attribute?

xrfi uses the noise to quantify the outlier-ness of data points. Having no noise will result in divisions by zero. Have unrealistic noise will result in unrealistic flagging. But if you abstract the performance in terms of SNR, I suppose any level should be ok. Of course the detrending of the actual signal isn't going to be perfect, so there will be some mutual coupling between noise and signal level, but I think that's second order.

[r-pascua]

@lwhitler can you post a screenshot of the error you showed in the telecon?

[lwhitler]

@adampbeardsley

If run_check=True in xrfi.xrfi_run(), this happens:

And run_check=False is a different error:

[adampbeardsley]

Ah, this is an pyuvdata version issue. If you update pyuvdata (may need to do so from source, I'm not sure if it's in a release yet), you should be good to go.