10/02/23 Meeting notes - Sun subtraction follow Up (pre-peeling)

[Victoria-Samboco]

According the meeting we had today, I will be pointing the progress of the Sun subtraction follow up for discussion, following the steps below:

1. After reshaping the MS
2. Image the dirt image of the Sun with Wsclean with -niter set to 0
3. Image again the Sun with a mask(converted from ds9 region to a fits mask with breizorro) and -niter different to zero. Deconvolution with wsclean in other to find a good model of the SUN.
4. Predict the SUN_MODEL_DATA
5. Subtract SUN_MODEL_DATA from the CORRECTED_DATA (re phased) with wsclean...

Q: Is there a way to specify that we want to subtract SUN_MODEL_DATA from CORRECTED_DATA if the -subtract-model parameter in wsclean use the default MODEL_DATA?

[o-smirnov]

Q: Is there a way to specify that we want to subtract SUN_MODEL_DATA from CORRECTED_DATA if the -subtract-model parameter in wsclean use the default MODEL_DATA?

No, wsclean will only subtract MODEL_DATA (and predict into MODEL_DATA).

To keep things clean, if you want to manipulate extra columns, I suggest you use the msutils.addcol, msutils.copycol, and taql_update cabs. Here are some examples:

https://github.com/o-smirnov/jovial/blob/reduce/jove-pol.yml#L335

So: image and deconvolve the Sun per-scan. Wsclean will leave a model for it in MODEL_DATA. Now you need to rephase that back to the original field centre (does chgcentre also update MODEL_DATA?) After rephasing, copy MODEL_DATA to SUN_MODEL_DATA. Then:

Experiment 1: subtract SUN_MODEL_DATA from CORRECTED_DATA (into a new column) and re-image that column -- hopefully the aliases are reduced/gone

Experiment 2: repeat selfcal on the field (without dEs), using MODEL_DATA for the field. Your QuartiCal model should look like MODEL_DATA:SUN_MODEL_DATA, and you should ask it to subtract direction 1. Hopefully, you get a better image of the field than in Experiment 1, because now selfcal is taking the Sun into account, and also subtracting it from the corrected data.

Experiment 3: repeat selfcal on the field with solvable dEs -- so now you're solving for a dE towards the Sun. Subtract direction 1. Hopefully, you get an even better image of the field.

@IanHeywood does that sound like a reasonable plan?

[IanHeywood]

Sounds good to me.

does chgcentre also update MODEL_DATA?

I believe so, but I'm not sure if it will handle custom columns (e.g. SUN_MODEL_DATA or custom columns that are produced as part of selfcal / peeling. We'll need to be careful with that. One possibility is to not have wsclean update any models (-no-update-model-required). After the imaging when the data are rephased, you can then just use the model images (of the Sun and the target field) to predict the required columns.

hopefully the aliases are reduced/gone

I think it's worth emphasising that the aliasing is an unexpected additional problem that we have encountered with this observation. Although it is definitely another aspect by which the Sun can have a detrimental effect on the science target, I think it's best if this is considered to be separate from the main issue, which is the removal of sidelobes from the Sun that are encroaching on the map centre. If the imager is setup correctly then aliasing shouldn't be a problem, whereas the main problem is where the R&D is at.

Experiment 2: repeat selfcal on the field (without dEs), using MODEL_DATA for the field.

Won't you have to re-predict the field centre model into MODEL_DATA prior to doing any selfcal? I think that column at this stage will contain the model for the Sun. Doing definitive predictions ourselves as part of the workflow and not relying on the imager (as I was suggesting as a possibilty at the top) might make the bookkeeping easier.

[Victoria-Samboco]

does chgcentre also update MODEL_DATA?

Yes, form the Wsclean documentation, as soon we are not specifying -datacolumn in chgcentre to only phase-rotate the visibilities in the given column, the columns DATA, MODEL_DATA and CORRECTED_DATA will all be processed if they exist. When a measurement set contains multiple data columns (e.g., DATA, MODEL_DATA, CORRECTED_DATA), each column will be updated (as long as they have a standard name).

[o-smirnov]

I believe so, but I'm not sure if it will handle custom columns (e.g. SUN_MODEL_DATA or custom columns that are produced as part of selfcal / peeling.

I'm pretty sure you can use wsclean to image any custom-named column. It's just the predict step that's hardwired for MODEL_DATA.

One possibility is to not have wsclean update any models (-no-update-model-required). After the imaging when the data are rephased, you can then just use the model images (of the Sun and the target field) to predict the required columns.

Hmmm, but when you -predict from a model image, does it not assume that the centre of the image corresponds to the current phase centre? I have a suspicion this is the case. If so, then we have to -predict the Sun using the rephased MSs, into MODEL_DATA, then phase-rotate them back to the field, then copy MODEL_DATA to SUN_MODEL_DATA, then -predict the field model into MODEL_DATA.

I think it's worth emphasising that the aliasing is an unexpected additional problem that we have encountered with this observation. Although it is definitely another aspect by which the Sun can have a detrimental effect on the science target, I think it's best if this is considered to be separate from the main issue, which is the removal of sidelobes from the Sun that are encroaching on the map centre.

Completely agree. But in this case the aliases are actually a useful diagnostic for us, because their (hoped-for) disappearance will be an additional indication of the Sun being subtracted correctly.

Won't you have to re-predict the field centre model into MODEL_DATA prior to doing any selfcal?

Yep, see above.

Doing definitive predictions ourselves as part of the workflow and not relying on the imager (as I was suggesting as a possibilty at the top) might make the bookkeeping easier.

Absolutely, I've tended to make the predict step separate in all my recipes now, precisely for that reason. Also makes it easier to resume a workflow from the predict step.

[IanHeywood]

I'm pretty sure you can use wsclean to image any custom-named column. It's just the predict step that's hardwired for MODEL_DATA.

Indeed, but the question/answer here was about chgcentre!

Hmmm, but when you -predict from a model image, does it not assume that the centre of the image corresponds to the current phase centre? I have a suspicion this is the case. If so, then we have to -predict the Sun using the rephased MSs, into MODEL_DATA, then phase-rotate them back to the field, then copy MODEL_DATA to SUN_MODEL_DATA, then -predict the field model into MODEL_DATA.

Ah maybe. Is what you are pointing out here is an unavoidable issue when predicting using degridding? I guess if you're using crystalball / DFTs then you could predict the Sun into an MS that was shifted back to the phase centre. We should tread carefully here and make sure the software is doing what we want.

[o-smirnov]

Ah maybe. Is what you are pointing out here is an unavoidable issue when predicting using degridding?

It's avoidable if the degridder knows to apply a phaseshift term. I guess we could check if Andre does that, but there's no design reason why he should (since he refuses to do it for gridding, providing chgcentre instead...)

Crystallball, on the other hand, will predict correctly regardless. I guess @Victoria-Samboco should just check how many components she gets in the source list for the Sun -- if not too many, then just crystallball away and don't worry about image models.

[landmanbester]

The experimental wgridder routines have this functionality (well sort of, you need to tell it what the lm coordinates are at the center of the image) so we could just write a script that does that and slot it into the recipe if needs be. I am not sure how quickly this will materialize considering we have the week at EPFL next week but I am trying to make this much easier using pfb-clean

[Victoria-Samboco]

Crystallball, on the other hand, will predict correctly regardless. I guess @Victoria-Samboco should just check how many components she gets in the source list for the Sun -- if not too many, then just crystallball away and don't worry about image models.

Okay I will check...

[Victoria-Samboco]

Crystallball, on the other hand, will predict correctly regardless. I guess @Victoria-Samboco should just check how many components she gets in the source list for the Sun -- if not too many, then just crystallball away and don't worry about image models.

In the source list for the Sun I got nr sources = 7960 .

This are the restored images of the field, from 1GC and 2GC

After rephasing the MS this are the results of the imaging and deconvolution of the Sun running with the parameters ...
wsclean -name obs2/perscan_ms/clean_image_sun/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12 -data-column CORRECTED_DATA -channels-out 8 -weight briggs 0 -size 6075 6075 -scale 1.5asec -niter 1000000 -fits-mask /home/samboco/solarkat/SUN_IMAGING_STEPS/sun.mask.fits -auto-threshold 3.0 -mgain 0.8 -padding 1.5 -save-source-list -no-update-model-required -use-wgridder -temp-dir temp_dir -log-time -wgridder-accuracy 1e-06 obs2/perscan_ms/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12.ms

Then I used r.renamecol("MODEL_DATA", "ORIGINAL_MODEL_DATA") to rename the MODEL_DATA columnof the Original visibilities.

Predicted the model of the Sun (MODEL_DATA) with crystalball

dd_predict_1:
      cab: crystalball 
      info: "Predicts off-axis source into separate column"
      params:                                       
        ms: '{recipe.dir_out}/{recipe.perscan_dir_out}/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12.ms' #'{recipe.ms}'                           
        within: /home/samboco/solarkat/SUN_IMAGING_STEPS/GGRS_sun.reg                  
        output_column: MODEL_DATA
        num_sources: 10000 # /1000                  
        sky_model: '{recipe.dir_out}/{recipe.perscan_dir_out}/{recipe.clean_image_sun}/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12-sources.txt'  

After predict the MODEL_DATA of the sun ...I proceeded to subtract the MODEL_DATA (model of the Sun) from the CORRECTED_DATA (the rephased data) and image the CORRECTED_DATA. And this is the resultant image after subtracting the Sun with wsclean...

wsclean -name obs2/perscan_ms/subtracted_sun_images/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12 -data-column CORRECTED_DATA -channels-out 8 -weight briggs 0 -size 6075 6075 -scale 1.5asec -niter 0 -auto-threshold 3.0 -mgain 0.8 -padding 1.5 -no-update-model-required -use-wgridder -temp-dir temp_dir -log-time -wgridder-accuracy 1e-06 obs2/perscan_ms/1671435077_sdp_l0_1024ch_GRS1747-312_scan_12.ms

The Subtration does not look is working as the image after Sun subtraction still will the Sun there, that's what I got from now, I'm now going back to the deconvolution step and clean more deeper to see if the problem can be the MODEL of the Sun that its not complete...I'm not sure why its not working...:( I don't know if is the model or something else I'm doing well)

[IanHeywood]

Hi @Victoria-Samboco, here are some thoughts on your post above.

• I think the model is not deep enough, as evidenced by the large amount of emission that remains in your residual images. Increasing the number of iterations is probably not having the intended effect because the use of -auto-threshold 3.0 will terminate the cleaning process at the same stage each time. I would consider lowering this or perhaps deciding on a suitable hard -threshold value.

• It looks like some subtraction has taken place, you can see the sunspot in the lower left of the final image above has become donut shaped, so it looks like the peak of that was captured in the model and subtracted successfully. So I suspect your overall process is working, it's just that the model is not deep enough. It's critical to get this fixed before proceeding. There's no need to go through the full predict and subtract process, we will be able to tell how appropriate the model is from examining model and residual images.

• It looks like your circular region file mask has a radius of 0.5 degrees, but it should be a diameter of 0.5 degrees. It is much bigger than the solar disk, which you can see the edge of in your images above. A half degree circle should encompass only that region.

[Victoria-Samboco]

Hi @Victoria-Samboco, here are some thoughts on your post above.

• I think the model is not deep enough, as evidenced by the large amount of emission that remains in your residual images. Increasing the number of iterations is probably not having the intended effect because the use of -auto-threshold 3.0 will terminate the cleaning process at the same stage each time. I would consider lowering this or perhaps deciding on a suitable hard -threshold value.

I've reduced the auto-threshold to to 0.5 and also the radius to 0.33 degrees fitting the solar disk.
This are the residual and the restored image locked to the same scale. We can see from the comparison between the residual and the restored image that the emission decreased significantly comparing with the image I posted before.

[IanHeywood]

This is looking great now, excellent work. :)

diameter to 0.33 degrees

Did you mean radius here?

[Victoria-Samboco]

This is looking great now, excellent work. :)

Thanks you, but the subtraction still not good enough. Here is a comparison between the first image I posted before and the image of the Sun subtracted from the model correspondent to the residual image above. It shows some more flux subtracted but still not enough...so it means the model still have to be improved...I've cleaned it using a -auto-threshold of 0.5, so I will now use the option of manual -threshold, and see if it works..

diameter to 0.33 degrees

Did you mean radius here?

Yes, radius not diameter, thanks.

[Victoria-Samboco]

Good night Prof, Still in the saga of trying to eliminate the sun, still without success, I have here some images of the result of the subtraction of the Sun using different values ​​of thresholds to clean the Sun. Apparently images with values ​​like 1e-6 and -2e-6 tend to have a larger subtracted area but it also seems that the flow density remains intense in relation to the image where I used threshold the 50e-6 and 10e-6 for example. so I have doubts if I should increase or decrease the values... I ask for your opinion.

The strange thing about this case is that it seems that the more I decrease the threshold value there is an increase in emission in the result of subtracting the sun, which I expected to be the opposite. This are the same images zoomed in.

-2e-4 is a intentional negative threshold because when I made the comparison between 10e-6 and 1e-6 I noticed that the emission increased so I wanted to try a lower value as a test to see if the emission will continue increasing or not, and yes it looks like lower values increases the emission in the Sun in general...

[o-smirnov]

Hoiw exactly are you subtracting it? I suggest you upload the recipe to a gist and link to it. There must be something wrong with the recipe -- if everything was down correctly, you should be getting results as good as your residual image in #22 (reply in thread).

[Victoria-Samboco]

Hoiw exactly are you subtracting it? I suggest you upload the recipe to a gist and link to it. There must be something wrong with the recipe -- if everything was down correctly, you should be getting results as good as your residual image in #22 (reply in thread).

Here (https://gist.github.com/Victoria-Samboco/c7082dcbef7dba1968cc334c55d3daff) is a gist with the cab definition file and the recipe . For now I'm running the pipeline up to subtract_sun-1 step.