Skip to content

Linear Polarization Reduction. Beta version

Jump to bottom
knordsieck edited this page Jan 23, 2020 · 17 revisions

Unpack your data
Copy calibration database files
Carry out the polarimetric pipeline analysis
Analysis scripts
Release Notes

Here are the steps necessary to reduce linear polarimetric data from RSS with the "Beta" version of the polsalt package. The initial reduction to calibrated Stokes fits files is automated with the minimum amount of interaction. Then, a number of tools are available to re-reduce subsets of the data, plot the results, and do higher-level analysis. If you haven't installed polsalt, follow the Installation of Polsalt instructions, Beta version, to do so. Note the path to the working directory 'polsalt' in step 3. The pipeline script "reducepoldata_sc.py" is in the polsalt/scripts directory.

1. Unpack your data

in the same format it comes from SALT such that the raw data is inside a directory with the observing date in YYYYMMDD format. Copy the reducepoldata_sc.py script into the directory that contains your YYYYMMDD directories. For example, your data might be sitting in 20171203/raw/ (this will be used in YYYMMDD examples below, for clarity)

2. Copy calibration database files.

If desired (see below) copy any desired wavelength calibration database guesses arcdb_mm_guess.txt or fluxcal database files fluxdb_calsp_cn.txt into a 20171203/sci directory.

The pipeline

3. Carry out the polarimetric pipeline analysis on your data with

python reducepoldata_sc.py 20171203

Output would be in the 20171203/sci directory, which is created if it does not already exist. Specifically, the pipeline steps you will see in the reducepoldata_sc script are:

3.1 imred.py. Basic CCD image reductions include overscan subtraction, gain correction, crosstalk correction, mosaicing. This produces "mxgbPp.." fits files similar to those in your SALT product directory, but with additional bad-pixel and variance information necessary for the polarimetric reduction. This step would be logged in sci/im20171203.log. The steps below are logged in sci/specpol20171203.log.

3.2 specpolwavmap.py. For the wavelength calibration step, the data is first sorted using header keywords into object, spectrograph configuration, and polarimetric configuration categories to find arcs going with each. For the arcs the O (bottom) and E (top) beamsplitter beams are split apart and separate wavecals done for each. The specidentify interface will appear for you to identify lines. We suggest doing the manual identification for the first row and once you are happy with the solution on that row and have saved it, the rest of the rows can be auto-identified. The auto-identify step works best if you have 10-15 evenly spaced lines (no close pairs, which often result in misidentifications). The calibration step will occur twice (once for the O-beam and once for the E-beam). The E-beam uses the O-beam solution as its first guess, so it usually goes quite fast. For each arc image nn, it will save an arcdb_nn_0.txt (O beam) and arcdb_nn_1.txt (E beam) in the sci directory. The rms relative variation of wavelength along the slit is printed out. It should not be significantly larger than about .015%, which is the value expected from the spectral curvature. If it is larger than 0.05%, a fallback wavmap which has no curvature is used, to avoid spoiling the sky line subtraction.

Once a configuration has been successfully calibrated the first time, either of these arcdb files may be copied into future sci directories (see above) before running the pipeline, renaming it to arcdb_mm_guess.txt as a wavcal guess for arc image mm. It will be used as a first guess in the O-beam, which will now go quite fast.

For each spectropolarimetric image file Cosmic Ray rejection is now done (now that the beams are split), and a new file "wmxgbPp.." is saved, which has the beams split and a wavelength map plane added. We typically see about 25 CRs/sec of exposure.

3.3 specpolextract_sc.py. For each spectropolarimetric image, each beam is corrected for beamsplitter distortion and tilt using the brightest spectrum in the image and saved as a "cwmxgbPp.." file. Now, using the wavmap plane, the sky and spectrum is extracted vs wavelength and the extracted O and E spectra stored as a 1D "image" on a linear wavelength scale in a "ecwmxgbPp.." file. The science object is the brightest signal within a "locate" window along the slit. The default locate window is the whole slit: (-120,120) arcsec, but it may be changed in the reducepoldata_sc script, if desired. The object spectrum is then extracted from an "extract" window centered on the brightest object in the "locate" window. The "extract" window parameter may also be adjusted, to avoid nearby objects. We suggest leaving the default 10 arcsec extract window for the 300-line grating, to avoid the second order spectrum. For the other gratings, the extraction window may be adjusted to avoid neighboring objects or to allow for good or poor seeing.

Frequently, programs select a position angle such that a comparison object falls elsewhere on the slit. The comparison object may then also be extracted in the same reducepoldata_sc step by restricting the science locate window and by uncommenting the specpolextract_sc line in the script with the argument "docomp=True". If this is done, the object outside the science locate window which is brightest (in the "O" beam) is extracted as a comparison object, which will reduced for spectropolarimetry together with the science object, and will be named "COMP_". The position of the science object (and the comparison object, if selected) relative to the center of the slit is printed out in the log.

3.4 specpolrawstokes.py. Identifies "waveplate position pairs" (waveplate positions 4 stations apart, 45 degrees, which are taken together, and result in the linear polarization signal swapping between the O and E beams), and combines them into "raw stokes" fits files, which are basically single stokes parameters. They are stored unnormalized, with an "I" plane and an "S" plane, the stokes degree of polarization being S/I. The rawstokes files are named <object>_<config>_h<12>_<cycle>.fits, for instance WR113_c0_h26_02.fits for object WR113, first configuration, waveplate position pair 2 and 6, second cycle of waveplate pattern.

3.5 specpolfinalstokes.py. Associates raw stokes files with an object, configuration, and waveplate pattern. Combines multiple cycles, culling spikes, and evaluating a "sys %err" and a "chi-square" for each waveplate pair. For LINEAR-HI waveplate patterns, an additional chi-square estimate is made for each waveplate pair by comparing with the others. "sys %err" should be within +/-0.03% and Chi-square should be comparable to 1 (less than 2-3). If one gets significantly larger numbers, see below for "data culling advice".

Now it evaluates the full pattern to get Q and U, and applies the polarimetric zero-point and the waveplate efficiency and axis calibrations to give the final stokes parameters, which are saved as <object>_<config>_<cycles>_stokes.fits files, i.e. WR113_c0_1_stokes.fits for cycle #1, first configuration of WR113. These are again unnormalized stokes parameters I,Q,U, with degree of polarization being Q/I and U/I.

Finally, a flux calibration is applied by first finding existing fluxdb files in the sci directory (see above), then making new fluxdb files from ec extraction files associated with CAL_SPST stars, checking whether the configurations match, then multiplying I,Q,U by the sensitivity curve and the variances by its square. Fluxed stokes parameters are identified by FITS keyword "CUNIT3". A flux- (or intensity-, if not fluxed) mean polarization and its errors are finally printed out.

Analysis scripts

The data may now be examined and further analysis planned using individual standalone routines in the scripts directory. Just cd to the sci directory and execute

python script.py <scripts python command>

(the script.py routine was copied to the sci directory by reducepoldata_sc)

4. specpolfinalstokes.py. Data culling.

This is available as a stand-alone script so that it may be re-executed to experiment with culling rawstokes input to improve systematic errors. During its execution in the pipeline, "Syserr" and "chi-square" estimates based on comparison of redundant data are printed out. For multiple cycles of either LINEAR or LINEAR-HI patterns, each raw-stokes in a cycle is compared with the mean of the others.

For instance, for 3 LINEAR's:

  Observation: WR113_c0_123  Date: 20170808						
						
                culled          sys %err        mean chisq 						
      HW	  04      26	  04	  26	     04	     26
cycle  1:	  11	 137     0.047	-0.057	   1.04	   2.15
cycle  2:	   7	  28	-0.010	 0.045	   1.55	   1.20
cycle  3:	  12	  13	-0.037	 0.012	   1.06	   1.29
net    :	   0	   0	 0.020	 0.025	   1.22	   1.55

  Estimated sys %error: 0.022%   Mean Chisq:   1.38				

Wavelen	    % Q	    % U	   % Q Err	% U Err	    % P	    PA	 % P Err	PA  Err
5735.09	  -2.4793  -0.1436	0.0010	0.0011	   2.4835 91.657   0.0010	0.012

The first two columns give the number of wavelengths that were automatically culled as uncorrected cosmic rays based on the comparison (possible whenever there are at least 3 cycles to allow "voting"). The second two columns give the deviation of the mean, which is an estimate of the systematic polarization baseline fluctuation, likely due to imperfect repeatability of the waveplate angle. The third two columns list the mean chi-square for each rawstokes observation compared to the others after CR culling. The last line is the "net" over the three cycles: net culled is the number of wavelengths which were culled in more than one cycle (usually quite small for multicycle data). These data are fairly good high S/N data: we have been seeing Sys Err typically no better than .015%. However, we do see that there is one rawstokes measurement, the "26" observation for cycle 1, which has an elevated cull, sys error, and chisq. If desired, the data may be re-reduced leaving out this observation via:

python script.py specpolfinalstokes.py WR113*h04*.fits WR113*h26_0[2-3].fits

  Observation: WR113_c0_123_23  Date: 20170808

                culled      sys %err     mean chisq 
         HW    04   26      04      26    04    26
   cycle  1:   11    0   0.047   0.000  1.04  0.00 
   cycle  2:    7    9  -0.010   0.016  1.55  1.64 
   cycle  3:   12    9  -0.037  -0.016  1.06  0.93 
    net    :    0    9   0.020   0.012  1.22  1.28 

  Estimated sys %error: 0.016%   Mean Chisq:   1.25

which now gives an improved systematic error. The naming of the culled result now gives which cycles of each rawstokes are included, since they may be different.

For LINEAR-HI, even one cycle has redundancy, and each rawstokes may be compared with two of the others, for instance:

  Observation: WR113_c0_1  Date: 20170531

                     culled                   sys %err                   mean chisq       
         HW          all              04      15      26      37    04    15    26    37
      Linhi:          24           0.099  -0.036  -0.049   0.104  1.92  1.54  1.71  2.10 

  Estimated sys %error: 0.078%   Mean Chisq:   1.81

   Wavelen     % Q      % U     % Q Err  % U Err    % P      PA     % P Err  PA  Err 
    5734.25   -2.4916  -0.1135   0.0013   0.0013   2.4942   91.304  0.0013    0.014

Culling the 37 rawstokes, which has elevated syserr and chisq:

python script.py specpolfinalstokes.py WR*h04*.fits WR*h26*.fits WR*h15*.fits

  Observation: WR113_c0_1_1_1__  Date: 20170531

                     culled                   sys %err                   mean chisq       
         HW          all              04      15      26      37    04    15    26    37
      Linhi:          12           0.025  -0.035   0.025   0.000  1.54  1.54  1.54  0.00 

  Estimated sys %error: 0.029%   Mean Chisq:   1.54

   Wavelen     % Q      % U     % Q Err  % U Err    % P      PA     % P Err  PA  Err 
    5734.25   -2.4578  -0.1541   0.0015   0.0016   2.4626   91.794  0.0015    0.018

We see that the syserr is now more normal. The % P Err from photon statistics is slightly worse, so the small-scale noise will be slightly larger, but the overall mean polarization is more precise. (All the chi-squares are now the same since only 3 different rawstokes are available.) The naming shows that the last rawstokes in the pattern, 37, is not used.

5. specpolview.py. Text and plot output of optionally binned data.

python script.py specpolview.py <*stokes.fits files> <options>

If more than one stokes file is listed, results will be overplotted, but an individual text output file for each is output.

General options

  • save=
    default: the mean polarization result as seen in the specpolfinalstokes output is printed.

    text: A text file, labeled <stokesfile name>_<bin>.txt is saved, containing columns with binned data:
    Wavelen Intensity/Flambda % Q % U % Q Err % U Err % P PA % P Err PA Err

    plot: A pdf plot file (also displayed) is saved, with three plots, intensity or Flambda (unbinned), and two polarization plots (binned), depending on the "type" option below (default %P and PA). An input is requested for optional yscaling of the polarization plots, followed by optional wavelength scaling of all:
    Optional scale (bottom-top, comma sep):

    for instance, "80,180,2,3,4500,5000" plots PA between 80 and 100 degrees, and polarization 2-3%, for wavelengths between 4500 and 5000 Angstroms. Any values not given (though still comma separated) will be autoscaled.

    textplot: both text and plot

  • bin=
    unbin: (default)

    nnA: nn Angstrom bins

    f.f%: binned to given % error

Plot options These do not effect the text output:

  • type=
    Note: the case is significant: upper case is unnormalized, and lower case is normalized to I (in % or deg). Coming soon: polarized flux IPt and IQU.

    Ipt (default): plots intensity or flux, % P, and PA (deg)

    Iqu: plots intensity or flux, % Q, and % U. Asks for an optional non zero PA zeropoint, so that Q corresponds to that PA.

  • errors= False (default) or True

    if True, error bars are plotted.

  • connect=
    default polarization points are plotted, connected by lines.

    hist: if binned, the bins are shown histogram-style.

So, for instance

python script.py specpolview.py WR113_c0_13_123_stokes.fits bin=20A errors=True connect=hist save=textplot

6. specpolflux.py. Do flux-calibration step.

python script.py specpolflux.py <*stokes.fits>

Looks in the current directory for flux standard data and creates a fluxdb file, then looks in the current directory for fluxdb configurations that match the listed stokes files, and applies the flux calibrations to them.

7. specpolfilter.py. Compute synthetic filter polarization results for listed filters.

python script.py specpolfilter.py <filters> <*stokes.fits>

U, B, and V are the Johnson filter curves from the SALTICAM filters directory. R,I are the Cousins filter curves. One may also list the name(s) of text files in the CWD with lines listing wavelength and filter throughput. The results give the polarization weighted by the filter throughput and the flux (or intensity, if not fluxed).

python script.py specpolfilter.py B V R WR113_c0_13_123_stokes.fits

WR113_c0_13_123_stokes.fits
Filter     Q      Err         U      Err     
   B   -2.5559  0.0046     0.0563  0.0039  
   V   -2.6290  0.0024    -0.0162  0.0020  
   R   -2.5217  0.0019    -0.0305  0.0015

Release Notes

20171226

  • Error analysis (all routines) now includes resampling covariance, which makes the photon-limited chi-square come out to be 1.
  • Wavemap now falls back to straight spectra if the specid result is too noisy.
  • Large specpolextract_sc update to support target/comparison extraction.
  • Multi-cycle averaging in specpolfinalstokes is now done on normalized stokes basis, to avoid coupling of mean syserr into polarimetric spectral features.
  • Added specpolview "connect" (=hist) and "type" (=Iqu) options. Moved "errors" to be a proper separate option.

20180315

  • imred uses new version of mosaicing and CCD geometry parameters, correcting errors which led to faulty mosaicing for column binning other than 2.
  • Added optional wavelength scale specification in specpolview.

20180429

  • to RSSpol_LinearPAZeropoint add 20180410_v01 120.0, based on 20180427 HD298383 data, provisional calibration after large change due to waveplate mechanism work
  • update specpolfinalstokes to use dated PAZeropoint

20191021

  • update specpolwavmap, specpolsplit to allow for odd number of rows in mosaic
  • update specpolwavmap to allow use of two simultaneous different arcs
  • added capability for multi-iteration exposures
  • update specpolview to allow decimal Ang in binning

20191211

  • split specpolextract into separate correction, spectrum pieces to make optional specification of extraction windows using GUI easier
  • extraction improvements including docomp option, cr handling, spectrum location
  • update specpolrawstokes, finalstokes to use mean TELPA to increase PA precision
  • allow for correction of amplifier gains in extraction to improve flux cals

20200123

  • allow splitting of multiple block-visits to same target
  • uploaded rssmaptools module, left out of previous submission
Clone this wiki locally