Remove fixed pattern noise from maps

In particular, getting rid of the tartan patterns in MUSE velocity maps

Real-space filtering

• Period seems to be 292 pixels
• New [2016-12-18 Sun]
  • [X] Fix the plot file naming to be per map
  • [X] Polynomial detrend each chunk to order 2
  • [X] Allow adjustment of plot limits
  • [ ] Make lines thinner
  • [ ] Look into changing period in y
    • Actually, we want

Seperable in X and Y

The mean velocities can be dealt with this way

import sys
import os
import numpy as np
from astropy.io import fits
from matplotlib import pyplot as plt
import seaborn as sns

try: 
    infile = sys.argv[1]
except:
    sys.exit('Usage: {} FITSFILE [YMIN, YMAX] [NP]'.format(sys.argv[0]))

try:
    YMIN, YMAX = float(sys.argv[2]), float(sys.argv[3])
except IndexError:
    YMIN, YMAX = 0.85, 1.15

try:
    NP = int(sys.argv[4])
except IndexError:
    NP = 2

basename = os.path.basename(infile)
baseroot, _ = os.path.splitext(basename)

hdu = fits.open(infile)[0]
if hdu.data is None:
    hdu = fits.open(infile)[1]
hdr = hdu.header

ny, nx = hdu.data.shape

# Size of chunks
mx, my = 290, 290
xchunks, ychunks = nx//mx, ny//my
xprofile = np.nanmean(hdu.data, axis=0)
yprofile = np.nanmean(hdu.data, axis=1)
plotfile = 'pattern-noise-xy-{}.pdf'.format(baseroot)

sns.set(style='whitegrid', font_scale=1.5, color_codes=True)
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)

x = np.arange(mx)
xpstack = []
for ichunk in range(xchunks):
    xp = xprofile[ichunk*mx:ichunk*mx + mx]
    p = np.poly1d(np.polyfit(x, xp, 2))
    ax1.plot(x, xp/p(x),
             lw=1, label='X chunk {}'.format(ichunk))
    xpstack.append(xp/p(x))
xpstack = np.vstack(xpstack)
xpm = np.median(xpstack, axis=0)
ax1.plot(x, xpm, 'k', lw=6, alpha=0.2)
ax1.legend(ncol=2, fontsize='xx-small')
ax1.set(
    ylim=[YMIN, YMAX],
    ylabel='x profile',
)

y = np.arange(my)
ypstack = []
jshifts = {0: -4, 1: +2, 2: +3, 3: +2, 4: -4}
for jchunk in range(ychunks):
    yp = yprofile[jchunk*my:jchunk*my + my]
    if jchunk in jshifts:
        # Per-chunk shift if necessary
        yp = np.roll(yp, jshifts[jchunk])
    p = np.poly1d(np.polyfit(y, yp, 2))
    ax2.plot(y, yp/p(y),
             lw=1, label='Y chunk {}'.format(jchunk))
    ypstack.append(yp/p(y))
ypstack = np.vstack(ypstack)
ypm = np.median(ypstack, axis=0)
ax2.plot(y, ypm, 'k', lw=6, alpha=0.2)
ax2.legend(ncol=2, fontsize='xx-small')
ax2.set(
    ylim=[YMIN, YMAX],
    xlabel='pixel in chunk',
    ylabel='y profile',
)

fig.set_size_inches(6, 6)
fig.tight_layout()
fig.savefig(plotfile)

# Finally, reconstruct the pattern noise image
patmap = np.ones_like(hdu.data)
# standard_tile = xpm[None, :]*ypm[:, None]
standard_tile = 0.5*(xpm[None, :] + ypm[:, None])
for jchunk in range(ychunks):
    yslice = slice(jchunk*my, jchunk*my + my)
    if jchunk in jshifts:
        # Undo y shift if present
        tile = np.roll(standard_tile, -jshifts[jchunk], axis=0)
    else:
        tile = standard_tile
    for ichunk in range(xchunks):
        xslice = slice(ichunk*mx, ichunk*mx + mx)
        patmap[yslice, xslice] = tile

fits.PrimaryHDU(data=patmap,
                header=hdr).writeto(infile.replace('.fits',
                                                   '-pattern.fits'), clobber=True)

fits.PrimaryHDU(data=hdu.data/patmap,
                header=hdr).writeto(infile.replace('.fits',
                                                   '-patfix.fits'), clobber=True)

print(plotfile, end='')

D=../OrionMuse/LineMaps
python de-pattern-noise.py $D/mean-O_III-5007.fits

D=../OrionMuse/LineMaps
python de-pattern-noise.py $D/sigma-O_III-5007.fits 0.9 1.1 0

• This works pretty well, but there is still a horizontal ridge to clean up at the top and bottom
• Now look at the red lines, that are better s/n

D=../OrionMuse/LineMaps
python de-pattern-noise.py $D/mean-S_III-9069.fits 0.9 1.1

python de-pattern-noise.py LineMaps/mean-Ar_III-7136.fits

python de-pattern-noise.py LineMaps/sigma-Ar_III-7136.fits

And the lower ionization lines

python de-pattern-noise.py LineMaps/mean-S_II-6731.fits

python de-pattern-noise.py LineMaps/mean-O_II-7330.fits

python de-pattern-noise.py LineMaps/mean-O_I-8446.fits

• Wow! with this, we can see the base of the jets that must feed HH203/204
  • See blueshifted filaments in O I 8446

Non-seperable tile-by-tile

• For the velocity widths, the pattern is not a simple product of f(X) * g(Y)
• Rather, there is a complex pattern of subtiles
• I will try and deal with it in the simplest way possible, by taking the median of all 30 tiles

import sys
import os
import numpy as np
from astropy.io import fits

try: 
    infile = sys.argv[1]
except:
    sys.exit('Usage: {} FITSFILE'.format(sys.argv[0]))


basename = os.path.basename(infile)
baseroot, _ = os.path.splitext(basename)

hdu = fits.open(infile)[0]
if hdu.data is None:
    hdu = fits.open(infile)[1]
hdr = hdu.header

ny, nx = hdu.data.shape

# Size of chunks
mx, my = 290, 290
xchunks, ychunks = nx//mx, ny//my

# Initialize 3-d array to hold stack of tiles
ntiles = xchunks*ychunks
tilestack = np.zeros((ntiles, my, mx))

# Little nudges to get the peaks to line up for the different y chunks
jshifts = {
    0: -4,
    1: +2,
    2: +3,
    3: +2,
    4: -4,
}

# Put each tile on to the stack
ktile = 0
for jchunk in range(ychunks):
    yslice = slice(jchunk*my, jchunk*my + my)
    jshift = jshifts.get(jchunk, 0)
    for ichunk in range(xchunks):
        xslice = slice(ichunk*mx, ichunk*mx + mx)
        # Roll the data according to the jshift
        tile = np.roll(hdu.data[yslice, xslice], jshift, axis=0)
        # This has side effect of making tile be a copy rather than a view

        # Normalize each tile and dd to the stack
        tilestack[ktile, :, :] = tile / np.nanmedian(tile)
        ktile += 1

# Take the median down the stack to get the pattern
pattern_tile = np.nanmean(tilestack, axis=0)

# Now use copies of pattern_tile to make the pattern noise map
patmap = np.ones_like(hdu.data)
for jchunk in range(ychunks):
    yslice = slice(jchunk*my, jchunk*my + my)
    jshift = jshifts.get(jchunk, 0)
    for ichunk in range(xchunks):
        xslice = slice(ichunk*mx, ichunk*mx + mx)
        # Roll the data back again according to the jshift
        patmap[yslice, xslice] = np.roll(pattern_tile, -jshift, axis=0)

fits.PrimaryHDU(data=patmap,
                header=hdr).writeto(infile.replace('.fits',
                                                   '-patternx.fits'), clobber=True)

fits.PrimaryHDU(data=hdu.data/patmap,
                header=hdr).writeto(infile.replace('.fits',
                                                   '-patfixx.fits'), clobber=True)

D=../OrionMuse/LineMaps
for f in $D/sigma-*-????.fits; do
    echo "Processing  $f"
    python de-pattern-extreme.py $f
done

• [2016-12-19 Mon] We now use patfixx as the suffix to distinguish these from the ones in the previous section
  • This is so we can compare which technique is more effective in the case of the eman velocities
  • So, fix up the names of files we have already written for consistency

D=../OrionMuse/LineMaps
for f in $D/sigma-*-????-patfix.fits; do
    suff=$(basename $f .fits)
    mv $f ${suff}x.fits
done

• Try out this version on the mean velocities too
  • Run this in a shell

D=../OrionMuse/LineMaps
for f in $D/mean-*-????.fits; do
    echo "Processing  $f"
    python de-pattern-extreme.py $f
done

Fourier-space filtering

• Actually it looks like this is not going to be necessary