Skip to content

generateSampleData.py

Jump to bottom Edit New page
Brett M. Morris edited this page Sep 18, 2013 · 18 revisions

Contents

Using generateSampleData.py

Get the source generateSampleData.py here

Whether you're using oscaar for the first time and want to follow along with our tutorials or you're testing a super-complicated feature that you added to oscaar, it's useful to have a sample data set to experiment with. We've put together a script that will generate simulated images for you, since packaging and distributing a time series of real images would take up lots of memory and limit us to testing only one data set.

The sample data generator produces a series of small images with three stars in the field -- one target star, and two comparisons (see image below). A transit of the target star will occur in the middle of the simulated observations, and we use these data to see how our differential photometry routines perform. Though the simulated "stars" are exceedingly square approximations of what real stellar image shapes look like in real observations, some important sources noise are included in the simulation in order to test oscaar in as close to battle-field conditions as possible. We simulate dark current, sky background, comparison stars of different magnitudes and appropriate photon noise for each of those components. We also jitter the centroids of the stars around randomly in small two dimensional shifts to demonstrate the capabilities of oscaar's stellar centroid tracking algorithm. Below is one such simulated image with the star magnitudes significantly decreased so as to portray the shot noise in the background as well as in the stars in the same color mapping (the stars will be brighter in the out-of-the-box version of generateSampleData.py)

Simulated image with three stars

Oscaar will try to detect a transit of the target star, which we will simulate by modulating the flux of the simulated target.

If you double-click the generateSampleData.py file in Windows, or execute generateSampleData.py from the Command Prompt (Windows)/Terminal (Mac) in a directory that does not have the directory called oscaar in lower-case, with the following command

python generateSampleData.py

the script will create a new directory named "images" and fill it with simulated flats, darks, and on-sky images, and use OSCAAR's flat-calculating algorithm to make a flat field for you, called "masterFlat.fits". Now all you have to do is make a regions file and start running OSCAAR on your simulated data set.

Running OSCAAR on your simulated data

Next, if you've already installed OSCAAR successfully, you can initialize the OSCAAR GUI with the command

python -c "from oscaar import oscaarGUI"

Click the "Browse" buttons and enter the corresponding files for each field. The darks end in "d", the master flat is named "master flat", and the data images end in "r" (for the Red filter, like we usually use at the University of Maryland Observatory).

Here we'll list some good running parameters for OSCAAR when using this simulated data set:

  • Tracking Zoom = 15
  • CCD Gain = 1
  • Aperture Radius = 4.5
  • Smoothing Constant = 3
  • Ingress = 2013/05/15 10:06:30
  • Egress = 2013/05/15 11:02:35.

How generateSampleData.py works

Get the source generateSampleData.py here

generateSampleData.py is a script that will generate some simulated data for us. We can use this simulated data to test OSCAAR as though it were doing photometry on real observations, and since we know what light curve parameters we put into the simulated data we can test to see how accurately OSCAAR extracts those parameters.

To begin, we will import the packages that we will use.

import numpy as np
import pyfits
from shutil import rmtree
from os import mkdir
from glob import glob
from matplotlib import pyplot as plt
import os
import oscaar

We need to set up bunch of parameters about our simulated data. Most of the parameters that you may want to tweak are here at the beginning of the file. They include the number of each type of image that you will produce, the mean number of counts in each of the flat fields (affects the photon noise), the format of your simulated detector (pixel dimensions). The stars will be approximated as box-shaped, and you may set the widths of each star and the relative flux of each star. There will be two comparison stars. You also get to choose the date/times of the simulated observation. I set the times here to be the times that I was writing the script, which results in an ingress time of 2013-05-15, 10:06:30 and egress 2013-05-15 11:02:35.

In this step we will set the planetary system parameters for the star with a transiting planet (the simulated target star). We set those parameters in the list modelParams, whose parameters in order are [R_p/R_s, a/R_s, period, inclination, gamma_1, gamma_2, eccentricity, longitude of pericenter, epoch].

#################################################################
## Tweak these parameters, if you like!
NdataImages = 200        ## Number of data images to generate
NdarkImages = 3          ## Number of dark frames to generate
NflatImages = 3          ## Number of flat fields to generate
flatFieldCounts = 20000  ## Approx counts per pixel in flat field
imageDimensionX = 120    ## Pixel dimensions of each image
imageDimensionY = 40
starDimensions = 4       ## Pixel dimensions of the stars
skyBackground = 500      ## Background counts from sky brightness
darkBackground = 100     ## Background counts from detector
targetFluxOOT = 10000    ## Flux (in counts) from each pixel of the unocculted target star (out-of-transit)
relativeFluxCompA = 0.85 ## Flux from comp A relative to target
relativeFluxCompB = 0.95 ## Flux from comp B relative to target
plotModel = False        ## Plot the injected transit light curve
createMasterFlatNow = True  ## Use oscaar to create a master flat from the freshly generated flat frames

## Set ingress/egress times, transiting system parameters 
## In GD: Ingress = 2013-05-15;10:06:30; egress = 2013-05-15;11:02:35
jd0 = 2456427.88890 
exposureTime = 45/(60*60*24.) ## Convert s -> hr
times = np.arange(jd0,jd0+exposureTime*NdataImages,exposureTime)
# [p,ap,P,i,gamma1,gamma2,e,longPericenter,t0]
modelParams = [ 0.1179, 14.71, 1.580400, 90.0, 0.23, \
                0.30, 0.00, 0.0, np.mean(times,dtype=np.float64)]
## Isn't it nice to control the signal to noise?
#################################################################

We're going to place the simulated images in a directory that we'll make (or overwrite if there is one present already) called images/

## Delete `images` directory, if there is one, and
##      make a fresh one.
if len(glob(os.path.join(os.path.dirname(__file__),'images'))) > 0: rmtree(os.path.join(os.path.dirname(__file__),'images'))
mkdir(os.path.join(os.path.dirname(__file__),'images'))

Let's place our stars in specific places on the detector where we want them. I've arranged for them to be equally spread along one dimension.

## Pixel positions of the stars (x,y)
targetX = [20-starDimensions/2,20+starDimensions/2]
compAX = [60-starDimensions/2,60+starDimensions/2]
compBX = [100-starDimensions/2,100+starDimensions/2]
starsY = [imageDimensionY/2-starDimensions/2,imageDimensionY/2+starDimensions/2]

Later we may run oscaar on these simualted data, and we may want to try to fit the resulting light curve that oscaar produces to a transit light curve and see if we extract the same system parameters that we put in. If that's the case, it would be handy to save a text file of the modelParams exactly the way we entered them so that we can refer to that text file later.

We may also want to see the analytical transit light curve that we'll use to modulate the flux from the target star to know what the simulated light curve will look like. If plotModel == True, this step will generate a plot to show you.

np.savetxt(os.path.join(os.path.dirname(__file__),'modelParams.txt'),modelParams)
modelLightCurve = oscaar.occultquad(times,modelParams)
if plotModel: 
    fig = plt.figure()
    ax1 = fig.add_subplot(111)
    def format_coord(x, y):
        '''Function to also give data value on mouse over with imshow.'''
        return 'JD=%1.6f, Flux=%1.6f' % (x, y)
    ax1.set_xlabel('Time (JD)')
    ax1.set_ylabel('Relative Flux')
    ax1.set_title('Injected Light Curve')
    ax1.plot(times,modelLightCurve)
    ax1.format_coord = format_coord
    plt.show()

We'll prepare two arrays, one in the shape of the whole image and one in the shape of each star so that we can use their sizes as templates throughout the rest of this script. We'll simulate dark frames by creating isotropic images at a particular background level simulating dark current, and add in normally distributed noise of order of the square-root of the background level. This is an approximation to shot noise.

Since usually your detector will measure counts in integers, we force the count numbers to integers here with np.require().

## For producing random arrays, initialize reference arrays with the proper shapes
imageShapedMatrix = np.zeros([imageDimensionY,imageDimensionX])
starShapedMatrix = np.zeros([starDimensions,starDimensions])

## Simulate dark frames with shot noise
for i in range(NdarkImages):
    darkFrame = darkBackground + np.random.normal(imageShapedMatrix,np.sqrt(darkBackground))
    darkFrame = np.require(darkFrame,dtype=int)   ## Require integer counts
    pyfits.writeto(os.path.join(os.path.dirname(__file__),'images/simulatedImg-'+str(i).zfill(3)+'d.fits'),darkFrame)

We'll do a similar procedure to create flat fields, but at a higher count rate than the dark frames. We will use OSCAAR's routine oscaar.standardFlatMaker() to produce the flats, just as you would if you were using the oscaarGUI.

## Simulate ideal flat frames (perfectly flat)
for i in range(NflatImages):
    ## Flats will be completely flat -- ie, we're pretending that we have a 
    ##      perfect optical path with no spatial flux variations.
    flatField = np.random.normal(imageShapedMatrix,np.sqrt(flatFieldCounts)) +  flatFieldCounts
    flatField = np.require(flatField,dtype=int)## Require integer counts
    pyfits.writeto(os.path.join(os.path.dirname(__file__),'images/simulatedImg-'+str(i).zfill(3)+'f.fits'),flatField)


## Create master flat now using oscaar's standard flat maker
if createMasterFlatNow:
    flatPaths = glob(os.path.join(os.path.dirname(__file__),'images/simulatedImg-???f.fits'))
    flatDarkPaths = glob(os.path.join(os.path.dirname(__file__),'images/simulatedImg-???d.fits'))   ## Use the same darks
    masterFlatSavePath = os.path.join(os.path.dirname(__file__),'images/masterFlat.fits')   ## Where to save the master
    oscaar.standardFlatMaker(flatPaths,flatDarkPaths,masterFlatSavePath,plots=False)

Now we'll create the simulated data images in a longer loop. These images will have background noise dark current and also sky background that have photon noise incorporated. We'll simulate the target star now too, with its own photon noise, and modulate its flux using the model light curve that we produced earlier.

## Create data images
for i in range(NdataImages):
    ## Produce image with sky and dark background with simulated photon noise for each source
    simulatedImage = darkBackground + skyBackground +\
        np.random.normal(imageShapedMatrix,np.sqrt(darkBackground)) +\
        np.random.normal(imageShapedMatrix,np.sqrt(skyBackground))
    
    ## Create two box-shaped stars with simulated photon noise
    targetBrightness = targetFluxOOT*modelLightCurve[i]  ## Scale brightness with the light curve
    target = targetBrightness + np.random.normal(starShapedMatrix,np.sqrt(targetBrightness))

We'll make the comparison stars similarly, though their mean fluxes will remain constant barring random noise.

    compBrightnessA = targetFluxOOT*relativeFluxCompA
    compBrightnessB = targetFluxOOT*relativeFluxCompB
    compA = compBrightnessA + np.random.normal(starShapedMatrix,np.sqrt(compBrightnessA))
    compB = compBrightnessB + np.random.normal(starShapedMatrix,np.sqrt(compBrightnessB))

We'd like the stars to jiggle around in place to simulate the imperfect pointing of most small telescopes. Rather than making a bigger simulated image and having the stars drift across it, as will likely happen in an amateur's observations, we'll just jitter the stars around randomly within one or two pixels of the center of the exposure.

    ## Add stars onto the simulated image with some position jitter
    randomPositionJitterX = np.sign(np.random.uniform(-1,1))	## +/- 1 pixel stellar centroid position jitter
    randomPositionJitterY = np.sign(np.random.uniform(-1,1))	## +/- 1 pixel stellar centroid position jitter
    simulatedImage[starsY[0]+randomPositionJitterY:starsY[1]+randomPositionJitterY,targetX[0]+randomPositionJitterX:targetX[1]+randomPositionJitterX] += target
    simulatedImage[starsY[0]+randomPositionJitterY:starsY[1]+randomPositionJitterY,compAX[0]+randomPositionJitterX:compAX[1]+randomPositionJitterX] += compA
    simulatedImage[starsY[0]+randomPositionJitterY:starsY[1]+randomPositionJitterY,compBX[0]+randomPositionJitterX:compBX[1]+randomPositionJitterX] += compB

Again we'll force the count numbers to integers. Lastly, we want to write the header for our FITS files with the times of each observation so that OSCAAR will find the exposure times later. We'll do that using the pyfits.Header() object and save the FITS file with pyfits.writeto(). Typically you'll save your images with some part of the data indicating which filter you used. Here we write a suffix to each file name "r" for the "red" filter that we pretend to have used to make this simulated data.

    ## Force counts to integers, save.
    simulatedImage = np.require(simulatedImage,dtype=int)   ## Require integer counts before save

    header = pyfits.Header()
    header.append(('JD',times[i],'Simulated Time (Julian Date)'))
    pyfits.writeto(os.path.join(os.path.dirname(__file__),'images/simulatedImg-'+str(i).zfill(3)+'r.fits'),simulatedImage,header=header)

If you run the script successfully, there will be a directory of images with the prefix "simulatedImg-" in the directory "images/". If you make a regions file by opening the first frame of the data images simulatedImg-000r.fits and clicking first on the left-most star (the target) before clicking on the next two comparisons, you'll be ready to select all of your files in the oscaarGUI and to try producing a light curve.

Add a custom footer
Add a custom sidebar
Clone this wiki locally