Basic GOT Data Reduction

Assuming you had a successful observing run, you now have probably of order 100 images on disk, comprising observations of your primary targets and standard stars, flat fields, darks, and zeros. The goal is to turn what was recorded by the CCD electronics (which is what you have) into a credible rendition of what the telescope actually saw (which is what you want), so that you can begin to do some science.

Where to begin?

Before you do anything else, read Phil Massey's User's Guide to CCD Reductions with IRAF. But DON'T use it as a "cookbook" for your data. GOT data differ in important ways from the data used in the User's Guide (no overscan region, for instance). Use it to acquaint yourself with the necessary reduction steps for direct imaging data and the IRAF tasks you'll be using.

This page will take you through the procedures step by step, referring to examples in the User's Guide. If you follow the examples carefully, you should be able to reduce your data in a few hours.

Important Note: All of the data reduction documentation refers to images explicitly in IRAF's "imh" format. If you have decided to do all of your reduction in FITS format, you should make sure you have configured your login.cl file as recommended, and you should substitute ".fts" for ".imh" wherever it appears. With "fts" defined as your default imtype, you can give just the root name without the extension, and IRAF will be default assume the extension ".fts".

1. Index files

Let's suppose that you have a night's worth of V and R data on the open cluster M67, along with standard star observations and calibration frames. A partial list of your frames might consist of:

  name          exp        target

dark01.imh       10         dark
dark02.imh       10         dark
dark03.imh       10         dark
dark04.imh       30         dark
dark05.imh       30         dark
dark06.imh       30         dark
dark07.imh      300         dark
dark08.imh      300         dark
dark09.imh      300         dark
flat01.imh       20     V flat, evening
flat02.imh       40     "     "     "
flat03.imh       90     R flat, evening
flat04.imh      120     "     "     "
flat05.imh      180     R flat, morning
flat06.imh      150     "     "     "
flat07.imh       60     V flat, morning
flat08.imh       30     "     "     "
n1001.imh        30     standard star, R
n1002.imh        30     standard star, R
n1003.imh        30     standard star, V
n1004.imh        30     standard star, V
n1005.imh       300        M67, R
n1006.imh       300        M67, R
n1007.imh       300        M67, V
n1008.imh       300        M67, V
n1009.imh        10     standard star, R
n1010.imh        10     standard star, R
n1011.imh        10     standard star, V
n1012.imh        10     standard star, V
zero01.imh        0         zero
zero02.imh        0         zero
zero03.imh        0         zero
zero04.imh        0         zero
zero05.imh        0         zero

Use a text editor to write index files of your frames, sorted by type, exposure time, and filter. Each file (plain text) will contain nothing but a list of file names. For these data, you would need to make the following index files:

darkframes.all   flatframes.all  objframes.all   zeroframes.all
darkframes.10s                   objframes.10s   
darkframes.30s                   objframes.30s   
darkframes.300s                  objframes.300s

It's also useful, though not absolutely essential, to make index files of the flats and object frames sorted by filter:

flatframes.V     objframes.V
flatframes.R     objframes.R

The "objframes" files should list the exposures of all real astronomical targets, standard stars included. Other than that, the names ought to be self-explanatory. For instance, the index file "darkframes.30s" for our fictional night would look like this:

dark04.imh
dark05.imh
dark06.imh

And "objframes.V" would contain

n1003.imh
n1004.imh
n1007.imh
n1008.imh
n1011.imh
n1012.imh

Obviously, if you have observed in more than 2 filters, you will need more index files.

Proofread your index files carefully, since they must be error-free!

2. Add information to the image headers

The tasks in the CCDRED package make use of some header keywords that the CCDOPS software does not write into the image headers, so we have to add these manually. Start the IRAF cl (in the correct home directory!) and load the 'noao', 'imred', and 'ccdred' packages. After you do this, the IRAF prompt should be "cc>". Go to the directory where your data and index files are, and do the following:

cc> hedit @objframes.all imagetyp object add+ verify-
cc> hedit @flatframes.all imagetyp flat add+ verify-
cc> hedit @darkframes.all imagetyp dark add+ verify-
cc> hedit @zeroframes.all imagetyp zero add+ verify-

Your image headers have an EXPTIME keyword that gives the total exposure time, but they do not have the DARKTIME keyword, which gives the total time that the detector was accumulating charge. We are assuming these are the same for our camera; however, we still need to add the DARKTIME keyword and give it the correct value. Here is a single command which will make the change to all images in one pass:

cc> hedit *.imh darktime "(exptime)" add+ ver-

You also need to add the CCD gain and readnoise to the headers. The most recently measured values can be found in the GOT Observer's Guide. The needed header keywords can be added like this:

cc> hedit *.imh gain ### add+ verify-
cc> hedit *.imh rdnoise ### add+ verify-

where ### should be replaced with the correct numerical values.

Finally, IRAF recongizes the header keyword FILTERS, but the CCDOPS software calls this FILTER. This command will add the FILTERS keyword to all the flat-field and object headers and give it the same value as FILTER:

cc> hedit @flatframes.all,@objframes.all filters "(filter)" add+ ver-

Notice that there is no space after the comma!

3. Set up the CCDRED package parameters

We are now at section 3.3 of the User's Guide (p. 7). There is no need to look at all the options for 'setinstrument'; simply type

cc> setinstrument direct

This pops you into the parameter editor for the CCDRED package (p. 10). Check to be sure that the pixeltype is "real real", then exit the parameter editor with CTRL-D (not CTRL-Z, as the Guide says!). This takes you to the parameter editor for the 'ccdproc' task. The trimsec and biassec should both be set to "image", but it really doesn't matter if they're not. Exit this editor with CTRL-D also. You should now be able to use 'ccdlist' to verify that things are set correctly, e.g.,

cc> ccdlist flat04.imh
flat04.imh[1530,1020][ushort][flat][R]

The pixel type will remain "ushort" until you actually begin the processing. Note that the image type is now correctly given as "flat" and the filter is shown too. You can check that all the image types were assigned as you wanted by using your index files, e.g.,

cc> ccdlist @objframes.all

should give you a list of images all with image type "object". But remember that the image types will be correct only if your index files are correct! You can check the filters in a similar way, e.g.,

cc> ccdlist @flatframes.I

You can doublecheck that you have the exposure times indexed correctly by doing, for example,

cc> imhead @darkframes.300s l+ | match EXPTIME
EXPTIME =     3.0000000000E+02 / EXPOSURE IN SECONDS
EXPTIME =     3.0000000000E+02 / EXPOSURE IN SECONDS
EXPTIME =     3.0000000000E+02 / EXPOSURE IN SECONDS

But if any of the exposure times are off you will have to figure out for yourself which ones they are.

4. Examine the zeros

You can use your index files with the 'imstat' task to check the basic properties of your zeros:

cc> imstat @zeroframes.all 
#               IMAGE      NPIX      MEAN    STDDEV       MIN       MAX
           zero01.imh   1560600     101.7     6.019       76.      189.
           zero02.imh   1560600     99.69     6.002       73.      177.
           zero03.imh   1560600     98.66     5.981       72.      271.
           zero04.imh   1560600      98.7      6.01       72.      284.
           zero05.imh   1560600     99.66     6.009       74.      356.

You want to verify that all of your zeros have roughly the same mean and standard deviation. In this example they all look OK. If any of the frames seems deviant, edit your index file and delete the name of the deviant image so that it won't get used.

Use the image display to visually examine all of the good zero frames. Zoom out and move around so that you can look at the whole image. You are looking for images that have significantly different shape from the others (something that 'imstat' might not have revealed). If you find any, delete them from the index file. It is not uncommon for the first zero of a set to be weird. You don't want to be deleting more than one or two of your zeros; if you are finding that you need to throw out a lot of them, there is some problem with your data. But remember that some variation in the images is natural. Don't put too much weight on the colors in the displayed image (if you are displaying with a color table), because IRAF is automatically scaling the colors based on the pixel values. When you 'display' an image, IRAF reports back the z1 value, below which all pixels are displayed as black, and the z2 value, above which they are all white. If you want to set the z1 and z2 manually, it's done like this:

cc> display zero06.imh 1 zsc- zra- z1=80. z2=200.

5. Examine the darks

Now do the same thing with the darks. You should find that dark frames of the same exposure time should all be similar. Check them with 'imstat' AND by visual inspection, and delete any serious oddballs from your index files. (Remember that you may have each image listed in more than one file!)

If you have dark frames at 3 or more different exposure times, it is good to check the linearity of the dark current using the results from 'imstat'. Plot the mean of the dark image minus the mean of the zeros against the exposure time. The points ought to fall on a straight line through the origin. The slope of the line times the gain should be consistent with the dark rate given in the GOT Observer's Guide for your operating temperature.

6. Combine the zeros

Section 3.4 of the User's Guide describes the use of the 'zerocombine' task. Edit the parameters in the standard way using 'epar'. The input name should be "@zeroframes.all" (without the quotes). The output name can be "Zero" or some variation. (It is a helpful convention to give combined images names starting with a capital letter.) The rest of the parameters should be set as shown in Figure 5. Keep in mind as you go that the figures in the Guide will not always correspond exactly to what you see because the Guide was written several IRAF versions ago.

Run the task. Your output should resemble Figure 5. Then examine your combined Zero using 'imstat' and the image display. Its properties should be consistent with those of the frames that were combined.

7. Combine the darks

This is where our procedure diverges substantially from the User's Guide. There are two reasons for this: first, our CCD does not operate at liquid nitrogen termperatures and so there is a substantial dark current that has to be subtracted explicitly; second, our so-called zero frames are not really zero-second exposures so we want to use them only where we are forced to. Consider first one of our fictitious 300 second object exposures. This contains 300 s worth of what the telescope actually saw, plus 300 s of dark current, plus the zero (bias) level. A 300 second dark frame contains 300 s of dark current plus the bias. So if we subtract the second from the first we get rid of both the dark current and the bias, despite the fact that we couldn't accurately measure the bias with our zero frames. Obviously what we really want to use is not a single dark frame but a combined version of of all the 300 s darks, but the idea is the same. Similarly, if we combine the 30 s darks to use with the 30 s object frames, and the 10 s darks to use with the 10 s object frames, we won't have to use the zero frames on them at all!

Then why, you ask, did we take the zeros in the first place? Because not all of our images are 10, 30, or 300 seconds long. In particular, the flats have non-standard exposure times. For these, we have to use the usual procedure implemented in 'ccdproc', which goes like this: First we compute a combined dark frame which is bias-subtracted using the combined Zero. This corrected dark count should scale linearly with the exposure time. Then the flat-field exposures are also zero-subtracted. Finally the dark count is scaled to the exposure time of each flat and subtracted. This procedure is slightly imperfect only because our zeros aren't really zeros; but for well-exposed flat fields the difference will be minuscule. (If you also have object frames that don't have darks of the same exposure time, treat them the same way we treat the flats in this example.)

So first you need to make a combined dark frame for each of your dark exposure times. In our example we have 10, 30, and 300 s darks, so we'll run 'darkcombine' 3 times. The parameters are shown in Figure 9. For the first run we'll make the input "@darkframes.10s" and the output "Dark10s". All the other parameters are as they are in Figure 9, EXCEPT that "process" must be set to "no". This is crucial. Run the task, then reset the input name to your next dark index file and the output name to something appropriate. For our example data we make 3 images, Dark10s.imh, Dark30s.imh, and Dark300s.imh.

Inspect these combined darks, and make sure they look reasonable.

Next, edit the parameters for 'ccdproc' (epar ccdproc). In the second group of parameters, that asks what corrections you want to apply, set everything to "no" except "zerocor" (zero level correction); set that to "yes". In the next section down, set "zero" (zero level calibration image) to the name of your combined zero. In our example this is Zero.imh. DO NOT run the task. Instead, exit the parameter editor with CTRL-D, saving the changes. (Don't exit with CTRL-C, because this won't save the changes.) Now go back and edit the parameters for 'darkcombine'. Set the input to "@darkframes.all", the output to "DarkZ" (for Zero-corrected), and set "process" to "yes". Exit with :go, running the task.

Inspect the zero-corrected combined dark for major problems. It's also interesting, by the way, to see what 'ccdlist' now says:

cc> ccdlist dark*.imh,Dark*.imh
dark01.imh[1530,1020][real][dark][][Z]:
dark02.imh[1530,1020][real][dark][][Z]:
dark03.imh[1530,1020][real][dark][][Z]:
dark04.imh[1530,1020][real][dark][][Z]:
dark05.imh[1530,1020][real][dark][][Z]:
dark06.imh[1530,1020][real][dark][][Z]:
dark07.imh[1530,1020][real][dark][][Z]:
dark08.imh[1530,1020][real][dark][][Z]:
dark09.imh[1530,1020][real][dark][][Z]:
Dark10s.imh[1530,1020][real][dark][]:
Dark30s.imh[1530,1020][real][dark][]:
Dark300s.imh[1530,1020][real][dark][]:
DarkZ.imh[1530,1020][real][dark][][Z]:

Notice the Z that has appeared in the last bracket, indicating those images that have been Zero corrected!

8. Dark- and zero-correct the flat fields

This step and the next are similar to what is described in section 3.5 of the User's Guide. Edit the parameters for 'ccdproc' (Figure 6). Make sure that "zerocor" and "darkcor" are turned on, all other corrections are turned off. The zero calibration image should be the same as in the previous step. Enter "DarkZ" for the dark count calibration image. The input image list is "@flatframes.all" and the output and ccdtype fields are blank. Run the task. Notice that the scaling of the dark correction is different for each image, depending on the exposure time.

If you have object frames of non-standard exposure times, repeat the above, using their index files for the input image list and leaving everything else the same.

9. Dark- and zero-correct the object frames

Now you want to process your object frames using the combined Dark of the correct exposure time. But first it's useful, if you haven't already done so, to look at some of your images. Particularly if you are observing faint objects, you will probably notice that your images seem to be full of "snow" and it's difficult to see anything. You're going to fix that now. Remember that in our example we have 10, 30, and 300 second exposures. Again epar ccdproc; leave the dark correction on but turn the zero correction and all other corrections off. Set the input image list to "@objframes.10s" and the dark calibration image to "Dark10s". Run the task. The check that you are running with the dark of the correct exposure time is that the dark scale factor should be 1.0 for all images.

Repeat, changing the input image list and dark calibration image name, for your other exposure times. 'ccdlist' should now indicate that the flats have been zero and dark [ZD] corrected, while the object frames are only [D]. Any object frames of non-standard exposure times should have been processed like the flats, so they should appear as [ZD].

If you examine your images now, you should find that the "snow" has disappeared, and you can actually start to see your faint targets!

10. Examine the flats

A bad flat field can ruin your science, so you need to give all of your flat field exposures a careful look with 'imstat' and the image display. The flats will have "shape" to them, reflecting the illumination pattern of the telescope and the sensitivity variation across the chip. You'll probably see the doughnut-shaped shadows of dust specks on the filters and the CCD window. It is normal for flats in different filters to look slightly different.

You may need to throw out overexposed images (pixel values approaching 60,000), underexposed images (pixel values under a few thousand), or images taken before the CCD was uniformly cool (look for funny stuff in the corners that diminishes in progressively later exposures). Stars in the twilight flats are not by themselves a problem, since they are usually rejected by 'flatcombine', but you will want to avoid exposures that have stars in the same place. If you need to throw out some frames, delete their names from the index files. But be judicious; obviously, you don't want to throw out so much data that you have nothing left.

11. Combine the flats

Now the flats need to be combined into one master flat for each filter (User's Guide section 3.8). If you have made the modifications to the image headers (sec. 2 above), and 'ccdlist' is listing all your flats and object exposures with the correct filters, then the bookkeeping will be done automatically.

Edit the parameters for flatcombine. They should look like the list in Figure 10, except for the following: the input list should be "@flatframes.all", "process" should be set to "no", and "scale" should be set to "none". Run the task; the output will show you what frames are being combined for each filter. In our example we have data in V and R, so we will get two combined flats, FlatV.imh and FlatR.imh.

Inspect the combined flats. They should still show the same basic shape, and many of the doughnut-shaped dust shadows will still be there. But there should not be any bright spots or things that look like stars or parts of stars. If there are, it probably means that you have combined images that have stars or cosmic rays in the same place, and you may need to edit your input list and run 'flatcombine' again to avoid this happening. Don't settle for "good enough" on your flats. Every imperfection in the flat will be permanently etched into your final data.

12. Flat-field division

The last basic reduction step is dividing your object frames by the normalized flat field of the appropriate filter. Edit the 'ccdproc' parameters again, and set all the corrections except flat-fielding to "no". Set the flat correction image name to "Flat*.imh", and the input image list to "@objframes.all". If you're nervous about having all the filters and flats match up the way they are supposed to, you can do a dry run by setting "noproc" to "yes". When you run the task it will list the processing steps but not do them. Then you can epar ccdproc again, reset "noproc" and run the task for real. There is a lot of arithmetic being done here, so it may take a while to process all of your images.

Finally, have a look at your data. If all the steps have been successful, you should see nice clean images of your objects, with beautiful flat skies and no "garbage". If that's what you have, give yourself a pat on the back and celebrate--- but not too much, because now you have to start on the science!

Last updated 2004 April 4. Written and maintained by Tom Statler