3.2. Example 1 - Crowded with offset to sky - Using the “reduce” command line¶
This chapter will guide you on reducing GSAOI data using command line tools. In this example we reduce a GSAOI observation of the resolved outskirt of a nearby galaxy. The observation is a dither-on-target with offset-to-sky sequence. Just open a terminal to get started.
While the example cannot possibly cover all situations, it will help you get acquainted with the reduction of GSAOI data with DRAGONS. We encourage you to look at the Tips and Tricks and Issues and Limitations chapters to learn more about GSAOI data reduction.
DRAGONS installation comes with a set of scripts that are used to reduce astronomical data. The most important script is called “reduce”, which is extensively explained in the Recipe System Users Manual. It is through that command that a DRAGONS reduction is launched.
3.2.1. The dataset¶
If you have not already, download and unpack the tutorial’s data package. Refer to Downloading the tutorial datasets for the links and simple instructions.
The dataset specific to this example is described in:
Here is a copy of the table for quick reference.
Kshort-band, on target, 60 s
Lamp on, Kshort, for science
Lamp off, Kshort, for science
Kshort, standard star, 30 s
A master dark is not needed for GSAOI. The dark current is very low.
3.2.2. Set up the Calibration Service¶
3.2.3. Check files¶
For this example, all the raw files we need are in the same directory called
../playdata/example1. Let us learn a bit about the data we have.
Ensure that you are in the
playground directory and that the
environment that includes DRAGONS has been activated.
Let us call the command tool “typewalk”:
$ typewalk -d ../playdata/example1 directory: /data/workspace/gsaoiimg_tutorial/playdata/example1 S20170504S0114.fits ............... (GEMINI) (GSAOI) (IMAGE) (RAW) (SIDEREAL) (SOUTH) (UNPREPARED) ... S20170505S0030.fits ............... (AZEL_TARGET) (CAL) (DOMEFLAT) (FLAT) (GEMINI) (GSAOI) (IMAGE) (LAMPON) (NON_SIDEREAL) (RAW) (SOUTH) (UNPREPARED) ... S20170505S0060.fits ............... (AZEL_TARGET) (CAL) (DOMEFLAT) (FLAT) (GEMINI) (GSAOI) (IMAGE) (LAMPOFF) (NON_SIDEREAL) (RAW) (SOUTH) (UNPREPARED) ... S20170505S0095.fits ............... (GEMINI) (GSAOI) (IMAGE) (RAW) (SIDEREAL) (SOUTH) (UNPREPARED) ... S20170505S0110.fits ............... (GEMINI) (GSAOI) (IMAGE) (RAW) (SIDEREAL) (SOUTH) (UNPREPARED) Done DataSpider.typewalk(..)
This command will open every FITS file within the folder passed after the
flag (recursively) and will print an unsorted table with the file names and the
associated tags. For example, calibration files will always have the
tag. Flat images will always have the
FLAT tag. This means that we can
start getting to know a bit more about our data set just by looking at the tags.
The output above was trimmed for presentation.
3.2.4. Create File lists¶
This data set contains science and calibration frames. For some program, it could have different observed targets and different exposure times depending on how you like to organize your raw data.
The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.
The first step is to create input file lists. The tool “dataselect” helps with that. It uses Astrodata tags and “descriptors” to select the files and send the filenames to a text file that can then be fed to “reduce”. (See the Astrodata User Manual for information about Astrodata.)
First, navigate to the
playground directory in the unpacked data package:
126.96.36.199. A list for the flats¶
Let us create the list containing the domeflats:
$ dataselect --tags FLAT ../playdata/example1/*.fits -o flats_Kshort.list
We know that our dataset has only one filter (Kshort). If our dataset
contained data with more filters, we would have had to use the
option to select the appropriate filter as follows:
$ dataselect --tags FLAT --expr "filter_name=='Kshort'" ../playdata/example1/*.fits -o flats_Kshort.list
To see the name of the filter, use “showd” (show descriptor):
$ showd ../playdata/example1/*.fits -d filter_name ------------------------------------------------------------- filename filter_name ------------------------------------------------------------- ../playdata/example1/S20170504S0114.fits Kshort_G1105&Clear ... ...
188.8.131.52. A list for the standard star¶
In this case we have only one standard star. Indeed, we can confirm that by selecting on partner calibrations and showing the object name:
$ dataselect --expr 'observation_class=="partnerCal"' ../playdata/example1/*.fits | showd -d object ------------------------------------------------- filename object ------------------------------------------------- ../playdata/example1/S20170504S0114.fits 9132 ../playdata/example1/S20170504S0115.fits 9132 ../playdata/example1/S20170504S0116.fits 9132 ../playdata/example1/S20170504S0117.fits 9132
If we had more than one object, a list for each standard star is created by
object descriptor as a selection criterium in “dataselect”:
$ dataselect --expr 'object=="9132"' ../playdata/example1/*.fits -o std_9132.list
184.108.40.206. A list for the science observations¶
The rest is the data with your science target. Before we create a new list, let us check that indeed we have only one science target and a unique exposure time:
$ dataselect --expr 'observation_class=="science"' ../playdata/example1/*.fits | showd -d object,exposure_time ------------------------------------------------------------------ filename object exposure_time ------------------------------------------------------------------ ../playdata/example1/S20170505S0095.fits NGC5128 60.0 ../playdata/example1/S20170505S0096.fits NGC5128 60.0 ... ../playdata/example1/S20170505S0109.fits NGC5128 60.0 ../playdata/example1/S20170505S0110.fits NGC5128 60.0
Just to demonstrate how expression are built, let us consider that we need to
select only the files for which
object is NGC5128 and
is 60 seconds. We also want to pass the output to a new list:
$ dataselect --expr '(observation_class=="science" and exposure_time==60.)' ../playdata/example1/*.fits -o science.list
3.2.5. Bad Pixel Mask¶
Starting with DRAGONS v3.1, the bad pixel masks (BPMs) are now handled as calibrations. They are downloadable from the archive instead of being packaged with the software. They are automatically associated like any other calibrations. This means that the user now must download the BPMs along with the other calibrations and add the BPMs to the local calibration manager.
To add the static BPM included in the data package to the local calibration database:
caldb add ../playdata/example1/bpm*.fits
3.2.6. Create a Master Flat Field¶
The GSAOI Kshort master flat is created from a series of lamp-on and lamp-off dome exposures. They should all have the same exposure time. Each flavor is stacked (averaged), then the lamp-off stack is subtracted from the lamp-on stack and the result normalized.
We create the master flat field and add it to the calibration manager as follows:
$ reduce @flats_Kshort.list
@ character before the name of the input file is the “at-file” syntax.
More details can be found in the "at-file" Facility documentation.
Because the database was given the “store” option in the
the processed dark will be automatically added to the database at the end of
The file name of the output processed flat is the file name of the
first file in the list with
_flat appended as a suffix. This the
general naming scheme used by “reduce”.
- If you wish to inspect the processed calibrations before adding them
- to the calibration database, remove the “store” option attached to the
database in the
dragonsrcconfiguration file. You will then have to add the calibrations manually following your inspection, eg.
caldb add S20170505S0030_flat.fits
The master flat will be saved in the same folder where reduce was
called and inside the
./calibrations/processed_flat folder. The latter
location is to cache a copy of the file. This applies to all the processed
Here is an example of a master flat:
Note that this figure shows the masked pixels in white color but not all the detector features are masked. For example, the “Christmas Tree” on detector 2 can be easily noticed but was not masked.
3.2.7. Reduce Standard Star¶
The standard star is reduced essentially the same way as the science target (next section). The processed flat field that we added earlier to the local calibration database will be fetched automatically. Also, in this case the standard star was obtained using ROIs (Regions-of-Interest) which do not match the flat field. The software will recognize that the flat field is still valid and will crop it to match the ROIs.
$ reduce @std_9132.list
reduce command will automatically align and stack the images.
Therefore, it is no longer necessary to use the
disco_stu tool for
3.2.8. Reduce the Science Images¶
This is an observation of a galaxy with offset to sky. We need to turn off the additive offsetting of the sky because the target fills the field of view and does not represent a reasonable sky background. If the offsetting is not turned off in this particular case, it results in an over-subtraction of the sky frame.
Unlike the other near-IR instruments, the additive
parameter is used by default to adjust the sky frame background for
GSAOI instead of the multiplicative
scale_sky parameter. It was
found to work better when the sky background per pixel is very low,
which is common due to the short exposure time needed to avoid
saturating stars and the small pixel scale. The reader is encouraged
to experiment with
offset_sky does not seem to
lead to an optimal sky subtraction.
(Remember that when the source is extended, both parameters normally need to be turned off.)
The sky frame comes from off-target sky observations. We feed the pipeline all the on-target and off-target frames. The software will split the on-target and the off-target appropriately using information in the headers.
Once we have our calibration files processed and added to the database, ready
for retrieval, we can run
reduce on our science data.
$ reduce @science.list -p skyCorrect:offset_sky=False
This command will generate flat corrected files, align them,
stack them, and orient them such that North is up and East is left. The final
image will have the name of the first file in the set, with the suffix
_image. The on-target files are the ones that have been flat corrected
_flatCorrected), and scaled (
_countsScaled). There should be nine
The figure above shows the final flat-corrected, aligned, and stacked frame.
For absolute distortion correction and astrometry,
reduce can use a
reference catalog provided by the user. Without a reference catalog, like
above, only the relative distortion between the frames is accounted for.
The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the “SCI” extension, the variance is in the “VAR” extension, and the data quality plane (mask) is in the “DQ” extension.