4.2. Example 2 - Separate CCDs - Using the “reduce” command line¶
This is a GMOS-N imaging observation of the galaxy Bootes V obtained in the r-band using the “reduce” command that is operated directly from the unix shell.
Instead of running the default recipe, we will run the recipe to reduce the CCDs separately instead of mosaicing them before the stack. Doing the reduction this way and not mosaicing the CCDs is used when the science objective require very accurate photometry that needs to take into account color-terms and the different color responses of the CCDs.
Just open a terminal and load the DRAGONS conda environment to get started.
4.2.1. The dataset¶
If you have not already, download and unpack the tutorial’s data package. Refer to Downloading the tutorial datasets for the links and simple instructions.
The dataset specific to this example is described in:
Here is a copy of the table for quick reference.
350 s, i-band
4.2.2. Set up the Calibration Service¶
4.2.3. Check files¶
For this example, all the raw files we need are in the same directory called
../playdata/example2. Let us learn a bit about the data we have.
Ensure that you are in the
playground directory and that the
environment that includes DRAGONS has been activated.
Let us call the command tool “typewalk”:
$ typewalk -d ../playdata/example2 directory: /Users/klabrie/data/tutorials/gmosimg_tutorial/playdata/example2 N20220613S0138.fits ............... (CAL) (FLAT) (GEMINI) (GMOS) (IMAGE) (NORTH) (RAW) (SIDEREAL) (TWILIGHT) (UNPREPARED) ... N20220613S0180.fits ............... (BIAS) (CAL) (GEMINI) (GMOS) (NORTH) (RAW) (UNPREPARED) ... N20220627S0115.fits ............... (GEMINI) (GMOS) (IMAGE) (NORTH) (RAW) (SIDEREAL) (UNPREPARED) ... N20220627S0222.fits ............... (BIAS) (CAL) (GEMINI) (GMOS) (NORTH) (RAW) (UNPREPARED) ... bpm_20220303_gmos-n_Ham_22_full_12amp.fits ................................... (BPM) (CAL) (GEMINI) (GMOS) (NORTH) (OVERSCAN_TRIMMED) (PREPARED) (PROCESSED) Done DataSpider.typewalk(..)
This command will open every FITS file within the folder passed after the
flag (recursively) and will print an unsorted table with the file names and the
associated tags. For example, calibration files will always have the
tag. Flat images will always have the
FLAT tag. This means that we can start
getting to know a bit more about our data set just by looking the tags. The
output above was trimmed for presentation.
4.2.4. Create File lists¶
This data set contains science and calibration frames. For some programs, it could have different observed targets and different exposure times depending on how you like to organize your raw data.
The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.
First, navigate to the
playground directory in the unpacked data package:
188.8.131.52. Lists of Biases¶
We are going to use two sets of biases, one for the science and one for the twilights. The reason for that is that the twilights and the science were obtained weeks apart and it is always safer to use biases that were obtained close in time with the data we want to use them on. It is also a good example to show you how to specify a date range in the dataselect expression.
Let’s first check the dates for the various observations.
$ showd -d object,ut_date ../playdata/example2/N*.fits -------------------------------------------------------------------------------- filename object ut_date -------------------------------------------------------------------------------- ../playdata/example2/N20220613S0138.fits Twilight 2022-06-13 ../playdata/example2/N20220613S0139.fits Twilight 2022-06-13 ../playdata/example2/N20220613S0140.fits Twilight 2022-06-13 ../playdata/example2/N20220613S0141.fits Twilight 2022-06-13 ../playdata/example2/N20220613S0142.fits Twilight 2022-06-13 ../playdata/example2/N20220613S0180.fits Bias 2022-06-13 ../playdata/example2/N20220613S0181.fits Bias 2022-06-13 ../playdata/example2/N20220613S0182.fits Bias 2022-06-13 ../playdata/example2/N20220613S0183.fits Bias 2022-06-13 ../playdata/example2/N20220613S0184.fits Bias 2022-06-13 ../playdata/example2/N20220627S0115.fits Disrupting UFD Candidate 2022-06-27 ../playdata/example2/N20220627S0116.fits Disrupting UFD Candidate 2022-06-27 ../playdata/example2/N20220627S0117.fits Disrupting UFD Candidate 2022-06-27 ../playdata/example2/N20220627S0118.fits Disrupting UFD Candidate 2022-06-27 ../playdata/example2/N20220627S0119.fits Disrupting UFD Candidate 2022-06-27 ../playdata/example2/N20220627S0222.fits Bias 2022-06-27 ../playdata/example2/N20220627S0223.fits Bias 2022-06-27 ../playdata/example2/N20220627S0224.fits Bias 2022-06-27 ../playdata/example2/N20220627S0225.fits Bias 2022-06-27 ../playdata/example2/N20220627S0226.fits Bias 2022-06-27
The science frames were obtained on 2022-06-27 and the twilights on 2022-06-13. We will create two lists, one of the biases obtained on each of those two days.
The bias files are selected with dataselect:
$ dataselect --tags BIAS ../playdata/example2/*.fits --expr="ut_date=='2022-06-13'" -o biastwi.lis $ dataselect --tags BIAS ../playdata/example2/*.fits --expr="ut_date=='2022-06-27'" -o biassci.lis
184.108.40.206. List of Flats¶
Now we build a list for the FLAT files:
$ dataselect --tags FLAT ../playdata/example2/*.fits -o flats.lis
If your dataset has flats obtained with more than one filter, you can add the
--expr 'filter_name=="r"' expression to get only the flats obtained within
the r-band. For example:
$ dataselect --tags FLAT --expr 'filter_name=="r"' ../playdata/example2/*.fits -o flats.lis
220.127.116.11. List for science data¶
The rest is the data with your science target. The simplest way, in this case, of creating a list of science frames is excluding everything that is a calibration:
$ dataselect --xtags CAL ../playdata/example2/*.fits -o sci.lis
This will work for our dataset because we know that a single target was observed with a single filter and with the same exposure time. But what if we don’t know that?
$ dataselect --expr 'observation_class=="science"' ../playdata/example2/*.fits | showd -d object,exposure_time ----------------------------------------------------------------------------------- filename object exposure_time ----------------------------------------------------------------------------------- ../playdata/example2/N20220627S0115.fits Disrupting UFD Candidate 350.0 ../playdata/example2/N20220627S0116.fits Disrupting UFD Candidate 350.0 ../playdata/example2/N20220627S0117.fits Disrupting UFD Candidate 350.0 ../playdata/example2/N20220627S0118.fits Disrupting UFD Candidate 350.0 ../playdata/example2/N20220627S0119.fits Disrupting UFD Candidate 350.0
To select on target name and exposure time, specify the criteria in the
expr field of “dataselect”:
$ dataselect --expr '(object=="Disrupting UFD Candidate" and exposure_time==350.)' ../playdata/example2/*.fits -o sci.lis
We have our input lists and we have initialized the calibration database, we are ready to reduce the data.
Please make sure that you are still in the
4.2.5. Bad Pixel Mask¶
Starting with DRAGONS v3.1, the bad pixel masks (BPMs) are now handled as calibrations. They are downloadable from the archive instead of being packaged with the software. They are automatically associated like any other calibrations. This means that the user now must download the BPMs along with the other calibrations and add the BPMs to the local calibration manager.
To add the static BPM included in the data package to the local calibration database:
caldb add ../playdata/example2/bpm*.fits
4.2.6. Create a Master Bias¶
We start the data reduction by creating the master biases for the science and the twilight data. Note that the reduction of the biases does not mosaic the biases and it keeps the CCDs separated, always. Because of that, the reduction of the biases for the “Separate CCDs” recipe is exactly the same as for the default recipe.
The biases are created and added to the calibration database using the commands below:
$ reduce @biastwi.lis $ reduce @biassci.lis
@ character before the name of the input file is the “at-file” syntax.
More details can be found in the "at-file" Facility documentation.
Because the database was given the “store” option in the
the processed bias will be automatically added to the database at the end of
To check that the master bias was added to the database, use
The file name of the output processed bias is the file name of the
first file in the list with
_bias appended as a suffix. This the
general naming scheme used by “reduce”.
If you wish to inspect the processed calibrations before adding them
to the calibration database, remove the “store” option attached to the
database in the
dragonsrc configuration file. You will then have to
add the calibrations manually following your inspection, eg.
caldb add N20220613S0180_bias.fits
The master bias will be saved in the same folder where reduce was
called and inside the
./calibrations/processed_bias folder. The latter
location is to cache a copy of the file. This applies to all the processed
4.2.7. Create a Master Flat Field¶
Twilight flats images are used to produce an imaging master flat and the result is added to the calibration database. Note that the reduction of the flats does not mosaic the flats and it keeps the CCDs separated, always. Because of that, the reduction of the flats for the “Separate CCDs” recipe is exactly the same as for the default recipe.
$ reduce @flats.lis
Note “reduce” will query the local calibration manager for the master bias and use it in the data reduction.
4.2.8. Create Master Fringe Frame¶
4.2.9. Reduce Science Images¶
Once we have our calibration files processed and added to the database, we can
reduce on our science data. Instead of using the default recipe, we
will explicitly call the recipe
$ reduce @sci.lis -r reduceSeparateCCDs
This recipe performs the standardization and corrections needed to convert the raw input science images into a stacked image. To deal with different color terms on the different CCDs, the images are split by CCD midway through the recipe and subsequently reduced separately. The relative WCS is determined from mosaicked versions of the images and then applied to each of the CCDs separately.
The stacked images of each CCD are in separate extension of the file
reduce -r display the image, you will notice that some sources
appear on two CCDs. This is because the each CCD has been stacked individually
and because of the dithers some sources ended up moving from to the adjacent
The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the “SCI” extension, the variance is in the “VAR” extension, and the data quality plane (mask) is in the “DQ” extension.
There is another similar recipe that can be used to reduce the CCDs
reduceSeparateCCDsCentral. The difference is that
the relative WCS is determined from the central CCD
(CCD2) and then applied to CCDs 1 and 3, while in
the whole image is used to adjust the WCS. The “Central” recipe can be
faster than the other but potentially less accurate if you do not have
a lot of sources in CCD2.