3. Reduction using API

There may be cases where you would be interested in accessing the DRAGONS’ Application Program Interface (API) directly instead of using the command line wrappers to reduce your data. Here we show you how to do the same reduction we did in the previous chapter but using the API.

3.1. The dataset

If you have not already, download and unpack the tutorial’s data package. Refer to Downloading the tutorial datasets for the links and simple instructions.

The dataset specific to this example is described in:

Here is a copy of the table for quick reference.

Science
N20170614S0201-205
10 s, i-band
Bias
N20170613S0180-184
N20170615S0534-538
 
Twilight Flats
N20170702S0178-182
40 to 16 s, i-band

3.2. Setting Up

3.2.1. Importing Libraries

We first import the necessary modules and classes:

1
2
3
4
5
import glob

from gempy.adlibrary import dataselect
from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce

Importing print_function is for compatibility with the Python 2.7 print statement. If you are working with Python 3, it is not needed, but importing it will not break anything.

glob is Python built-in packages. It will be used to return a list with the input file names.

dataselect will be used to create file lists for the darks, the flats and the science observations. The cal_service package is our interface with the local calibration database. Finally, the Reduce class is used to set up and run the data reduction.

3.2.2. Setting up the logger

We recommend using the DRAGONS logger. (See also Double messaging issue.)

8
9
from gempy.utils import logutils
logutils.config(file_name='gmos_data_reduction.log')

3.2.3. Setting up the Calibration Service

Before we continue, let’s be sure we have properly setup our calibration database and the calibration association service.

First, check that you have already a rsys.cfg file inside the ~/.geminidr/. It should contain:

[calibs]
standalone = True
database_dir = /path_to_my_data/gmosimg_tutorial_api/playground

This tells the system where to put the calibration database. This database will keep track of the processed calibrations as we add them to it.

Note

The tilde (~) in the path above refers to your home directory. Also, mind the dot in .geminidr.

The calibration database is initialized and the calibration service is configured as follow:

10
11
12
13
14
caldb = cal_service.CalibrationService()
caldb.config()
caldb.init()

cal_service.set_calservice()

The calibration service is now ready to use. If you need more details, check the Using the caldb API in the Recipe System User’s Manual .

3.3. Create list of files

The next step is to create lists of files that will be used as input to each of the data reduction steps. Let us start by creating a list of all the FITS files in the directory ../playdata/.

15
16
all_files = glob.glob('../playdata/*.fits')
all_files.sort()

The sort() method simply re-organize the list with the file names and is an optional step. Before you carry on, you might want to do print(all_files) to check if they were properly read.

Now we can use the all_files list as an input to select_data(). The dataselect.select_data() function signature is:

select_data(inputs, tags=[], xtags=[], expression='True')

3.3.1. List of Biases

Let us, now, select the files that will be used to create a master bias:

17
18
19
20
21
list_of_biases = dataselect.select_data(
    all_files,
    ['BIAS'],
    []
)

Note the empty list [] in line 20. This positional argument receives a list of tags that will be used to exclude any files with the matching tag from our selection (i.e., equivalent to the --xtags option).

3.3.2. List of Flats

Next we create a list of twilight flats for each filter. The expression specifying the filter name is needed only if you have data from multiple filters. It is not really needed in this case.

22
23
24
25
26
27
list_of_flats = dataselect.select_data(
     all_files,
     ['FLAT'],
     [],
     dataselect.expr_parser('filter_name=="i"')
)

3.3.3. List of Science Data

Finally, the science data can be selected using:

27
28
29
30
31
32
list_of_science = dataselect.select_data(
    all_files,
    [],
    ['CAL'],
    dataselect.expr_parser('(observation_class=="science" and filter_name=="i")')
)

Here we left the tags argument as an empty list and passed the tag 'CAL' as an exclusion tag through the xtags argument.

We also added a fourth argument which is not necessary for our current dataset but that can be useful for others. It contains an expression that has to be parsed by expr_parser(), and which ensures that we are getting science frames obtained with the i-band filter.

3.4. Make Master Bias

We create the master bias and add it to the calibration manager as follow:

33
34
35
36
37
reduce_bias = Reduce()
reduce_bias.files.extend(list_of_biases)
reduce_bias.runr()

caldb.add_cal(reduce_bias.output_filenames[0])

The Reduce class is our reduction “controller”. This is where we collect all the information necessary for the reduction. In this case, the only information necessary is the list of input files which we add to the files attribute. The runr() method is where the recipe search is triggered and where it is executed.

Once runr() is finished, we add the master bias to the calibration manager (line 37).

3.5. Make Master Flat

We create the master flat field and add it to the calibration database as follow:

38
39
40
41
42
reduce_flats = Reduce()
reduce_flats.files.extend(list_of_flats)
reduce_flats.runr()

caldb.add_cal(reduce_flats.output_filenames[0])

3.6. Make Master Fringe Frame

Warning

The dataset used in this tutorial does not require fringe correction so we skip this step. To find out how to produce a master fringe frame, see Create Master Fringe Frame in the Tips and Tricks chapter.

3.7. Reduce Science Images

We use similar statements as before to initiate a new reduction to reduce the science data:

43
44
45
reduce_science = Reduce()
reduce_science.files.extend(list_of_science)
reduce_science.runr()

The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the “SCI” extension, the variance is in the “VAR” extension, and the data quality plane (mask) is in the “DQ” extension.