1. Data Reduction Packages

In the context of the Recipe System, a Data Reduction Package is a python package containing one or more subpackages, each providing instrument-specific sets of data processing primitive classes and recipe libraries. These packages provide attributes on certain components of the package, which make them discoverable by the Recipe System. Developers are entirely free to build their own data reduction package, or “dr-package.”

As stated at the beginning of Chapter 4, the default and only data reduction package provided by DRAGONS is geminidr. This package is included in the DRAGONS distribution. Unless specifed otherwise, it is the geminidr package that serves targets for the Recipe System mappers. Readers are encouraged to examine the geminidr package to familiarize themselves with components.

1.1. Building a new Data Reduction Package

The Recipe System mappers require the primitives and the recipes to be organized under a specific directory structure. To help build that structure the Recipe System provides a script called makedrpkg that we will introduce later in this section. First, we address the requirements.

  • The Data Reduction (DR) package must be importable. (It must have a __init__.py.)

  • The DR package must be found in one of the sys.path directories.

  • The instrument packages must be found at the first subdirectory level in the DR package.

  • The instrument packages (directory name) must be named after the instrument they associate with. The instrument package name must match the lower case version of the AstroData descriptor instrument for the data it supports.

  • The recipes must be in a subdirectory of the instrument package. That directory must be named recipes. That name is hardcoded in the moduleattribute RECIPEMARKER in utils.mapper_utils.

  • The recipes must be assigned a mode (one of “sq”, “qa”, “ql”).

  • The mode-specific recipes must be located in a subdirectory of recipes. That directory must be named to match the mode.

  • The recipes directory and the mode directories but all have an __init__.py in them and be importable.

The directory structure can be created by hand but to simplify the process and avoid mistakes, it is recommended to use the makedrpkg script provided with the Recipe System. The script is used from a normal terminal, not from Python. Here is a few usage examples.

Get help:

% makedrpkg -h

Create mydrpkg DR package with a coolinstrument instrument package and a sq mode subdirectory:

% makedrpkg mydrpkg -i coolinstrument -m sq

Same as above but with both sq and qa modes. Note that if a directory already exists it will just be skipped.

% makedrpkg mydrpkg -i coolinstrument -m sq qa

Add two instrument packages with qa mode:

% makedrpkg mydrpkg -i instA instB -m qa

Add a sq mode to an existing instrument package:

% makedrpkg mydrpkg -i instA -m sq

Once you have that structure in place, the primitives and the parameters modules go in the instrument package main directory, and the recipes in the recipes/<mode> directory.

1.2. Using a third-party Data Reduction Package

To activate a specific DR package, the drpkg attribute or option (depending on what is being used) needs to be set. The default setting is geminidr.

1.2.1. From the reduce command line tool

From the reduce command line tool one uses the --drpkg option. For example:

% reduce *.fits --drpkg mydrpkg

1.2.2. From the Reduce class API

When using the Reduce class API, the attribute to set is drpkg. For example:

>>> from recipe_system.reduction.coreReduce import Reduce
>>> reduce = Reduce()
>>> reduce.drpkg = 'mydrpkg'

1.2.3. From the mappers

When using the mappers directly, again the attribute to set in either mapper, PrimitiveMapper or RecipeMapper, is drpkg. For example:

>>> from recipe_system.mappers.primitiveMapper import PrimitiveMapper
>>> from recipe_system.mappers.recipeMapper import RecipeMapper
>>> tags = ad.tags
>>> instpkg = ad.instrument(generic=True).lower()
>>> pmapper = PrimitiveMapper(tags, instpkg, drpkg='mydrpkg')
>>> rmapper = RecipeMapper(tags, instpkg, drpkg='mydrpkg')

1.3. Requirements for Primitives and Recipes

Instructions on how to write primitives and recipes is beyond the scope of this manual. However, the Recipe System does impose some requirements on the primitives and the recipes. We review them here.

1.3.1. Requirements on Recipes

  • A recipe library must contain a module attribute named recipe_tags that contains a Python set of the AstroData tags applicable to the library.

  • A recipe library must contain a module attribute named _default that sets the name of the default recipe for this library. The _default attribute needs to be set below the recipe function definition for Python to pick it up.

  • A recipe signature must accept the primitive set as the first argument with no other “required” arguments. Other arguments must be optional.

1.3.2. Requirements on Primitives

The requirements on the primitives are highly Gemini centric for the moment.

  • A primitive class must inherit PrimitiveBASE class defined in geminidr/__init__py, or bring a copy of it in a third-party DR package.

  • A primitive class must be decorated with the parameter_override decorator located in recipe_system.utils.decorators.

  • The Gemini fork of LSST pexconfig package, gempy.library.config, must be used to handle input parameters.

  • The signature of the __init__ of primitive class should be:

    def __init__(self, adinputs, **kwargs):

  • The signature of a primitive, a method of a primitive classe should be:

    def primitive_name(self, adinputs=None, **params)

  • Primitive parameter must be defined in a class matching the name of the primitive followed by “Config”. For example:

    class primitive_nameConfig(any_other_parameter_class_to_inherit):
    param1 = config.Field("Description of param", <type>, <default_value>)

  • Each primitive class must define a tagset attribute containing a set of AstroData tags identifying which data this primitive class is best suited for. The tags are string literals. This tagset attribute is what the primitive mapper uses to find the most appropriate primitive class.

Here is an example putting most of the above requirements to use:

from geminidr.core import Image, Photometry
from .primitives_gmos import GMOS
from . import parameters_gmos_image
from recipe_system.utils.decorators import parameter_override

@parameter_override
class GMOSImage(GMOS, Image, Photometry):

    tagset = set(["GEMINI", "GMOS", "IMAGE"])

    def __init__(self, adinputs, **kwargs):
        super().__init__(adinputs, **kwargs)
        self._param_update(parameters_gmos_image)

    def some_primitive(self, adinputs=None, **params):
        [...]
        return adinputs

It is technically possible to decide not to use the parameter_override decorator. In that case, there will be no transparent passing of the input AstroData objects, all primitives will have to have their parameter explicitely defined when called, the primitive class will have to have exact signature the get_applicable_primitives uses when initializing the class. This is not a mode we have experimented with and there might be additional limitations.

The primitive class signature must be able to accept this instantiation call:

primitive_actual(self.adinputs, mode=self.mode, ucals=self.usercals,
                 uparms=self.userparams, upload=self.upload)

 adinputs: Python list of AstroData objects
 mode:     One of 'sq', 'qa', or 'ql'
 ucals:    Python dict with format
              {(<data_label>, <type_of_calib>): <calib_filename>},
              one key-value pair for each input, with the type of
              calibration matching one from the keys in
              cal_service.caldb.REQUIRED_TAG_DICT.
 uparms:   Python dict with format ``{'<prim>:<param>': <value>}``
 upload:   Python list of any combination of 'calibs', 'metrics', or
              'science'.

1.3.3. Requirement to use AstroData

The Recipe System expects to work on AstroData objects. In particular, it requires the tags to be defined and the instrument() descriptor to be defined.

Therefore, for data from a new, still unsupported instrument, the first step is to write the AstroData configuration layer. In DRAGONS, the AstroData configuration layers are found in the package gemini_instruments. The convention is to name the modules with the tags and descriptor adclass.py. This is just a convention.

For more information on the AstroData configuration, see the Astrodata Programmer Manual.