2. Calibration Database

The recipe_system.cal_service module includes a CalDB class (and subclasses called UserDB, LocalDB, and RemoteDB) that provide DRAGONS with ways to store and retrieve calibrations. Any number of databases can be used in a cascading manner, with the first one attached directly to the PrimitivesBASE object as the caldb attribute, and each subsequent database attached via the add_database() method.

2.1. Methods

Each CalDB object has the following methods:

  • add_database: append a new database to the end of the current cascade

  • get_calibrations: retrieve calibrations of a specified type for a list of input AstroData objects

  • set_calibrations: manually set a calibration for a list of input AD objects

  • unset_calibrations: manually remove a calibration for a list of input AD objects

  • clear_calibrations: clear all manually-set calibrations

  • store_calibration: store a calibration for later retrieval via association rules

These methods work not just on the database object upon which they are called, but also subsequent databases in the cascade. So a database’s get_calibrations method will pass on those AD objects for which it was unable to find a calibration and then assemble the complete set of result into an ordered list. If a processed calibration is passed, then store_calibration will write the AD object to disk in the appropriate calibrations/ subdirectory and then pass the filename onto each of the databases; if a processed science frame is passed, then the unmodified AD object is passed on.

2.2. The configuration file

DRAGONS has a global configuration file whose default location is ~/.dragons/dragonsrc. This can be overridden by the $DRAGONSRC environment variable, or specified via the config (-c) argument of reduce. When the appropriate subclass of PrimitivesBASE is instantiated, this config file is read. At the present time, the only section is related to calibrations and the format should be something like:

[calibs]
databases = ~/.dragons/
  fits.hi.gemini.edu

Databases are listed one per line, with indentation required as shown. These are parsed with the cal_service.parse_databases() function and, after a UserDB object is attached to the PrimitivesBASE, they are appended one-by-one in a cascade. A database is assumed to refer to an instance of a LocalDB if the file exists or its name contains a forward slash (but not two consecutive slashes), and a RemoteDB otherwise.

The default configuation for each database is retrieval only, so the storeCalibration primitive will not automatically add the calibration to any database. This can be changed by including store on the line after the name of the database, but get also needs to be included.

2.3. CalDB subclasses

2.3.1. The UserDB class

The UserDB class should always be the first database in a cascade. When instantiating the PrimitivesBASE object, the user calibrations in the ucals parameter are passed to this object, and these are returned if they match the requested calibration type. The UserDB class also handles user-defined calibration via a dictionary with entries of the form:

{(ad.calibration_key(), caltype): filename}

which can be defined via the setCalibration() primitive. This dictionary is cached to disk as a pickle in calibrations/calindex.pkl. The UserDB class is the only one for which the set, unset, and clear methods have any effect, while store does nothing.

Finally, UserDB also handles the standard MDFs for the instrument, which it locates via p.inst_lookups.

2.3.2. The LocalDB class

The LocalDB class is a lightweight layer over the LocalManager class for retrieving calibrations via a set of calibration association rules, and adding files stored on disk. Processed science files are not stored by this class.

This class has additional methods to provide an easy-to-use API. These are add_cal, add_directory, remove_cal, and list_files. There is also an init method to initialize a database but this can be done automatically when the LocalDB object is instantiated (and this is what happens during normal operations).

These methods are also accessible via the command-line caldb script. This script creates a LocalDB object by reading the config file and therefore requires that there is one and only one local database listed in that file. Alternatively, a database file can be specified via the -d option.

2.3.3. The RemoteDB class

The RemoteDB class is used for retrieving and storing calibrations via URLs and is currently very Gemini-specific. It could be subclassed but it seems unlikely that another observatory will have an interface that has enough in common to make subclassing preferable to creating a completely new class.

This class stores processed calibrations and processed science frames. Storage is only permitted when it is instantiated if the store flag is set in the config file and the appropriate string is in the uploads attribute of the PrimitivesBASE object. This class has a store_science parameter as well as the store_cal parameter it shares with LocalDB.