5.1.1.2. recipe_system.cal_service package

recipe_system.cal_service.get_calconf()[source]

recipe_system.cal_service.get_db_path_from_config()[source]

Read the path of the local database specified in the config file. An error will be raised if there is no such database, or more than one. This function is used by the “caldb” script and the set_local_database() function here.

5.1.1.2. Parameters

config: str: name of the configuration file

5.1.1.2. Returns

db_pathstr: the path to the local database file

recipe_system.cal_service.init_calibration_databases(inst_lookups=None, procmode=None, ucals=None, upload=None)[source]

Initialize the calibration databases for a PrimitivesBASE object.

5.1.1.2. Parameters

inst_lookupsstr: local of the instrument lookups package (for the MDF lookup table)
ucalsdict: user calibrations
uploadlist: things to upload (we’re concerned about “calibs” and “science”)

5.1.1.2. Returns

A UserDB object, possibly linked to additional CalDB objects

recipe_system.cal_service.parse_databases(default_dbname='cal_manager.db')[source]

Parse the databases listed in the global config file. This returns a list provided information on how to build the cascase of databases, but does not instantiate any CalDB objects, so it can be used by the caldb script efficiently.

5.1.1.2. Parameters

default_dbnamestr: default name of database file (if only a directory is listed in the config file)

5.1.1.2. Returns

list of tuples (class, database name, kwargs)

recipe_system.cal_service.set_local_database()[source]: User helper function to define a local calibration database based on the “dragonsrc” config file.

5.1.1.2. Returns

A LocalDB object

5.1.1.2.1. Subpackages

5.1.1.2.1.1. recipe_system.cal_service.tests package

5.1.1.2.2. Submodules

5.1.1.2.3. recipe_system.cal_service.caldb module

class recipe_system.cal_service.caldb.CalDB(name=None, get_cal=True, store_cal=False, valid_caltypes=None, procmode=None, log=None)[source]

Bases: object

The parent class for all types of calibration database handlers.

5.1.1.2. Attributes

namestr: a name for this database (gets reported with the calibration, so there is some provenance)
loglogger object: the logger object to report to
get_calbool: allow retrievals from this database?
store_calbool: allow new calibrations to be stored to this database?
caldirstr: directory under which calibrations will be stored (either when createed, or when retrieved from a RemoteDB)
procmodestr: the default processing mode that retrieved calibrations must be suitable for
nextdbCalDB object/None: a place to send unfulfilled calibration requests
_valid_caltypeslist of str: valid calibration types for retrieval/storage

add_database(db)[source]

Adds a database to the end of the cascade. It does this recursively, calling this method on the next database in the casecade, until it gets to the end.

5.1.1.2. Parameters

dbCalDB instance: the database to add to the end of this cascade

clear_calibrations(*args, **kwargs)[source]

get_calibrations(adinputs, caltype=None, procmode=None, howmany=1)[source]

Returns the requested calibration type for the list of files provided.

5.1.1.2. Parameters

adinputslist of AstroData objects: the files requiring calibrations
caltypestr: the type of calibration required
procmodestr/None: minimum quality of processing needed for the calibration
howmanyint: maximum number of calibrations to return (for backwards compat). Use howmany>1 at your own risk!

5.1.1.2. Returns

CalReturnthe calibration filenames and the databases which: provided them

set_calibrations(*args, **kwargs)[source]

store_calibration(cal, caltype=None)[source]

Handle storage of calibrations. For all types of calibration (NOT processed_science) this saves the file to disk in the appropriate calibrations/ subdirectory (if store=True) and then passes the filename down the chain. For processed_science files, it passes the AstroData object.

5.1.1.2. Parameters

calsstr/AstroData: the calibration or its filename
caltypestr: type of calibration

unset_calibrations(*args, **kwargs)[source]

class recipe_system.cal_service.caldb.CalReturn(*args)[source]

Bases: object

A simple class to store files and origins (i.e., which CalDB instance provided the calibration) for calibration requests. The class can be instantiated with either a list of files and a list of origins, or a single list of 2-tuples (file, origin), where a single None can be used in place of (None, None).

Public attributes are: files : list of files (or list of lists of files) origins : list of origins

There is also an items() method that returns these lists as zipped pairs, like the dict.items() method. Iteration is set up to return first the complete list of files, then the complete list of origins, which helps with coding the cheeseCorrect() primitives.

items()[source]

recipe_system.cal_service.caldb.cascade(fn)[source]: Decorates methods such that, after execution, the method with the same name is called on the next CalDB instance in the cascade, with the same arguments.

5.1.1.2.4. recipe_system.cal_service.calrequestlib module

class recipe_system.cal_service.calrequestlib.CalibrationRequest(ad, caltype=None, procmode=None)[source]

Bases: object

Request objects are passed to a calibration_search() function

as_dict()[source]

recipe_system.cal_service.calrequestlib.generate_md5_digest(filename)[source]

recipe_system.cal_service.calrequestlib.get_cal_requests(inputs, caltype, procmode=None, is_local=True)[source]

Builds a list of CalibrationRequest objects, one for each ad input.

5.1.1.2. Parameters

inputs: <list>: A list of input AstroData instances.
caltype: <str>: Calibration type, eg., ‘processed_bias’, ‘flat’, etc.

5.1.1.2. Returns

rq_events: <list>: A list of CalibrationRequest instances, one for each passed

‘ad’ instance in ‘inputs’.

5.1.1.2.5. recipe_system.cal_service.calurl_dict module

5.1.1.2.6. recipe_system.cal_service.file_getter module

class recipe_system.cal_service.file_getter.CachedFileGetter[source]

Bases: object

This class implements a file getter with an optional system- (or user-) wide cache. To enable the cache, set system_calcache_dir to point at the cache directory in the [calibs] section of your dragonsrc. If this is unset or None, no caching is attempted. This cache is only really useful on machines which run lots of instances of reduce in separate working directories, notably for archive processing.

This class is not a singleton, and we don’t store any cache concurrency data in the class, we check file existence and compute md5 digests every time we service a request.

get_request(url, filename, calmd5)[source]

exception recipe_system.cal_service.file_getter.GetterError(messages)[source]: Bases: Exception

recipe_system.cal_service.file_getter.get_request(url, filename, calmd5)[source]: This function was moved from calrequestlib.py as it belongs more naturally in this module as it is the only thing that calls methods in this module. Those methods where replaced with the CachedFileGetter class. :param url: URL to fetch data from :param filename: Filename to put data into :param calmd5: md5 hexdigest of cal we are requesting :return: None

5.1.1.2.7. recipe_system.cal_service.localdb module

class recipe_system.cal_service.localdb.LocalDB(dbfile, name=None, valid_caltypes=None, procmode=None, get_cal=True, store_cal=True, log=None, force_init=False)[source]

Bases: CalDB

The class handling a calibration database stored on disk, via the LocalManager class. In addition to the methods required to interface with DRAGONS data reduction pipelines, other methods are used to provide a full API and effect actions of the “caldb” script.

An attempt to create an instance of this class without the LocalManager being importable will result in an error.

5.1.1.2. Attributes

dbfilestr: name of the file on disk holding the database
_calmgrLocalManager instance: the local calibration manager that will handle the requests

add_cal(calfile)[source]

add_directory(path, walk=False)[source]

Ingest one or more files from a given directory, optionally searching all subdirectories. This is not used by primitives in the DRAGONS data reduction pipelines.

5.1.1.2. Parameters

pathstr: directory containing files to be ingested
walkbool: add files from all subdirectories?

init(wipe=False)[source]

Initialize a calibration database. Callers will usually only want to do this once. But if called again, it will wipe the old database.

5.1.1.2. Parameters

wipe<bool>, optional: If the database exists and this parameter is True, the file will be removed and recreated before initializing

5.1.1.2. Raises

IOError: If the file exists and there a system error when trying to remove it (eg. lack of permissions).
LocalManagerError: If the file exists and wipe was False

list_files()[source]

List all files in the local calibration database. This is not used by primitives in the DRAGONS data reduction pipelines.

5.1.1.2. Returns

LocalManager.list_files: <generator>.: (See class docstring for example of how to use this generator.)

5.1.1.2. Raises

LocalManagerError: Raised when unable to read database.

remove_cal(calfile)[source]

Removes a calibration file from the database. Note that only the filename is relevant. All duplicate copies in the database will be removed.

5.1.1.2. Parameters

calfile<str>: Path to the file. It can be either absolute or relative

5.1.1.2.8. recipe_system.cal_service.localmanager module

5.1.1.2.9. recipe_system.cal_service.remotedb module

class recipe_system.cal_service.remotedb.RemoteDB(server, name=None, valid_caltypes=None, get_cal=True, store_cal=False, store_science=False, procmode=None, log=None, upload_cookie=None)[source]

Bases: CalDB

The class for remote calibration databases. It inherits from CalDB, but also has the following attributes:

5.1.1.2. Attributes

serverstr: URL of the server
store_sciencebool: whether processed science images should be uploaded
_upload_cookiestr: the cookie to send when uploading files
_calmgrstr: the URL for making requests to the remote calibration manager
_proccal_url, _science_urlstr: the URLs for uploading processed calibrations and processed science images, respectively.

recipe_system.cal_service.remotedb.retrieve_calibration(rqurl, rq, howmany=1)[source]

5.1.1.2.10. recipe_system.cal_service.userdb module

class recipe_system.cal_service.userdb.UserDB(name=None, get_cal=True, mdf_dict=None, mdf_key=None, user_cals=None, valid_caltypes=None, log=None)[source]

Bases: CalDB

The class handling the User-defined database, which can contain: (a) a filename for each caltype that should be returned whenever that

caltype is requested, irrespective of the AD “making” the request

a filename to be returned for a given caltype for a specific AD (actually, a specific ad.calibration_key(), which is normally just the data label)

This class also handles the standard MDFs when a “mask” is requested.

5.1.1.2. Attributes

user_calsdict: A dict of {caltype: filename} as described in (a) above
user_cachedict: A dict of {(calibration_key, caltype): filename} as described in (b) This is cached to disk as a pickle.
mdf_dictdict: A dict of {mdf_key: filename} for the standard focal plane masks
mdf_keytuple: A tuple of strings representing the attributes of the astrodata object to concatenate to form the keys of mdf_dict.

load_cache()[source]

save_cache()[source]