4.1.1.5. recipe_system.utils package¶
4.1.1.5.1. Subpackages¶
4.1.1.5.2. Submodules¶
4.1.1.5.3. recipe_system.utils.decorators module¶
This module provides decorator functions for implementing the (somewhat TBD) parameter override policy of the prototype primitive classes. This implementation is subject to any change in policy.
Currently, the policy defines an order of parameter precedence:
- user parameters– as passed by -p on the reduce command line. These take full precedence over any other parameter setting. User parameters may be primitive-specific, as in -p makeFringeFrame:reject_method=jilt in which case, only that primitive’s parameter will be overridden. Other primitives with the same parameter (e.g. reject_method is also a parameter for stackFlats) will remain unaffected. If a user passes an un-specified parameter, as in reject_method=jilt, then any primitive with that parameter will receive that parameter value.
- recipe parameters – as passed on a recipe function call, like recipe_name(par1=val1). These will be overridden by same named user parameters.
- default parameters – The default parameter sets as defined in package parameter files. These will be overridden by recipe parameters and then user parameters when applicable.
This policy is implemented in the decorator function,
parameter_override
This is the primitive decorator and should be the only one decorating a primitive class.
This decorator is fully enhanced by the make_class_wrapper decorator, which maps the parameter_override decorator function to all public methods on the decorated class.
E.g.,:
from pkg_utilities.decorators import parameter_override
@parameter_override
class PrimitivesIMAGE(PrimitivesGMOS):
def __init__(self, adinputs, uparms={}):
[ . . . ]
-
recipe_system.utils.decorators.__top_level_primitive__()[source]¶ Check if we are at the top-level, not being called from another primitive.
We only want to capture provenance history when we are passing through the uppermost primitive calls. These are the calls that get made from the recipe.
-
recipe_system.utils.decorators.userpar_override(pname, args, upars)[source]¶ Implement user parameter overrides. In this implementation, user parameters always take precedence. Any user parameters passed to the Primitive class constructor, usually via -p on the ‘reduce’ command line, must override any specified recipe parameters.
Note: user parameters may be primitive-specific, i.e. passed as
-p makeFringeFrame:reject_method=’jilt’
in which case, the parameter will only be overridden for that primitive. Other primitives with the same parameter (e.g. stackFlats reject_method) will not be affected.
This returns a dict of the overridden parameters and their values.
4.1.1.5.4. recipe_system.utils.errors module¶
4.1.1.5.5. recipe_system.utils.findexe module¶
4.1.1.5.6. recipe_system.utils.mapper_utils module¶
Utility functions for Mappers.
find_user_recipe – searches for a user specified recipe, if any. dotpath() – build a python import path for dr packages.
-
recipe_system.utils.mapper_utils.dotpath(*args)[source]¶ Build an import path from args.
Parameters: args (<list>, implied by *) – a set of arguments of arbitrary length Returns: a dot path to an importable module Return type: <str>
-
recipe_system.utils.mapper_utils.find_user_recipe(dashr)[source]¶ Function receives the value of the reduce [-r, –recipe] flag, if passed. This will be a path to a recipe file and a dotted recipe name, which exists as a function in the recipe file. A properly specified user recipe shall contain one, and only one, dot operator.
If the recipefile.recipename cannot be found on the path, whether specified or implied (as cwd), then None is returned.
The string value of dashr will look like,
-r ‘/path/to/users/recipes/recipefile.recipe_function’
-r ‘recipefile.recipe_function’ – recipe file in cwd.
A recipe name with no dot operator implies a recipe name in the system recipe library.
Parameters: dashr (<str>) – a path to a recipe file dotted with a recipe function name. Returns: imported recipe function OR None Return type: <type ‘function’> or None
4.1.1.5.7. recipe_system.utils.md5 module¶
-
recipe_system.utils.md5.md5sum(filename)[source]¶ Generates the md5sum of the data in filename, returns the hex string.
-
recipe_system.utils.md5.md5sum_size_fp(fobj)[source]¶ Generates the md5sum and size of the data returned by the file-like object fobj, returns a tuple containing the hex string md5 and the size in bytes. f must be open. It will not be closed. We will read from it until we encounter EOF. No seeks will be done, fobj will be left at eof
4.1.1.5.8. recipe_system.utils.reduceActions module¶
This module provides a number “action” classes, subclassed from the argparse.Action class. These classes only override the __call__() method. This actions class library supplies ad hoc functionality to DPDG requirements on the reduce command line interface.
Action classes provided:
PosArgAction - positional argument BooleanAction - optional switches UnitaryArgumentAction - single value options ParameterAction - user parameters (-p, –param) CalibrationAction - user calibration services (–user_cal)
Becuase of requirements on the reduce interface, any new reduce options should specify one of these actions in the add_argument() call. But only one (1) PosArgAction should occur in a given parser.
These actions may be used in the add_argument() method call, such as,
- parser.add_argument(‘-f’, ‘–foo’, action=BooleanAction,
- help=”Switch on foo.”)
-
class
recipe_system.utils.reduceActions.BooleanAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶ Bases:
argparse.Action
-
class
recipe_system.utils.reduceActions.CalibrationAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶ Bases:
argparse.Action
-
class
recipe_system.utils.reduceActions.ParameterAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶ Bases:
argparse.Action
-
class
recipe_system.utils.reduceActions.PosArgAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶ Bases:
argparse.Action
-
class
recipe_system.utils.reduceActions.UnitaryArgumentAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶ Bases:
argparse.Action
4.1.1.5.9. recipe_system.utils.reduce_utils module¶
-
class
recipe_system.utils.reduce_utils.ReduceArgumentParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True)[source]¶ Bases:
argparse.ArgumentParserConverts an argument line from a user param file into an actual argument, yields to the calling parser.
-
class
recipe_system.utils.reduce_utils.ReduceHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)[source]¶ Bases:
argparse.RawDescriptionHelpFormatterReduceHelpFormatter class overrides default help formatting on customized reduce actions.
-
recipe_system.utils.reduce_utils.normalize_args(args)[source]¶ Convert argparse argument lists to single string values.
Parameters: args (<Namespace>) – argparse Namespace object or equivalent Returns: Same with converted types. Return type: <Namespace>
-
recipe_system.utils.reduce_utils.normalize_ucals(files, cals)[source]¶ When a user passes a –user_cal argument of the form,
–user_cal processed_bias:/path/to/foo.fits
The parser produces a user calibrations list like,
[‘processed_bias:/path/to/foo.fits’]
This list would pass to the Reduce __init__ as such, but, this function will translate and apply all user cals to all passed files.
{(ad.calibration_key(), ‘processed_bias’): ‘/path/to/foo.fits’}
This dictionary is of the same form as the calibrations dictionary for retrieved and stored calibrations.
User calibrations always take precedence over nominal calibration retrieval. User calibrations are not cached because they are not retrieved from fitsstore and are presumably on disk.
Parameters: Returns: normalz – a dictionary of the cal types applied to input files.
Return type: Example
- a returned dict,
- {(‘GS-2017A-Q-32-7-029’, ‘processed_flat’): ‘/path/to/XXX_flat.fits’}
-
recipe_system.utils.reduce_utils.normalize_upload(upload)[source]¶ For Recipe System v2.0, upload shall now be a list of things to send to fitsstore. E.g., $ reduce –upload metrics <file.fits> <file2.fits> $ reduce –upload metrics, calibs <file.fits> <file2.fits> $ reduce –upload metrics, calibs, science <file.fits> <file2.fits>
Result in
upload == [‘metrics’] upload == [‘metrics’, ‘calibs’] upload == [‘metrics’, ‘calibs’, ‘science’]
Parameters: upload (<list>) – upload argument received by the reduce command line. Returns: list of coerced or defaulted upload instructions. Return type: <list>
-
recipe_system.utils.reduce_utils.set_btypes(userparams)[source]¶ All cmd line args are delivered as strings. Find any user parameters that should be other python types and set them to those actual corresponding types.
I.e.,
‘None’ –> None ‘True’ –> True ‘False’ –> FalseParameters userparams: user parameters (if any) passed on the command line. Returns: A tuple of same parameters with converted None and boolean types. preserved with any specified primitive name. E.g., [(‘foo’,’bar’), (‘tileArrays:par1’,’val1’)] Return type: <list> of tuples.
4.1.1.5.10. recipe_system.utils.rs_utilities module¶
Set of functions in support of the recipe_system.
-
recipe_system.utils.rs_utilities.makedrpkg(pkgname, instruments, modes=None)[source]¶ Create the basic structure for a data reduction package that the recipe system will recognize.
Parameters: - pkgname (str) – Name of the new dr-package.
- instruments (list of str) – Name of the instrument(s) for which to create a directory structure.
- modes (list of str) – Name of the recipe modes that will be supported. Eg. modes = [‘sq’, ‘qa’] Default: [‘sq’] (‘sq’=science quality, ‘qa’=quality assessement)