5. Metadata and Headers

Try it yourself

Download the data package (Try it yourself) if you wish to follow along and run the examples. Then

$ cd <path>/ad_usermanual/playground
$ python

You need to import Astrodata and the Gemini instrument configuration package.

>>> import astrodata
>>> import gemini_instruments

5.1. Astrodata Descriptors

We show in this chapter how to use the Astrodata Descriptors. But first let’s explain what they are.

Astrodata Descriptors provide a “header-to-concept” mapping that allows the user to access header information from a unique interface, regardless of which instrument the dataset is from. Like for the Astrodata Tags, the mapping is coded in a configuration package separate from core Astrodata. For Gemini instruments, that package is named gemini_instruments.

For example, if the user is interested to know the effective filter used for an observation, normally one needs to know which specific keyword or set of keywords to look at for that instrument. However, once the concept of “filter” is coded as a Descriptor, the user only needs to call the filter_name() descriptor to retrieve the information.

The Descriptors are closely associated with the Astrodata Tags. In fact, they are implemented in the same AstroData class as the tags. Once the specific AstroData class is selected (upon opening the file), all the tags and descriptors for that class are defined. For example, all the descriptor functions of GMOS data, ie. the functions that map a descriptor concept to the actual header content, are defined in the AstroDataGmos class.

This is all completely transparent to the user. One simply opens the data file and all the descriptors are ready to be used.

Note

Of course if the Descriptors have not been implemented for that specific data, they will not work. They should all be defined for Gemini data. For other sources, the headers can be accessed directly, one keyword at a time. This type of access is discussed below. This is also useful when the information needed is not associated with one of the standard descriptors.

To get the list of descriptors available for an AstroData object:

>>> ad = astrodata.open('../playdata/N20170609S0154.fits')
>>> ad.descriptors
('airmass', 'amp_read_area', 'ao_seeing', ...
  ...)

Most Descriptor names are readily understood, but one can get a short description of what the Descriptor refers to by calling the Python help function. For example:

>>> help(ad.airmass)
>>> help(ad.filter_name)

The full list of standard descriptors is available in the Appendix List of Gemini Standard Descriptors.

5.2. Accessing Metadata

5.2.1. Accessing Metadata with Descriptors

Whenever possible the Descriptors should be used to get information from headers. This allows for maximum re-usability of the code as it will then work on any datasets with an AstroData class.

Here are a few examples using Descriptors:

>>> ad = astrodata.open('../playdata/N20170609S0154.fits')

>>> #--- print a value
>>> print('The airmass is : ', ad.airmass())

>>> #--- use a value to control the flow
>>> if ad.exposure_time() < 240.:
...     print('This is a short exposure.')
... else:
...     print('This is a long exposure.')
...

>>> #--- multiply all extensions by their respective gain
>>> for ext, gain in zip(ad, ad.gain()):
...     ext *= gain
...

>>> #--- do arithmetics
>>> fwhm_pixel = 3.5
>>> fwhm_arcsec = fwhm_pixel * ad.pixel_scale()

The return values for Descriptors depend on the nature of the information being requested and the number of extensions in the AstroData object. When the value has words, it will be string, if it is a number it will be a float or an integer. The dataset used in this section has 4 extensions. When the descriptor value can be different for each extension, the descriptor will return a Python list.

>>> ad.airmass()
1.089
>>> ad.gain()
[2.03, 1.97, 1.96, 2.01]
>>> ad.filter_name()
'open1-6&g_G0301'

Some descriptors accept arguments. For example:

>>> ad.filter_name(pretty=True)
'g'

A full list of standard descriptors is available in the Appendix List of Gemini Standard Descriptors.

5.2.2. Accessing Metadata Directly

Not all header content is mapped to Descriptors, nor should it. Direct access is available for header content falling outside the scope of the descriptors.

One important thing to keep in mind is that the PHU (Primary Header Unit) and the extension headers are accessed slightly differently. The attribute phu needs to be used for the PHU, and hdr for the extension headers.

Here are some examples of direct header access:

>>> ad = astrodata.open('../playdata/N20170609S0154.fits')

>>> #--- Get keyword value from the PHU
>>> ad.phu['AOFOLD']
'park-pos.'

>>> #--- Get keyword value from a specific extension
>>> ad[0].hdr['CRPIX1']
511.862999160781

>>> #--- Get keyword value from all the extensions in one call.
>>> ad.hdr['CRPIX1']
[511.862999160781, 287.862999160781, -0.137000839218696, -224.137000839219]

5.2.3. Whole Headers

Entire headers can be retrieved as fits Header objects:

>>> ad = astrodata.open('../playdata/N20170609S0154.fits')
>>> type(ad.phu)
<class 'astropy.io.fits.header.Header'>
>>> type(ad[0].hdr)
<class 'astropy.io.fits.header.Header'>

In interactive mode, it is possible to print the headers on the screen as follows:

>>> ad.phu
SIMPLE  =                    T / file does conform to FITS standard
BITPIX  =                   16 / number of bits per data pixel
NAXIS   =                    0 / number of data axes
....

>>> ad[0].hdr
XTENSION= 'IMAGE   '           / IMAGE extension
BITPIX  =                   16 / number of bits per data pixel
NAXIS   =                    2 / number of data axes
....

5.3. Updating, Adding and Deleting Metadata

Header cards can be updated, added to, or deleted from the headers. The PHU and the extensions headers are again accessed in a mostly identical way with phu and hdr, respectively.

>>> ad = astrodata.open('../playdata/N20170609S0154.fits')

Add and update a keyword, without and with comment:

>>> ad.phu['NEWKEY'] = 50.
>>> ad.phu['NEWKEY'] = (30., 'Updated PHU keyword')

>>> ad[0].hdr['NEWKEY'] = 50.
>>> ad[0].hdr['NEWKEY'] = (30., 'Updated extension keyword')

Delete a keyword:

>>> del ad.phu['NEWKEY']
>>> del ad[0].hdr['NEWKEY']

5.4. World Co-ordinate System attribute

The wcs of an extension’s nddata attribute (eg. ad[0].nddata.wcs; see Pixel Data) may contain an instance of astropy.wcs.WCS (a standard FITS WCS object) or gwcs.wcs.WCS (a “Generalized WCS” or gWCS object). These both define a transformation between array indices and some other co-ordinate system such as “World” co-ordinates (see APE 14). GWCS allows multiple, almost arbitrary co-ordinate mappings from different calibration steps (eg. CCD mosaicking, distortion correction & wavelength calibration) to be combined in a single, reversible transformation chain — but this information cannot all be represented as a FITS standard WCS. If a gWCS object is defined here, it gets stored as a table extension named ‘WCS’ when the AstroData instance is saved to a file (with the same EXTVER as the corresponding ‘SCI’ array). This is independent of any WCS information already stored in the FITS headers. The representation in the table is produced using ASDF, with one line of text per row. Likewise, when the file is re-opened, the gWCS object gets recreated in wcs from the table.

In future, it is intended that the wcs attribute will get populated from standard FITS headers where there is no overriding ‘WCS’ table extension and will get saved to standard FITS headers when its type is astropy.wcs.WCS. Also, where a gWCS object is used, a discrete sampling of the World co-ordinate values will be stored as part of the FITS WCS, following Greisen et al. (2006), S6 (in addition to the definitive ‘WCS’ table), allowing standard FITS readers to report accurate World co-ordinates for each pixel.

5.5. Adding Descriptors [Advanced Topic]

For proper and complete instructions on how to create Astrodata Descriptors, the reader is invited to refer to the Astrodata Programmer Manual. Here we provide a simple introduction that might help some readers better understand Astrodata Descriptors, or serve as a quick reference for those who have written Astrodata Descriptors in the past but need a little refresher.

The Astrodata Descriptors are defined in an AstroData class. The AstroData class specific to an instrument is located in a separate package, not in astrodata. For example, for Gemini instruments, all the various AstroData classes are contained in the gemini_instruments package.

An Astrodata Descriptor is a function within the instrument’s AstroData class. The descriptor function is distinguished from normal functions by applying the @astro_data_descriptor decorator to it. The descriptor function returns the value(s) using a Python type, int, float, string, list; it depends on the value being returned. There is no special “descriptor” type.

Here is an example of code defining a descriptor:

class AstroDataGmos(AstroDataGemini):
    ...
    @astro_data_descriptor
    def detector_x_bin(self):
        def _get_xbin(b):
            try:
                return int(b.split()[0])
            except (AttributeError, ValueError):
                return None

        binning = self.hdr.get('CCDSUM')
        if self.is_single:
            return _get_xbin(binning)
        else:
            xbin_list = [_get_xbin(b) for b in binning]
            # Check list is single-valued
            return xbin_list[0] if xbin_list == xbin_list[::-1] else None

This descriptor returns the X-axis binning as a integer when called on a single extension, or an object with only one extension, for example after the GMOS CCDs have been mosaiced. If there are more than one extensions, it will return a Python list or an integer if the binning is the same for all the extensions.

Gemini has defined a standard list of descriptors that should be defined one way or another for each instrument to ensure the re-usability of our algorithms. That list is provided in the Appendix List of Gemini Standard Descriptors.

For more information on adding to Astrodata, see the Astrodata Programmer Manual.