Next: OAPI: the OPUS Application Programming Interface
Up: Data Pipelines
Previous: On-The-Fly Calibration at STScI
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Micol, A., Durand, D., Pirenne, B., Gaudet, S., & Hodge, P. E. 1999, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, eds. D. M. Mehringer, R. L. Plante, & D. A. Roberts (San Francisco: ASP), 191

On The Fly Re-calibration of NICMOS and STIS Data

Alberto Micol1, Daniel Durand2, Benoît Pirenne3, Séverin Gaudet4, Phil Hodge5


In 1997 the HST payload was rejuvenated with two new instruments. After an initial period during which the instrument in-flight responses were assessed and understood and the calibration software and data flow stabilised, the CADC and ST-ECF started working together on the extension of the on-the-fly (OTF) calibration pipeline to support NICMOS and STIS. In this article the design and the implementation of such pipeline are shown. The design is based on a usage of database queries, a rule system and an automatic code generation system written in PERL.

1. Introduction

The On The Fly calibration system (OTF), originally implemented by the CADC and the ST-ECF staff is becoming a very important aspect of the HST archive. The main goals behind the implementation of the OTF system were to take advantage of:

As shown by Lubow & Pollizzi (1999), more than 90% of the datasets located within the HST archive need to be recalibrated! Although sometimes the gain is minimal, the potential of extracting better science from the HST data is clear.

This is even more evident for the data acquired at the beginning of the life of an instrument. In fact, during the first months of NICMOS and STIS operations both the pipeline and the calibration files rapidly evolved as the instrument scientist's understanding of the instrumental responses and environmental conditions improved.

2. Recalibration Steps

In order to perform this operation for STIS and NICMOS, major changes to the existing OTF system were required, because of the complexity of these new instruments.

The STSDAS calibration system is ``header driven'': calibration switches and files, to be used by the calibration pipeline, are listed in the header of the raw file. This is a clever design since it allows to easily cope with changes in the system (e.g., the availability of new calibration steps, or of new reference files) by changing the relevant keywords in the raw file header before running the calibration software.

2.1 RAW Files

The first step in the OTF is to identify the correct list of raw files required for the calibration in order to retrieve them.

2.2 Finding the Best References Files

After retrieving the files, we query the Calibration Database System (CDBS, which is replicated to ECF and CADC from STScI) to get the best references files for a given dataset. This is a complex query since, because of timing differences, sometimes a calibration file is needed but not available yet at our respective centers. We have developed for this case a fall back mechanism allowing us to get the previous best calibration file. Here is an example of the query result:

    CDBS: shadfile, N/A
    CDBS: lamptab,  otab$i9u1550oo_lmp.fits
    CDBS: ccdtab,   otab$h1v1158do_ccd.fits
    CDBS: darkfile, oref$ibk1356ko_drk.fits
    CDBS: wbiafile, oref$iag16560o_bia.fits
    CDBS: sptrctab, N/A
    CDBS: inangtab, N/A
    CDBS: apdestab, otab$hbp11488o_apd.fits

The result of this query is stored within the proper ckwxxx pset file which will then be used to edit the raw file header. If a reference file is not available at all, the related calibration switch will be set to omit, independently of the rule system (see later).

The list of reference files is built at CADC and checked every day for consistency and completeness. The changes are then replicated to ST-ECF using database and files system synchronisation tools.

2.3 Updating Values of Keywords from the Database

Often, one would find wrong keyword values within the header. Once a HST file has been archived it is impossible to correct its header; instead, STScI updates a database which stores the keyword values. Therefore the next step is to extract the value of each critical keyword from that database.

The mapping of header keywords and the HSTDADS database is kept in a configuration file. SQL queries are built on the fly from it.

This model will be used for the new OTF calibration system that STScI is putting in place for WFPC2.

2.4 Translate the Rules in PERL Code

\scriptsize \tablecaption{Rule file example for NICM...
...HTOBJ \nl
NIC & DARKCORR & PERFORM & 4 & T = T \nl

The rule file (Table 1) is the key for the STIS and the NICMOS OTF calibration pipeline. It is part of the OPUS system used at STScI to calibrate the observations as soon as the are received from HST. The rule file assembles the knowledge from the instrument scientist about the calibration steps to be performed according to the instrument mode in use.

Although quite cryptic the file is being parsed dynamically when the OTF is started; a PERL script is then built. Here is an example of the perl code created:

         if ( ( $nicmos_cl{'TARGNAME'} eq "DARK" ) ) {
            $nicmos_cl{'DARKCORR'} = "OMIT";
            $reason = 19;
          } elsif ( ( $nicmos_cl{'FILTER'} =~ /BLANK$/ ) ) {
            $nicmos_cl{'DARKCORR'} = "OMIT";
            $reason = 20;
          } elsif ( ( $nicmos_cl{'OPMODE'} eq "BRIGHTOBJ" ) ) {
            $nicmos_cl{'DARKCORR'} = "OMIT";
            $reason = 21;
          } else { 
            $nicmos_cl{'DARKCORR'} = "PERFORM";
            $reason = 22;

2.5 Compute Calibration Switches

We have now acquired all the information needed to perform the switch evaluation. For the example case which we have used so far, the decision algorithm is summarised here:

    For each switch in the rules
       Does the switch need a reference file?
       > no:  switch is set by evaluating the rule %, e.g.:
       > yes: did CDBS return a file for this (switch,dataset) key ?
              > no:  missing reference file, switch is set to omit
              > yes: switch is set evaluating the rule, e.g.:
                     Evaluation for DARKCORR is PERFORM
                     Reason being 22

The result is put within a cl script which will be executed to alter the raw file header.

2.6 Replace Raw Header Keywords with the New Values and Calibration

Each pset is finally executed via the putcal IRAF task, which modifies the header keywords in the specified raw files. The datasets are then ready to be calibrated with the most recent version of the STSDAS calibration S/W.

3. Conclusions

The OTF system contributed (and still contributes) to the reliability of the calibration software: we found and reported problems to the STScI/STSDAS group, which quickly fixed them. As soon as a new version of the calibration software is released we install it in our pipeline. Our archive users, with their archival requests, also contribute to testing the software. The OTF pipeline is used at CADC to produce NICMOS and STIS preview images/spectra of all the available datasets (15 minutes after release date!), further contributing to testing the pipeline.

In other words, the OTF pipeline, being in a never-ending development phase and continuously receiving new reference files, is a dynamic system. An observation calibrated two months ago is different from the one calibrated today. Only at the end of the life of an instrument, when the ``final archive'' is produced, (i.e., no further development is foreseen), will this process stop and the best(?) calibration pipeline be available to the community.

Since the pipeline is entirely modular, it is quite easy for us to add more to it. One example is the preview pipeline. All the HST preview images and spectra which are used at STScI, ST-ECF and CADC are generated at CADC, as an extension of the on the fly calibration system. Their production, which takes place just after the data become public, contributes to:

At the time of writing the HST archive is composed of 338 CDs for the RAW data, and has 18 GBytes of calibration files.

The HST OTF service is available at CADC and ST-ECF. Please contact these respective sites through or


Lubow, S. & Pollizzi, J. 1999, this volume, 187


... Micol1
ESA/ESO Space Telescope European Coordinating Facility
... Durand2
National Research Council of Canada, Canadian Astronomy Data Centre
... Pirenne3
ESA/ESO Space Telescope European Coordinating Facility
... Gaudet4
National Research Council of Canada, Canadian Astronomy Data Centre
... Hodge5
Space Telescope Science Institute

© Copyright 1999 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: OAPI: the OPUS Application Programming Interface
Up: Data Pipelines
Previous: On-The-Fly Calibration at STScI
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint