Next: A Pipeline for an Automatic Autonomous Observatory: Application to TAROT
Up: Data Pipelines and Quality Control
Previous: Doing Theoretical Astrophysics With HST's Data Processing Pipeline Software: Can Your Pipeline Software Do That?
Table of Contents - Subject Index - Author Index - PS reprint -

Swade, D. A. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, eds. N. Manset, C. Veillet, D. Crabtree (San Francisco: ASP), 441

Generating Calibration Reference Files with an OPUS Pipeline

D. A. Swade
Computer Sciences Corporation, Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218


Calibration reference file processing can be automated by incorporation into an OPUS pipeline. This paper outlines the processing steps necessary for production of calibration files that are applicable for long periods of time, as well as those that are applicable only to contemporaneous science data.

1. Introduction

OPUS is the data processing software system developed at STScI (Rose et al. 1995, Rose 1998, Swade & Rose 1998). In addition to performing data processing for HST, OPUS is currently being used by the FUSE, AXAF, SIRTF, and INTEGRAL missions.

The current data processing ground system configuration for HST utilizes the capabilities of OPUS with regards to constructing a complex data processing system using a number of OPUS pipelines. OPUS pipelines can be run stand-alone, in series, or in parallel. A processing step in one pipeline can bridge to another pipeline. This paper presents concepts on how OPUS can be used to incorporate calibration reference file processing into the pipeline production of science data. Two types of calibration reference data will be addressed.

  1. Calibration reference files, such as optical and UV darks and flats, that are stable for time scales longer than a few days, can be created and associated with other relevant observations through a database mechanism. For HST, the Calibration DataBase System (CDBS) reference files exemplify this case.
  2. For IR observations, the calibration data may only be applicable to a particular observation or set of exposures taken within a short time period. In this case, contemporaneous calibration reference files need to generated and associated with the corresponding science data.
This second mechanism for generating contemporaneous calibration reference files will become more important with future IR observations using the WFC3/IR channel on HST and the NGST science instruments.

2. CDBS Reference Files

Generation of routine calibration reference files can be incorporated into an OPUS pipeline. This provides a more timely mechanism for getting better calibrated data to users, as well as saving the manual effort involved in generating standard calibration reference files. The OPUS pipeline would perform the following steps to create and make available a CDBS reference file:

In addition, the newly created calibration reference file is sent to the relevant science instrument team for evaluation of its quality and to be sure no problems exist with the file. The calibration reference file is installed in the CDBS system, after which it will be available for BESTREF.

3. WFPC2 Darks - a Specific Example

An example of a particularly labor intensive task is the generation of dark calibration images for the HST WFPC2. This process could not only be automated with an OPUS pipeline, but quality of calibration performed in the pipeline could also be improved.

Currently, the WFPC2 instrument executes five long dark exposures each week. These five exposures are combined to generate a weekly dark CDBS reference file. This file is inspected and placed in the science data processing pipeline. About once a year, each weekly dark taken over that period is combined to form a high quality super-dark. The super-dark is made available to data users through the archive, but is not used in the science data processing pipeline.

An OPUS pipeline could perform the following processing for WFPC2 darks:

An identical process could be used for WFPC2 bias exposures.

4. Contemporaneous Calibration Files

Contemporaneous reference files can be handled by OPUS in much the same manner as described above. In this case, there is no point in recording the existence of the file in a database, since it is only applicable to a small number of exposures. A better mechanism is to associate the calibration file directly to the science data through a keyword in the science data header. To generate and immediately use a contemporaneous reference file, an OPUS pipeline would perform the following steps:

5. NICMOS Post-SAA Darks - a Potential Example

The second generation of NICMOS data will utilize a contemporaneous reference file in attempt to calibrate the effect of SAA persistence (Najita, Dickinson, & Holfeltz 1998). Internal dark exposures will be taken immediately after a SAA passage when science exposures are scheduled to follow within 50 minutes. These post-SAA darks will be used to generate a cosmic ray map that can be applied to an image in order to reduce noise from cosmic ray persistence induced by SAA charged particle hits on the detector.

However, generation of the cosmic ray map cannot be automated at this time because it is not possible to remove the DC pedestal prevalent in NICMOS data within a pipeline environment. When, or if, automation of DC pedestal removal is possible, an OPUS data processing pipeline would be capable of utilizing the above scheme to remove SAA persistence.

6. Conclusion

Both types of calibration reference file pipelines described above can be simultaneously integrated into the current OPUS science data processing system for HST. More timely, or contemporaneous, generation of calibration reference files would improve the quality of data produced by the HST science data processing pipeline.

The OPUS baseline system is currently distributed on CD-ROM to help other institutions with their data processing pipeline management (Swade & Rose 1999). The OPUS CD-ROM comes complete with the Process Manager, the Observation Manager, a set of sample applications, and all the resource files required to get a sample pipeline running. In addition, the OPUS CD-ROM and the sample pipeline are fully documented in the OPUS Frequently Asked Questions (FAQ). The FAQ is available on the OPUS CD-ROM or at


Thanks to WFPC2 instrument scientist Sylvia
Baggett for useful discussions about WFPC2 calibration processing.


Najita, J., Dickinson, M., & Holfeltz, S. 1998, NICMOS ISR-98-001 (Baltimore: STScI)

Rose, J. Akella, R. Binegar, S. Choo, T. H. Heller-Boyer, C. Hester, T. Hyde, P. Perrine R. Rose, M. A. & Steuerman K. 1995, in ASP Conf. Ser., Vol. 77, Astronomical Data Analysis Software and Systems IV, ed. R. A. Shaw, H. E. Payne, & J. J. E. Hayes (San Francisco: ASP), 429

Rose, J. 1998, in ASP Conf. Ser., Vol. 145, Astronomical Data Analysis Software and Systems VII, ed. R. Albrecht, R. N. Hook, & H. A. Bushouse (San Francisco: ASP), 344

Swade, D. A., & Rose, J. F. 1998 in proceedings of AIAA/USU Conference on Small Satellites (Logan: Utah State University)

Swade, D. A., & Rose, J. F. 1999, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, ed. D. M. Mehringer, R. L. Plante, & D. A. Roberts (San Francisco: ASP), 111

© Copyright 2000 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: A Pipeline for an Automatic Autonomous Observatory: Application to TAROT
Up: Data Pipelines and Quality Control
Previous: Doing Theoretical Astrophysics With HST's Data Processing Pipeline Software: Can Your Pipeline Software Do That?
Table of Contents - Subject Index - Author Index - PS reprint -