Next: On the Need for Input Data Control in Pipeline Reductions
Up: Dataflow and Scheduling
Previous: Enhanced HST Pointing and Calibration Accuracy: Generating HST Jitter Files at ST-ECF
Table of Contents -- Index -- PS reprint -- PDF reprint

Astronomical Data Analysis Software and Systems VII
ASP Conference Series, Vol. 145, 1998
Editors: R. Albrecht, R. N. Hook and H. A. Bushouse

HST Paper Products: A New Way to Look at HST Data

W. Hack
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD, 21218, Email:

J.-C. Hsu
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD, 21218, Email:



Two new instruments, STIS and NICMOS, were added to HST during the SM97 servicing mission, each with new types of data that need to be processed. As a result, a completely new system for generating paper products was created which provides a detailed summary of the observations and of the calibrations applied to the observations in the HST pipeline. This new version of the paper products generating software produces a totally redesigned summary for the older instruments (WFPC2, FOS, GHRS, and FOC) as well as creating paper products for the newly installed STIS and NICMOS. In this paper, we discuss the design and algorithms used to handle the extremely large amounts of data produced by the instruments on board HST, as well as the new task's limitations. This new version of the HST paper product software is now available in STSDAS Release V2.0.


1. Introduction

The Hubble Space Telescope (HST) not only generates a great deal of data in a single day, but a wide variety of data as well - from 1-D and echelle spectra to images and time-tag data. All of this data is processed on the ground to apply basic instrumental calibrations and prepare the data to be ingested into the HST archive and then analyzed by the observer. One final step in this processing is to create a summary for initially judging the quality of the observations and for determining the status of the calibrations applied to the data. This summary, however, must be efficient in its reporting and yet comprehensive enough in its results to be useful. Generating such a report for each instrument's data (WFPC2, FOC, FOS, GHRS, STIS and NICMOS) while retaining a similar enough format for easy comprehension was the primary goal for the software. In addition, the software must run quickly enough to prevent backlogs when ingesting into the archive. These two primary problems guided our effort to create a single software system to provide summaries known as `paper products' for all HST data. This paper will summarize the major features of this software system.

2. Overview of Data Processing

The paper products software performs the following steps in creating the summaries for each set of observations:

This processing will produce a report tailored to each instruments data and contain the following estimated number of pages:

Cover Page and Help Page 1 Page each
Help Page 1 Page
Exposure Summary 1+ Pages
Data Display 1 Page/observation (FOC)
  1- 2 Pages/external observation (WFPC2)
  1-3 Pages/observation (FOS)
  1+ Page/observation (STIS)
  32 dithered obs/page + 1 page/image (NICMOS)
Observation Summary 1-2 pages

HST archive operations currently used Helvetica narrow fonts for all paper products. Use of different fonts may cause overlapping of text in the report.

3. Operation of the Software

The paper products software was designed to be run within the IRAF CL environment. The package consists of one primary task, pp_dads, and many hidden tasks to control each instrument's output. The pp_dads task only uses a small number of parameters; namely, input, device, fits_conv.

The parameter input controls which files are read in for processing, including using wildcards, list files or simply one image name. The parameter device can be set to a PostScript device to create PostScript files or a printer queue to send the output directly to a printer. This relies on using IRAF's printer queues, graphics kernels, and STSDAS's PostScript kernel.

4. Use of FITSIO

All HST data is routinely delivered to observers on tape in FITS format, and archived in the DADS system in FITS format as well. The paper products software, therefore, relies on the latest FITS kernel in IRAF to work with STIS and NICMOS data whilst also working directly with FOC, WFPC2, FOS, and GHRS data that are stored in the GEIS format.

5. SPP, CL and IGI

Most data processing is handled by routines written specifically for each instrument's data. These functions are written in SPP for NICMOS, FOC, WFPC2, and FOS, while CL scripts are used to process STIS and GHRS observations. Modular design can be readily modified for each instrument or even have new code added to support future new instruments. The use of SPP provides efficient access to image headers, the ability to perform statistical calculations while the image remains open, and great control of the output as IGI commands. The CL scripts used for STIS and GHRS are readily edited and can be used without compiling, however, they lack the efficient file I/O that SPP offers.

The results of the calculations and instructions for displaying the data are sent to an output file to be interpreted by IGI, the Interactive Graphics Interface. This provides several advantages for creating the paper products:

6. Computational Efficiency

The paper products software must run fast enough to keep up with the data stream coming from HST in order to avoid backlog. This led to the use of SPP code to generate the paper products for most instruments. In general, the computation of the paper products takes from 30-60 seconds per observation on a Sparc 5 workstation with only one user, and about 2-3 minutes per observation on a DEC Alpha workstation with several users. This is quick enough to avoid most backlog situations. The real bottleneck is the printing of the paper products. A local Printserver 17 printer can print about 15-20 Mb of PostScript output in an hour, however, the paper products code can generate 24Mb of PostScript in about 6 minutes on a Sparc 5, or about 20 minutes on an Alpha workstation. Therefore, a great deal of emphasis has been placed on setting up ways to print these jobs in parallel on multiple printers to keep up with the stream of data coming from HST. A drawback to this is the need for large temporary disk space for block averaged WFPC2/STIS/NICMOS images, PostScript files, and temporary IGI files and an effort is being made to reduce this need.

7. Summary

The new paper products package within STSDAS can process observations from a mix of instruments in one pass resulting in clear and concise documentation. It relies on many of IRAF's built-in functionality as well as many separate IRAF/STSDAS tasks to produce the paper products. The following packages are necessary for the operation of the paper products software: IRAF Version 2.10.4p2, or newer (preferably Version 2.11) and STSDAS/TABLES Version 1.3.5, Version 2.0, or newer. The simple, one task interface makes it easy to run, both interactively by a user after extracting data from DADS and in batch mode from a script, as done by the HST archive. This experience with processing daily observations has proved that this task can function efficiently and effectively with minimal intervention.

© Copyright 1998 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA

Next: On the Need for Input Data Control in Pipeline Reductions
Up: Dataflow and Scheduling
Previous: Enhanced HST Pointing and Calibration Accuracy: Generating HST Jitter Files at ST-ECF
Table of Contents -- Index -- PS reprint -- PDF reprint