Next: The IDL Wavelet Workbench
Up: Applications
Previous: The Mosaic Data Capture Agent
Table of Contents -- Index -- PS reprint -- PDF reprint

Astronomical Data Analysis Software and Systems VII
ASP Conference Series, Vol. 145, 1998
Editors: R. Albrecht, R. N. Hook and H. A. Bushouse

Packaging Radio/Sub-millimeter Spectral Data in FITS

Zhong Wang
Smithsonian Astrophysical Observatory, Cambridge, MA 02138 U.S.A. Email:



We report on an experiment to incorporate the multiple extension FITS file format in building data processing software for spectroscopic observations in radio and millimeter-submillimeter wavelengths. By packaging spectral header information into table extensions and the actual spectral data into either table or image extensions, this approach provides a convenient and effective way to organize large amount of spectroscopic data, especially for pipeline-like processing tasks. Tests conducted based on a software package developed for the SWAS mission have demonstrated this to be a viable alternative to the conventional ways of organizing spectral data at these wavelengths.


1. Introduction: Characteristics of Submillimeter Spectral Data

Historically, spectral data were taken roughly one spectrum at a time, and most existing data formats reflect such a methodology. The traditional ways in which FITS can be used for spectral data are less than adequate in meeting the needs of a new generation of instruments that produce large amount of relatively homogeneous data, which nevertheless need to be reduced individually. A set of more efficient, better streamlined reduction procedures are necessary, which in turn require more careful considerations in data packaging. This is particularly important for projects that adopt non-interactive, pipeline-style data processing as their primary mode of operation.

The details of the SWAS mission can be found on the internet in SAO's SWAS homepage at SWAS (Submillimeter Wave Astronomy Satellite) mission is essentially a small orbiting radio telescope working at relatively high (about 500 GHz) frequencies (Wang 1995). It is designed to make simultaneous spectroscopic measurements at several sub-mm line wavelengths that are inaccessible from the ground. The spectral data have some special characteristics:

The data reduction plan for the mission calls for a highly efficient pipeline processing scheme in which a large amount of spectroscopic data can be sorted, selected, calibrated and co-added based on some user-changeable criteria. All processing operations have to be recorded and traceable, and are preferably done with minimal human intervention.

2. Software Approach: Packaging Data in FITS

We have selected IRAF as our primary programming platform for the SWAS pipeline software, and use FITS as the main format for our data product. However, to store individual scan data in the conventional one- (single spectrum) or two-dimensional (``multispec'' or echelle) format would be very inefficient and even confusing.

Our approach is to make use of the BINTABLE and IMAGE extensions of FITS and store spectral header and pixel data in these two types of extensions in a single FITS data file. The basic rules we adopted are:

The choice between using the table extension of FITS alone versus using table plus image extensions depends on the actual application. In principle, saving everything in a single table is structurally more appealing and efficient, but not many existing software tools can accept variable-length vector column FITS tables -- which means more development work for basic tools. On the other hand, making use of 2-D images can facilitate data inspection procedures with various tools that examine FITS images. Its shortcoming is that information about a single spectrum is saved in two or more extensions, requiring care in managing them.

3. Advantages and Efficiency Gains

Given the lack of a widely-accepted standard in formatting a largely homogeneous spectroscopic dataset, we find the packaging using FITS extensions to be a viable alternative to the conventional ways in which spectral data are saved:

Many of the existing software tools that deal with FITS table and image data are readily available for use with these new data files, sometimes with minor modifications. This can significantly save the programming effort and shorten the software development cycle.

4. Comments and Future Work

Despite the considerable merits of this approach, some problems remain to be addressed, and we are working on future enhancement of our software.

One of the main problems is the interface with existing software tools (such as visualization tools from certain existing tools packages). In practice, we were able to partly circumvent this shortcoming by writing customized interfaces, but the solutions are not always satisfactory. It appears that if a tabular style file format is deemed sufficiently important, a more fundamental (kernel-level) interface for spectral data in FITS should be developed.

The approach described in this paper is an explorative step in enhancing the use of FITS, especially in taking advantage of the multiple extension development and the use of vector columns in FITS tables. With the rapid advancement in database and data warehousing technology, one would ideally like to have a more rigorous database-like framework to deal with the management of the observational data. This, however, needs to be balanced with the requirement of cost-effective and quick prototyping of many projects' practical applications. The latter often means maximizing the use of existing software tools and adopting conventional approaches. The multiple extension FITS adopted by the community and being incorporated by NOAO and STScI's IRAF teams (e.g., Zarate & Greenfield 1996, Zarate 1998) and other software groups is an important and innovative development, which preserves the usefulness of many old tools while allowing new methods to be tested for astronomical data applications. We believe that more efforts are needed in exploring ways to take full advantage of this very useful data format.


Wang, Z. 1995, in Astronomical Data Analysis Software and Systems IV, ASP Conf. Ser., Vol. 77, eds. R. A. Shaw, H. E. Payne & J. J. E. Hayes (San Francisco, ASP), 402

Zarate, N. & Greenfield, P. 1996, in Astronomical Data Analysis Software and Systems V, ASP Conf. Ser., Vol. 101, eds. G. H. Jacoby and J. Barnes (San Francisco, ASP), 331

Zarate, N. 1998, this volume


We are grateful to Phil Hodge and Nelson Zarate for their help in the software development discussed in this paper. Dr. Matthew Ashby has participated in implementation and testing of the SWAS software.

© Copyright 1998 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA

Next: The IDL Wavelet Workbench
Up: Applications
Previous: The Mosaic Data Capture Agent
Table of Contents -- Index -- PS reprint -- PDF reprint