Next: CalFUSE v2.2: An Improved Data Calibration Pipeline for the Far Ultraviolet Spectroscopic Explorer (FUSE)
Up: Data Management and Pipelines
Previous: Status of the BIMA Image Pipeline
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Cavanagh, B., Hirst, P., Jenness, T., Economou, F., Currie, M. J., Todd, S., & Ryder, S. D. 2003, in ASP Conf. Ser., Vol. 295 Astronomical Data Analysis Software and Systems XII, eds. H. E. Payne, R. I. Jedrzejewski, & R. N. Hook (San Francisco: ASP), 237

ORAC-DR: One Pipeline for Multiple Telescopes

Brad Cavanagh, Paul Hirst, Tim Jenness, Frossie Economou
Joint Astronomy Centre, 660 N. A'oh{\={o\/}}k{\={u\/}} Place, University Park, Hilo, HI 96720

Malcolm J. Currie
Starlink Project, Rutherford Appleton Laboratory, Chilton, Didcot, Oxon, OX11 0QX, United Kingdom

Stephen Todd
UK Astronomy Technology Centre, Blackford Hill, Edinburgh, EH9 3HJ, United Kingdom

Stuart D. Ryder
Anglo-Australian Observatory, P.O. Box 296, Epping, NSW 1710, Australia


ORAC-DR, a flexible and extensible data reduction pipeline, has been successfully used for real-time data reduction from UFTI and IRCAM (infrared cameras), CGS4 (near-infrared spectrometer), Michelle (mid-infrared imager and echelle spectrometer), at UKIRT; and SCUBA (sub-millimeter bolometer array) at JCMT. We have now added the infrared imaging spectrometers IRIS2 at the Anglo-Australian Telescope and UIST at UKIRT to the list of officially supported instruments. We also present initial integral field unit support for UIST, along with unofficial support for the imager and multi-object spectrograph GMOS at Gemini. This paper briefly describes features of the pipeline along with details of adopting ORAC-DR for other instruments on telescopes around the world.

1. Background on ORAC-DR

ORAC-DR is the data reduction pipeline for the Observatory Reduction and Acquisition Control project (ORAC1, Wright et al 2001). The ORAC-DR pipeline design was presented previously by Economou et al (1998) and Jenness & Economou (1999) and makes extensive reuse of existing data reduction packages (Economou et al 1999). Allan et al (2002) described groundwork for turning ORAC-DR into a truly generic data reduction pipeline. For the past three years it has been used at the United Kingdom Infrared Telescope (UKIRT) and for SCUBA observations on the James Clerk Maxwell Telescope (JCMT).

2. Reducing the Workload

When developing the ORAC-DR pipeline, we recognized that many of the tasks performed by instrument-specific pipelines are shared. File handling, calibration systems, and header translations are tasks that are often duplicated. Also, the actual data reduction tasks are similar from one instrument to the next. Repeating the implementation of these tasks by developing a specific pipeline for each new instrument was considered to be a waste of time and resources, so we sought a generic solution.

To accomplish this, we created a generic, and yet advanced, object-oriented ORAC-DR infrastructure which handles file conversions, header translations, calibration information, and other generic tasks. It also runs on a `primitive'-based system, where one primitive (Perl script) does the work of one astronomically meaningful step. These primitives also do a bare minimum of computing; processing of bulk data is done by CCDPACK (Draper, Taylor & Allan 2000), KAPPA (Currie & Berry 2000), and Figaro (Shortridge et al 1999; Shortridge 1993), all supported by Starlink.

3. Necessary Modifications

As ORAC-DR is a data-driven data reduction pipeline, it obtains most of the information required to reduce data from the data itself. For astronomical data, this means that FITS headers need to be complete and correct. Simple instrument-specific Perl modules translate FITS headers into internal headers that are used by any code relying on headers. These translations are not limited to changes of name, and may involve combinations of headers. Through inheritance one instrument can acquire header translations from another, further reducing the number of header-translation modifications necessary to use ORAC-DR with a new instrument.

This same philosophy applies to file formats. As the actual data reduction calculations are performed by Starlink tasks whose internal data format is the Extensible N-Dimensional Data Format (NDF, Warren-Smith 2000), ORAC-DR minimizes the number of conversions from a different format (FITS, for example) by doing a conversion to NDF when it first sees a data file. Which conversion ORAC-DR does is instrument-specific, and this is programmed in such a way that adding a new instrument and new data conversions requires minimal effort.

Inheritance is also used for recipes and primitives. A large collection of generic recipes and primitives exists. Should an instrument require a new or modified recipe or primitive then this new code is placed in an instrument-specific subdirectory. When ORAC-DR searches for a primitive, it looks in the instrument-specific directory, then in the instrument-agnostic directory.

Occasionally new observing modes are needed, or instruments with different capabilities are developed. An example of this is the imaging spectrometer UIST, which also has an integral field unit. ORAC-DR had no previous IFU support, so it was necessary to develop a new set of primitives and recipes to reduce IFU data.

Modifying ORAC-DR to be used on an instrument similar to currently supported instruments takes approximately one to two weeks of programming time. If new observing modes are needed, then this time is increased to one to four months, depending on the number of recipes needed and the instrument's complexity.

4. Implementation of Modifications

4.1 IRIS2

IRIS2 is an infrared imager and spectrograph currently being commissioned for the 3.9-meter Anglo-Australian Telescope. In addition to wide-field imaging and long-slit spectroscopy, it also offers multi-object spectroscopy. It writes data in FITS format, not the NDF that UKIRT instruments use.

In July 2002 ORAC-DR was commissioned for imaging with IRIS2, needing only two weeks of programming effort to become fully usable. As ORAC-DR relies on FITS headers to process data, a side-effect of porting it to other instruments is an ability to find omissions and deficiencies in header information. During the commissioning of ORAC-DR it was noticed that an incorrect value for the array read noise of IRIS2 was being reported. This was traced to a bug in the data acquisition software and was immediately fixed. Had ORAC-DR not been running at the AAT at the time, this bug would have been found much later and after much more effort.

During the commissioning of ORAC-DR, IRIS2 made a deep image of a section of sky that would later be observed with the Very Large Telescope. As ORAC-DR was able to calculate a reduced mosaic of the region in real-time, it was possible to send this mosaic to the principle investigator. He then used this mosaic to select out high- z galaxy candidates from low-mass stars for follow-up VLT multi-object spectroscopy a few days later. ORAC-DR was able to save the principle investigator the time and effort necessary to reduce their data and did so needing only minimal changes.

4.2 UIST

UIST is an imaging spectrometer very similar to IRIS2 in capabilities. Adapting ORAC-DR to work with imaging and spectroscopy modes with UIST was thus trivial, requiring approximately two weeks of effort to run correctly.

In addition to imaging and spectroscopy modes, UIST has an integral field unit which allows it to simultaneously obtain spectra at a number of points on a field. This mode is one that had not been previously supported by ORAC-DR, so programming effort required to get ORAC-DR properly reducing IFU data was approximately three months by a programmer with no previous experience with ORAC-DR.

4.3 GMOS

GMOS is a multi-object spectrograph with imaging, long-slit spectroscopy, and IFU modes. Basic System Verification data were released for imaging and multi-object spectrograph modes, and imaging data was used for ORAC-DR. The FITS headers for this SV data are incomplete and raw calibration data are not available, so ORAC-DR support for GMOS is minimal. However, ORAC-DR can successfully reduce publicly available GMOS data. The programming time needed to attain this level of support was eight hours.


We would like to thank STARLINK for distributing ORAC-DR and supporting many packages ORAC-DR uses. We thank Gemini for releasing GMOS System Verification data for its use with ORAC-DR.


Allan, A., Jenness, T., Economou, F., Currie, M. J., & Bly, M. 2002, in ASP Conf. Ser., Vol. 281, Astronomical Data Analysis Software and Systems XI, ed. David A. Bohlender, Daniel Durand and T. H. Handley (San Francisco: ASP), 311

Currie, M. J., Berry, D. S. 2000, Starlink User Note 95, Starlink Project, CCLRC

Draper, P. W., Taylor, M., & Allan, A. 2000, Starlink User Note 139, Starlink Project, CCLRC

Economou, F., Bridger, A., Wright, G. S., Rees, N. P., & Jenness, T. 1998, in ASP Conf. Ser., Vol. 145, Astronomical Data Analysis Software and Systems VII, ed. R. Albrecht, R. N. Hook, & H. A. Bushouse (San Francisco: ASP), 196

Jenness, T., & Economou, F. 1999, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, ed.  David M. Mehringer, Raymond L. Plante, & Douglas A. Roberts (San Francisco: ASP), 171

Shortridge, K. 1993, in ASP Conf. Ser., Vol. 52, Astronomical Data Analysis Software and Systems II, ed. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes (San Francisco: ASP), 219

Shortridge, K., et al. 1999, Starlink User Note 86, Starlink Project, CCLRC

Warren-Smith, R. F. 2000, Starlink User Note 33, Starlink Project, CCLRC

Wright, G. S. et al. 2001, in ASP Conf. Ser., Vol. 238, Astronomical Data Analysis Software and Systems X, ed. F. R. Harnden, Jr., Francis A. Primini, & Harry E. Payne (San Francisco: ASP), 137


... (ORAC1
ORAC is a joint project of the Astronomy Technology Centre, Edinburgh and the Joint Astronomy Centre, Hawaii.

© Copyright 2003 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: CalFUSE v2.2: An Improved Data Calibration Pipeline for the Far Ultraviolet Spectroscopic Explorer (FUSE)
Up: Data Management and Pipelines
Previous: Status of the BIMA Image Pipeline
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint