Next: MAXI Software System: Photon Event Database
Up: High Performance Computing
Previous: Automated reduction and analysis of images from multiple data archives
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Butler, R. 2003, in ASP Conf. Ser., Vol. 314 Astronomical Data Analysis Software and Systems XIII, eds. F. Ochsenbein, M. Allen, & D. Egret (San Francisco: ASP), 448

An IRAF-based data reduction and analysis pipeline for the xFOSC family of instruments

Raymond F. Butler
Department of Physics, National University of Ireland, Galway, Ireland


I describe the design of a simple IRAF-based reduction and analysis pipeline, developed for the BFOSC instrument on the 1.52m Cassini Telescope at Loiano, run by the Osservatorio Astronomico di Bologna. The original motivation for this was pedagological: to enable our NUIG undergraduate students to quickly process their observations while still 'at the telescope', thus enriching their learning experience during their annual field-trip to Loiano. However, the current and future development of the pipeline is also being driven by our research programmes involving BFOSC data. On the basis of header keywords, raw frames are automatically grouped and processed (CCD reduction, coaddition, photometry, deconvolution, RGB-tricolour representation, and basic astrometry, with spectroscopy partially implemented as of now). Of particular interest is that the xFOSC family of instruments produced by the Astronomical Observatory of Copenhagen, which includes BFOSC, share identical design and operation. This should make it simple to adapt the pipeline to any of the ten FOSC instruments: ALFOSC on the Nordic Optical Telescope, DFOSC on the ESO/Danish 1.54m, and so on.

1. Introduction

In 2000, NUI Galway established Ireland's first undergraduate degree program where Astronomy is the major subject taken in all four years. Every Spring, we take our twenty 2nd-year students on their own observing run to Loiano, run by the Osservatorio Astronomico di Bologna, Italy. The Loiano 1.52m Cassini Telescope is equipped with BFOSC (Bologna Faint Object Spectrograph & Camera, one of the excellent xFOSC family of instruments produced by the Astronomical Observatory of Copenhagen. Each year, our students acquire masses of BFOSC data - but what has been missing from their learning experience was the ability to quickly process their observations while still 'at the telescope'. This prompted me to develop an automatic reduction and analysis pipeline, which is IRAF-based.

However, the current and future development of the pipeline is also being driven by our research programmes involving BFOSC data: variability studies in ultra-cool dwarf stars (Ó Tuairisg et al. 2004); evolution of binary stars in older open clusters; the supernova remnant-pulsar connection; abundance variations within globular clusters. So our frequent research observing runs on the 1.52m are also benefitting substantially.

2. Step 1: Classification & Auxiliary Header Information

On the basis of FITS header keywords, raw frames are automatically grouped and processed. The pipeline is header-driven.

Existing keywords OBJECT & IMAGETYP determine whether each frame is a bias, dark frame, flatfield, HeAr lamp spectrum, or scientific target. For the latter, all frames of a given target are grouped together.

Keyword fields are automatically re-parsed to handle changes in punctuation and lower/upper case, producing a suitable unique NEWOBJEC classification. Accidental zero-length exposures are reclassified as bias frames.

Using only the NEWOBJEC keyword list, another IRAF script is automatically generated for checking if the target names actually match the frame content (necessary since observers frequently forget to update the OBJECT keword when changing targets). Running this checking script is one of the very few points where human interaction is required; the user edits another script to patch any wrong or missing keyword values, with reference to the observing log. This step should be part any archive data-integrity checking/correction.

Location: I added an entry for Loiano to /iraf/noao/lib/obsdb.dat. This is used when a new OBSERVAT keyword is set to LOIANO. Filters: The FILTERS keyword is an integer code, with the actual filter name in the adjacent comment field. This is parsed to produce a new descriptive (string) FILTNAME keyword. Slits: A similar operation is used to parse the integer APERTNR to the descriptor SLITNAME. Grisms: The grism-wheel setting is already correctly numbered in GRISMNR. The fact that GRISMNR=1 for no grism is used later, to distinguish images from spectra. Readout/Gain: BFOSC supports slow (18 sec) and fast (2 sec) readout modes, but there is no keyword to denote which was used, since in practice only slow frames are saved. However to allow for future flexibility, a CCDSPEED keyword is added. GAIN & READNOIS keywords are also added, with values (according to readout mode) from the BFOSC manual. Timestamping & Airmass: The EPOCH value is automatically obtained from the DATE-OBS of each image, via interim conversion to its JD equivalent. The LJD of all frames are computed, and checked that they are monotonically increasing - ``turnovers'' can occur, since DATE-OBS is set by local time, not by the UT value. The UT-MIDDLE and AIRMASS are also added. However, the bias and dark frames have no UT keyword (I regard this as a design bug in the BFOSC controller) - they must be assigned suitable UT values before their crucial LJDs can be computed.

3. Step 2: Final Grouping

Nested loops step through the lists of NEWOBJEC, FILTNAME, SLITNAME, GRISMNR, CCDSPEED, & LJD from steps 1 & 2. Efficient hierarchical searches are used. This produces new sublists (groups) of the raw frames for every observational-group permutation encountered (analagous to the ST-ECF HST associations - Micol et al. 1997 ). The criteria are as follows:

For nightly biases: NEWOBJEC(=``bias'') + CCDSPEED + LJD. For frames to be bias-calibrated: NEWOBJEC(!= bias) + CCDSPEED + LJD.

For nightly flatfields in different filters ( CCDSPEED ceases to matter):

NEWOBJEC(=flat_imaging or flat_spectral) + FILTNAME + LJD. For imaging frames to be flatfield-calibrated: NEWOBJEC(!= bias or dark or flat) + FILTNAME + LJD. For spectral frames to be flatfield-calibrated: NEWOBJEC(!= bias or dark or flat) + SLITNAME + GRISMNR + LJD.

For calibrated imaging frames to be combined ( LJD ceases to matter): NEWOBJEC(!= bias or dark or flat) + FILTNAME. For calibrated spectral frames to be combined: NEWOBJEC(!= bias or dark or flat) + SLITNAME + GRISMNR.

4. Step 3: Basic Frame Calibration

For greater flexibility in handling the groupings from step 2, and the unique instrumental modes, IRAF/ccdproc is not used. Instead, the pipeline's own scripts perform the automatic processing of the calibration products first: bias coaddition and checking, imaging flatfields production, spectral flatfields production. (BFOSC dark current is so low that we have no dark frames to calibrate as of yet). All of the final grouping permutations from step 3 are stepped through. Data quality (DQ) masks are produced to flag any saturated pixels or bad columns, to faciliate their (optional) subsequent exclusion from frame coadditions etc. Individual flatfield frames are automatically scaled and weighted by their median/mode values before coaddition, and automatically normalised or response-corrected afterwards.

Further scripts apply these calibration products to the science frames, according to the correct night/readout-speed/filter etc. All target grouping/association permutations can be stepped through in one pass, or a single target can be selected. DQ masks are also constructed in the process.

5. Step 4: Target Registration & Coaddition; Tricolour Composite

We now have science-ready individual frames. Coadding associations (images of the same object in the same filter, with substantial pointing overlap) requires accurate frame registration. Unfortunately, BFOSC does not store reliable WCS (World Coordinate System) information in the frame headers; the commanded RA & DEC are given, but they do not account for manual repositioning of the FOV, or tracking drift between guidestar acquisitions. Therefore registration demands user identification of the coordinates of a common star. In future this will be upgraded to the more automatic cross-correlation technique, which we have used in other work (Ó Tuairisg et al. 2004).

The frames are then filtered by DQ mask, registered, scaled by EXPTIME, corrected for zero-offset by the mode of their overlap, weighted, and averaged with cosmic-ray rejection, taking account of GAIN & READNOIS. The resulting mosaics in each filter are then registered to each other. The GAIN, READNOIS & EXPTIME keywords are updated to NEWGAIN, NEWREADN & NEWEXPTI, to reflect the averaging of NCOMBINE frames.

If 3 or more filters were used, an RGB tricolour composite can be created using IRAF/rgbsun. Dynamic range limits are automatically passed to the task, based on the mean background levels and standard deviations of each mosaic.

6. Step 5: Photometry & Astrometry

The pipeline now enters the realm where it is normal to have extensive human interaction. But for ``quick'' results, even crowded-field photometry has been completely automated. The user may choose to override ``typical'' settings in the IRAF/DAOPHOT-2 parameter sets.

All of the aligned mosaics are coadded, to maximse FOV and depth for star-detection. Using this master starlist, each mosaic is passed through the IRAF/ DAOPHOT-2 tasks phot, pstselect (about 100 stars), psf (analytic model only), & allstar. A second iteration improves the PSF fidelity, by cleaning away neighbouring stars and supplementing the analytic model with a LUT of residuals. Further optional iterations can use higher-order models to deal with the moderate SV-PSF (spatially varying PSF) characteristic of BFOSC.

Colour-magnitude diagrams are constructed from the photometry in different filters. Astrometric calibration is done against stars in the 2MASS point source catalog, as we have done for HST/WFPC2 images (Butler et al. 2002). This module is currently incomplete.

7. Conclusions & Future Work

The benefits to us of this BFOSC pipeline are already clear: faster feedback on data quality while observing; ability to explore calibration options more deeply; faster analysis of data to publication quality back home; enhancement of our students' experience of observing.

Modules under construction include astrometry, analysis of the calibrated spectral frames (based on IRAF/doslit), and using calibration products from different nights. I also aim is to identify the customisations needed to adapt the pipeline to other xFOSC instruments ( ALFOSC on the La Palma 2.5m NOT, DFOSC on the 1.54m Danish (ESO), TFOSC on the new 1.5m TT1, etc.) - for use either at the telescope, or to perform automatic ``best'' calibration of the raw data in the archives of each xFOSC instrument. As an example of the former case, it could be used in conjunction with custom xFOSC-compatible light filters which we are developing for cluster abundances work.


Many thanks to the Loiano TAC chair (Dr. Valentina Zitelli) & the Osservatorio director (Prof. Flavio Fusi Pecci); my fellow Loiano observers (Dr. Aaron Golden, Dr. Andy Shearer, Prof. Mike Redfern); our AT201 students; the Loiano telescope operators; Dr. Seathrún Ó Tuairisg for discussions on aspects of pipeline design; and the authors of the softwars (IRAF, DAOPHOT-2, STSDAS, COLOR, etc.) on which the pipeline scripts are based.


Butler R. F., Golden A., Shearer A. 2002, A&A, 395, 845

Micol A., Bristow P., Pirenne B. 1997, STScI Proc. ``The 1997 HST Calibration Workshop with a new generation of instruments'', eds. S. Casertano, R. Jedrzejewski, C. D. Keyes, M. Stevens, p349

Ó Tuairisg S., et al. 2004, this volume, 444

© Copyright 2004 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: MAXI Software System: Photon Event Database
Up: High Performance Computing
Previous: Automated reduction and analysis of images from multiple data archives
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint