We describe the techniques employed to meet the challenge of processing and disseminating the substantial data flows associated with this survey. Key areas addressed are: the pipelines developed to pre-process and process the data (e.g., flux and astrometric calibration) and the automation of routines to fulfill user requests for these data products, implemented through WWW interfaces.
The 2.5m Isaac Newton Telescope (INT) is the oldest of the three telescopes that comprise the Isaac Newton Group of telescopes on La Palma. In an era when several very large telescope projects will be coming on line in the near future (e.g., Gemini North and South, ESO VLT) a shift in emphasis of the role of the INT towards wide field optical image surveys is clearly highly desirable.
In the Spring of 1997 a new Wide Field Camera was commissioned on the INT. This consists of a mosaic of four EEV CCDs, each having a format of 2148 x 4128 pixels 13.5 microns. A single exposure covers a total of about 0.4 sq. degrees. In order to promote the use of the WFC for large scale survey work, the ING Joint Steering Committee decided to make substantial amounts of observing time (three or more weeks per semester) available for high quality survey programmes. This paper is a description of `The INT Wide Angle Survey' one of the survey programmes chosen to partake of this time.
The INT Wide Angle Survey is the first major fully digital, multi-epoch and multi-waveband wide angle survey over a significant area. The survey parameters cover a large range of areas of topical and future interest. Some of the scientific goals of the survey are:
Although the current survey is targeted at specific scientific goals, a judicious choice of passbands (the Sloan Digital Sky Survey passbands have been chosen for longevity), exposure times and regions will make these data useful for a much wider audience. The field list of field centres and the science programs these data will serve are listed in the Wide Field Survey WWW site.
An important element of all of the INT Wide Field Camera surveys is that the raw data are made available on line to the UK and Dutch communities almost as soon as they are observed. At a later stage semi-processed (debiassed and flat fielded) data are available. Finally, fully reduced (i.e., astrometrically and photometrically calibrated) frames will be produced.
Data reduction will be done with a specially written data reduction pipeline, wfcred. The package is written to take advantage of existing software and as such the vast part of the data manipulation is handled by IRAF. The individual data reduction tasks take the form of a number of Perl scripts. (Perl/Tk is used for the user interface.) These scripts are used to gather information about the nominated files, sorting them into subsets by filter, Julian Date, observation type (e.g., Flat field, Bias or Target), etc. Once the files are sorted into their requisite subsets, then they are sent to IRAF tasks such as ccdproc for processing.
The formation of good mean bias frames and in particular mean flat field frames can be tricky. Much care needs to be taken to ensure, for example, that frames with large overexposed regions are not included and thus this should be an interactive process. For these reasons the routines fftool and biastool have been written. As their names suggest these are interactive tools used for the creating and storing mean bias and flat field frames. Included with these are facilities to help the user decide whether or not to include an exposure (e.g., the ability to examine headers, display frames and do simple statistics). Once the user has narrowed down the list of files to be used in the formation of the mean frames, the processing is then a straight-forward application of standard CCD processing routines in IRAF. One of the most important aspects of wfcred is the ability to have libraries of mean bias and flat field frames. This is crucial as even in the best of observing conditions it is nearly impossible to get a full set of flat field exposures each and every night. Once a mean frame has been created the user has the ability to either store it in a library or discard it.
Central to the way in which wfcred works is its ability to group files into subsets according to various criteria (e.g., filter name or Julian date). This is done in a way which is invisible to the user who is presented with the data grouped in the most sensible way depending upon the stage of the reduction procedure. For example in fftool it is most likely that the user will want to see the files grouped by filter first. Having decided which filter's flats will be processed the user is then presented with the list of files in that filter subset, but regrouped by Julian date. The subsets presented by this final grouping are obviously the four frames (one for each chip) for a given exposure. When it comes to doing the CCD processing and combining it obviously makes far more sense that all data from a single chip are handled simultaneously. Thus once the user has made the final choice of frames the remaining files are regrouped once more by chip name before being sent to the various IRAF routines.
During early stages of development of wfcred it was discovered that the analogue-digital converter for one pair of chips was not behaving linearly. To take account of this effect a `D to I' conversion routine has been written and incorporated into the pipeline. For later dates when the problem has been resolved the converter can be disabled, but will remain as part of the pipeline for the reduction of older data.
In addition to the removal of instrumental signature, target frames also have an astrometrically accurate world coordinate system defined for them. The relevant section of the Guide Star Catalogue is extracted first and the coordinates are converted to (X,Y) with a rough transformation based on the coordinate centre and the known configuration of the chips. The GSC stars are matched up against the (x,y) positions of the brightest detected images found on the chip. Finally a bi-linear polynomial fit is done to (x,y) and using the IRAF routine ccmap with the ZPX projection and a known cubic distortion term. There are typically between 15 and 30 stars on a frame and the RMS residuals are generally 0.5 arcseconds or less.
Below is a list of future goals for wfcred.
We gratefully acknowledge the help of Drs. M.J. Irwin and R.G. McMahon for their help in preparing this paper.