Currently, Hubble Space Telescope (HST) data are calibrated as they are received at the Space Telescope Science Institute (STScI). Raw and calibrated data are stored in the HST archive (DADS). Frequently, users recalibrate data at their home sites to take advantage of better calibration files or software. To recalibrate, users can obtain the raw data and the recommended reference files from the Starview system. The user must then update the raw data headers with the names of the recommended reference files. The appropriate STSDAS calibration software is then run.
An on-the-fly calibration (OTFC) system at STScI will allow users to obtain data from the DADS archive which is calibrated with the most up-to-date calibration parameters, files, and software. In most cases, this capability requires that data be automatically calibrated at STScI when the data is requested by a user. An OTFC system has been successfully developed for HST data by the CADC/ST-ECF (Crabtree et al. 1995). The requirements for an OTFC system at STScI are somewhat different from the CADC/ST-ECF. More details about the OTFC system can be found on the web at
There are two main motivations for the OTFC system: saving data storage and providing users with improved calibration. These motivations were discussed in the HARP report (Hanisch et al. 1996). HST data are stored in an optical disk jukebox as part of the DADS system. The jukebox system has an online capacity of about 3 TB. The system is essentially full now. We have investigated the possibility of using DVD storage technology, but found that this is not yet ready for our purposes. Furthermore, the Advanced Camera for Surveys is scheduled for installation in HST within the next two years. It will substantially increase the rate at which data are received at STScI and would worsen the storage problem. With OTFC, only the raw data would need to be stored. Since raw data compresses better than calibrated data, there are substantial savings (of about a factor of ten in some cases) by storing only raw data in the archive in compressed form.
The other advantage of OTFC is in providing improved calibration to users. Nearly all (more than 90% of) datasets can benefit from recalibration (although the level of improvement is not always significant). Cameras such as WFPC2 produce dark files that typically should be retroactively used to calibrate observations taken in the prior week. Unfortunately, this means that the original calibration carried out in the STScI pipeline is not the optimal. Furthermore, this less than optimal calibration is stored in the DADS archive. An OTFC system would help solve this problem by providing users with the calibration having the appropriate dark, as soon as it is available.
Currently, instruments that undergo evolution of calibration files or calibration software require users to carry out their own recalibrations at their home sites. It may well be easier for both the users and the STScI support groups to have STScI carry out the recalibration.
Apart from reference files and software, improvements to calibration also can come about through changes to calibration switches and other keywords. Another goal of OTFC is to make such keyword changes possible without rearchiving the data in the optical disk archive.
We expect that the OTFC system will calibrate a large number of datasets each day, many times more than are calibrated by the current calibration pipeline for new data. For WFPC2, about 300 datasets are currently requested from DADS each day. Consequently, the system needs to be highly automated, while at the same time be robust. The system should provide a reasonable level of performance. It should not impose a substantial delay beyond the current DADS retrieval times.
Some users have established their own calibration pipelines, while others will sometimes customize the calibrations. To support such users, tools should be provided to determine the latest information about calibration parameters.
Another requirement is that the OTFC system should be compatible with other STScI systems, such as Starview and DADS. In fact, these existing systems will play a role in the OTFC system.
To describe changes to calibration keywords relative to the values in the archived raw data, a set of database tables is used. These tables contain entries for each dataset, keyword, and header (in the case of multiple headers). Each entry contains the new and old values of the keyword, the operation (insert, update, or delete) that should be applied to the header, as well as textual comments that describe the nature of the change. In essence, these tables describe how to edit the header. By using a database, we are able to provide a history of changes to keywords that is readily accessible to support staff. Furthermore, these entries are used to change the values for these keywords in the DADS catalog.
The change system also includes software to determine the calibration switch rules (Bestswitch) and the best reference files for each dataset (Bestref). Changes to parameters in the database for each dataset need to be carefully coordinated. For example, keyword changes need to be applied ahead of the determination of reference files. Furthermore, these operations need to be carried out as an atomic transaction so that the system is not left in an inconsistent state if errors occur.
The keyword changes are applied to the data header by an enhanced version of the getref/upref software. These tools will be provided for remote users to carry out their own calibrations. Getref can be accessed through a web interface, Upref is part of STSDAS.
Some datasets should not be calibrated by the OTFC system. For example, the data may be corrupt or it may take too long to calibrate. The OTFC change system contains a database table that describes such situations. The OTFC will not calibrate such data, but will return the raw data to the user.
Currently, for WFPC2, there are 368 keyword changes in the database and 11 datasets have been identified in the database for noncalibration.
We have developed a prototype system for WFPC2. We will release that system to external users in the near future. We will then extend OTFC to handle STIS, NICMOS, and ACS.
We thank Dan Durand for informative discussions.
Crabtree, D. R., Durand, D., Gaudet, S., & Hill, N. 1995, in ASP Conf. Ser., Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes (San Francisco: ASP), 505
Hanisch, R.J., et al. 1996, HARP: The Hubble Archive Re-Engineering Project Summary Report, STScI report
Rose, J. 1996, in ASP Conf. Ser., Vol. 125, Astronomical Data Analysis Software and Systems VI, ed. G. Hunt & H. E. Payne (San Francisco: ASP), 38