| prev toc next |
Traditional ground-based observatories have moved into a new era where the dominant consumers for their data products are astronomers who no longer come out to use the facility themselves. Using the JCMT and UKIRT as an example, we discuss the wide variety of software systems that are necessary for maintaining a high level of scientific involvement for the absentee observer. These include cradle-to-grave flexible scheduling support, advanced data reduction pipelines, VO integration and intelligent agent networks. Is the observational astronomer becoming an endangered species, and does this really matter?
The XMM-Newton Scientific Analysis System (SAS) is the package used for data calibration and reduction leading to more than 300 refereed scientific papers published in the last 2.5 years. Its maintenance, further development and distribution is under the responsibility of the XMM-Newton Science Operations Centre in coordination with the Science Survey Centre (University of Leicester), representing a collaborative effort of more than 30 scientific institutes.
Developed in C++, Fortran 90/95 and Perl, the SAS makes large use of freeware packages such as ds9 from the SAO-R&D Software Suite and Grace, as well as of several public libraries including pgplot, cfitsio, Qt and FFTW.
The combination of supporting several versions of SAS for multiple platforms (including SunOS, DEC, many Linux flavours and MacOS) in a wide distributed development process which makes use of a suite of external packages and libraries presents substantial issues for the integrity of the SAS maintenance and development.
A further aspect is derived from the necessity of maintaining the flexibility of a software package evolving together with progress made in instrument calibration and analysis refinement, whilst at the same time being the source of all official products of the XMM-Newton mission. To cope with this requirement, a sophisticated system for continuous integration and testing on several platforms of different branches have been put in place on top of a refined development model designed for this special S/W development case.
The SAS is considered now a mature system. We intend to review critically all the different aspects of its development, maintenance and distribution, extracting lessons learned for present and future projects of this magnitude.
Since the beginning of the VLT operations in 1998, substantial effort has been put in the development of automatic data reduction tools for the VLT instruments.
A VLT instrument pipeline is a complex system that has to be able to identify and classify each produced FITS file, optionally retrieve calibration files from a database, use an image processing software to reduce the data, compute and log quality control parameters, produce FITS images or tables with the correct headers, optionally display them in the control room and send them to the archive.
Each instrument has its own dedicated pipeline, based on a common infrastructure and installed with the VLT Data Flow System (DFS).
With the increase in the number and the complexity of supported instruments and in the rate of produced data, these pipelines are becoming vital for both the VLT operations and the users, and request more and more resources for development and maintenance.
This paper describes the different pipeline tasks with some real examples. It also explains how the development process has been improved to both decrease its cost and increase the pipelines quality using the lessons learned from the first instruments pipelines development.
The Combined Array for Research in Millimeter-wave Astronomy (CARMA) combines the BIMA, OVRO, and SZA millimeter arrays at a new high elevation site (7200', 2200m). The 23 CARMA antennas have sizes of 3.5, 6.1 and 10.4 meters in diameter and will be the first heterogeneous millimeter array when operational in 2005. The array will have a resolution of 0.1 arcseconds and process a maximum bandwidth of 4 GHz with a maximum resolution of 10 kHz.
The software system encompasses a monitor and control system, an archive and a pipeline. The monitor and control system is a distributed CORBA based system that provides for control of five sub-arrays. Monitor data from the distributed components are aggregated for control and displays and then stored into an RDBMS. The visibility data is stored locally and then moved to the off-site archive over the internet. The pipeline automatically produces images from the archive data.
Relevant technologies for the project include C++, Java, CORBA, CVS, Linux, RDBMS, CANbus, FPGAs and embedded microprocessors. Some software from the existing arrays will be reused, but the majority is new. The software team is distributed across the staff of five universities. Project management techniques include distribution of responsibilities, a well defined schedule, required definition of scope and design, and frequent communication.
An overview of the NOAO Mosaic Pipeline is presented with details provided in other contributions at this conference. The pipeline processes observations from the two NOAO 8K square CCD wide-field mosaic imagers. It produces a variety of data products and data quality measurements for the observers and the NOAO Archive. Application of the pipeline during observations for quick automatic data quality assessment is a key requirement for the pipeline. The implementation is easily configurable for other NOAO and non-NOAO mosaic imagers and, ultimately, other types of instruments.
The pipeline processes observations in near-real time immediately after an observation, for user and support feedback through GUI monitors. It also processes existing queued datasets. The design is highly modular, distributed and parallel to make use of a network of pipeline processing nodes. The processing is performed by a large number of independent modules, mostly IRAF-based, grouped into various interconnected subpipelines. The modules and subpipelines are coordinated by triggers (using the OPUS xpoll mechanism) such that each module independently takes on work when its trigger conditions are satisfied and sets triggers for other modules when it completes. Data management is provided by a Data Transport System (DTS) for input and output of the pipeline and by a Data Manager providing database faciliities and calibration libraries. The pipeline is monitored and controlled by optional GUI components which may be attached to and from the network as needed.
This overview highlights many novel design features, some of which are noted below. The modules and their interactions are described by an XML document which is used to generate the configuration files. The monitoring, control, and data management components plug in via sockets allowing customization and network distribution of the user interface. The data manager is itself a distributed system of databases and file managers connected using CORBA. The load balancing of the pipeline network is accomplished by cooperative processes on each node. The pipeline includes internal crash recovery such that if a module dies unexpectedly it is automatically restarted. The monitoring GUIs use historical data to provide adaptive alarms.
The pipeline is nearing a first operational demonstration for basic instrumental calibration, data quality monitoring, and source catalog generation during observations. Later phases of the pipeline will include dither stacking and time-domain analysis products. The Mosaic pipeline is the pioneer for future pipelines in the NOAO Data Product Pipelines program.
| prev toc next |