Next: The D4A Digitiser
Up: Surveys & Large Scale Data Management
Previous: Increasing the Accessibility of Green Bank Telescope Data
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Hiriart, R., Valdes, F., Pierfederici, F., Smith, C., & Miller, M. 2003, in ASP Conf. Ser., Vol. 314 Astronomical Data Analysis Software and Systems XIII, eds. F. Ochsenbein, M. Allen, & D. Egret (San Francisco: ASP), 74

The NOAO Pipeline Data Manager

R. Hiriart, F. Valdes, F. Pierfederici, C. Smith and M. Miller
National Optical Astronomy Observatories, Tucson, AZ 85719


The Data Manager for NOAO Pipeline system is a set of interrelated components that are being developed to fulfill the pipeline system data needs. It includes: (1) management of calibration files (flat, bias, bad pixel mask and xtalk calibration data.); (2) management of the pipeline stages' configuration parameters; and (3) management of the pipeline processing historic information, for each of the data products generated by the pipeline.

The Data Manager components uses a distributed, CORBA based architecture, providing a flexible and extensible object oriented framework, capable of accommodating the present and future pipeline data requirements. The Data Manager communicates with the pipeline modules, with internal and external databases, and with other NOAO systems such as the NOAO Archive and the NOAO Data Transport System.

1. Introduction

The NOAO Pipeline is a parallel and distributed system that processes observations from the two NOAO 8K-square CCD wide-field mosaic imagers in near real time. It performs basic CCD reduction, instrumental features removal, astrometric calibration, and zero point calibration (Pierfederici 2004; Valdes, Miller, this conference).

The core of system is composed of several processing modules, which are connected to form sub-pipelines. The OPUS1system is used to control the process execution environment for each of the modules.

The Data Manager is a subsystem of the NOAO Pipeline that provides several data management services to the system. These services include storage and retrieval of calibration files, management of pipeline module configuration parameters, and recording of processing information.

2. Architecture

The extensibility of the Data Manager has been an important goal in the design, in order to facilitate integration of new services as the pipeline system evolves and new requirements arise. Therefore, the Data Manager contains several components implemented as distributed CORBA objects.

Figure 1 presents the overall architecture of the Data Manager. The system is structured in three layers: the Interface layer, the Service layer, and the Resource layer. The Interface layer exposes the Data Manager services to the clients by calling the appropriate component(s) in the Service layer upon receiving a request. The Service layer, on the other hand, uses the components in the Resource Layer to access a number of internal services, such as persistent data in relational databases, and file storage in multiple repositories.

The Interface layer is composed of the following subsystems:

The subsystems in the Service layer implements the ``business logic'' of Data Manager operations. The services are based on a rich object-oriented data model that is maintained in a PostgreSQL database. The subsystems of this layer are:

Figure 1: Overall architecture of the Data Manager system.

The Service layer uses components from the Resource layer, which is composed by the following subsystems:

3. Conclusions

The NOAO Data Manager is being developed as part of the NOAO Pipeline project, with the goal of providing an extensible, robust, and high performance solution for the data management needs of the pipeline system, as well as to allow the integration of other NOAO systems that depends on the data generated in the pipeline executions, such as the NOAO Archive system.

The NOAO Data Manager is being developed using distributed object middleware and relational databases. These technologies provide the necessary framework over which this application can be successfully constructed.


Pierfederici, F., Valdes, F., Smith, C., Hiriart, R., & Miller, M. 2004, this volume, 476 .


... OPUS1

© Copyright 2004 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: The D4A Digitiser
Up: Surveys & Large Scale Data Management
Previous: Increasing the Accessibility of Green Bank Telescope Data
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint