Next: Algorithms
Up: Software Development and Testing
Previous: Refactoring DIRT
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Sherbert, L. E. & Nagel, L. 2003, in ASP Conf. Ser., Vol. 295 Astronomical Data Analysis Software and Systems XII, eds. H. E. Payne, R. I. Jedrzejewski, & R. N. Hook (San Francisco: ASP), 385

Systems Integration Testing of OPUS and the new DADS

Lisa E. Sherbert, Lauretta Nagel
Space Telescope Science Institute, Baltimore, MD 21218


The Data Archive and Distribution System (DADS) will be entering the IDR (Ingest Distribution Redesign) era soon and more major functions will be shifting from the VMS platforms to various Unix platforms. As the first phase, Distribution, is delivered to testing, interfaces with OPUS and OTFR (On The Fly Reprocessing) will change. We will give a current overview of the OPUS/DADS/OTFR system and identify interface changes that will impact the operators and archive users.

1. Introduction

We illustrate the back-end HST ground systems and the types of data that flow through them, from Goddard Space Flight Center (GSFC) through various pipelines and into the data archive at the Space Telescope Science Institute (STScI). Recent changes to the platforms and software will be highlighted in SMALL CAPS. The OPUS/DADS/OTFR super-system is a mixture of platforms connected by NFS cross-mounted disks, email, and socket connections. The whole super-system is moving towards a Unix-based platform and away from the VMS platforms, making tracking of the various states interesting and the system performance definition difficult. Testing relies on stable environments, attempts to validate the system against its requirements, and then benchmarks the system performance. Here we document the current testing environment.

2. Data Flow Overview

PACOR-A at Goddard Space Flight Center now flows the HST science and engineering data to STScI. STScI picks up the engineering data and runs it through the engineering data pipeline system (EDPS), from whence it is archived. The system now runs on Unix/Solaris instead of VMS.

STScI receives the science telemetry data in the form of POD files and runs them through the data receipt pipeline. The data receipt pipeline puts the POD files into the pre-archive science pipeline. The pre-archive science pipeline reforms and calibrates the data and sends the POD files and the raw data to DADS to be archived. The headers of the raw and calibrated data are used to populate the HST archive catalog via the DADS/Ingest software. The DADS/Ingest subsystem also archives the files directly to storage media and catalogs the file locations. The storage media used in this case is Magneto-Optical (MO) platters stored in jukeboxes. The software that runs the jukeboxes lives on a Solaris Unix machine and it interfaces with both DADS/Ingest and DADS/Distribution.

Non-HST and non-FUSE data are input into MAST with CDs as the storage media. The MAST software manages user distribution requests for this data. The StarView and MAST interfaces are used to peruse the science catalog and to request HST and FUSE data. The DADS/Distribution software manages these user requests.

The Distribution software is moving off the VMS operating system onto Unix. It will now communicate with StarView and MAST via port connections instead of via email files. Distribution will make OTFR requests through IPC connections: putting a message on a shared file system which both pieces of software have access to.

The changes that will impact the end users are:

3. Physical Block Overview

3.1 OPUS

Subsystem PACOR-A

Subsystem SCIENCE

Subsystem EDPS

Subsystem OTFR

3.2 DADS

Subsystem INGEST


Subsystem NSA

Subsystem Archive Database (dadsops, mastops)

Subsystem MAST

Subsystem MAST CDs

Subsystem Observer Interfaces: StarView (Java GUI Application) AND MAST (Web Page)

4. Conclusion

We have presented an overview of the flow of HST data into OPUS science processing and engineering processing systems and from there into the Data Archive and Distribution System. The types of operating systems and recent changes to the software have been highlighted.


The authors would like to thank the members of the Data Systems Branch (and those matrixed into it) for their help in describing parts of this complicated system.

© Copyright 2003 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: Algorithms
Up: Software Development and Testing
Previous: Refactoring DIRT
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint