Next: ORAC-DR: Pipelining With Other People's Code
Up: Software Development and Management
Previous: The Gemini Data Handling System: A Case History
Table of Contents -
Subject Index -
Author Index -
Search -
PS reprint -
PDF reprint
Ott, S., Gastaud, R., Guest, S., Delaney, M., Altieri, B., Ali, B., Abergel, A., Auguéres, J.-L., Aussel, H., Bernard, J-P., Biviano, A., Blommaert, J., Boulade, O., Boulanger, F., Cesarsky, C., Cesarsky, D., Chary, R-R., Charmandaris, V., Claret, A., Delattre, C., Désert, F-X., Deschamps, T., Didelon, P., Elbaz, D., Gallais, P., Ganga, K., Helou, G., Kong, M., Lacombe, F., Landriu, D., Laurent, O., Le Coupanec, P., Li, J., Metcalfe, L., Okumura, K., Perault, M., Pollock, A., Roman, P., Rouan, D., Rupen, M., Sam Lone, J., Sauvage, M., Siebenmorgen, R., Starck, J.-L., Tran, D., Van Buren, D., Vigroux, L., & Vivares, F. 1999, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, eds. D. M. Mehringer, R. L. Plante, & D. A. Roberts (San Francisco: ASP), 7
From a ``Launch Readiness'' System to an Astronomical Data Processing
System - a Review of Four Years of CIA Development
S. Ott
ISO Data Centre, Astrophysics Division, Space Science Dept. of
ESA, Villafranca, P.O. Box 50727, 28080 Madrid, Spain, Email:
sott@iso.vilspa.esa.es
R. Gastaud
DAPNIA/SEI-SAP, CEA/Saclay, F-91191 Gif sur Yvette Cedex, France, Email:
RGastaud@CEA.fr
S. Guest1,
M. Delaney2,
B. Altieri3,
B. Ali4,
A. Abergel5,
J.-L. Auguéres6,
H. Aussel7,
J-P. Bernard8,
A. Biviano9,
J. Blommaert10,
O. Boulade11,
F. Boulanger12,
C. Cesarsky13,
D. Cesarsky14,
R-R. Chary15,
V. Charmandaris16,
A. Claret17,
C. Delattre18,
F-X. Désert19,
T. Deschamps20,
P. Didelon21,
D. Elbaz22,
P. Gallais23,
K. Ganga24,
G. Helou25,
M. Kong26,
F. Lacombe27,
D. Landriu28,
O. Laurent29,
P. Le Coupanec30,
J. Li31,
L. Metcalfe32,
K. Okumura33,
M. Perault34,
A. Pollock35,
P. Roman36,
D. Rouan37,
M. Rupen38,
J. Sam Lone39,
M. Sauvage40,
R. Siebenmorgen41,
J-L. Starck42,
D. Tran43,
D. Van Buren44,
L. Vigroux45,
F. Vivares46
Abstract:
The ISOCAM Interactive Analysis System (CIA) is used to calibrate and to
perform the astronomical data processing of data from ISOCAM, the infrared
camera on board the Infrared Space Observatory (ISO).
CIA is generally available to the astronomical community and runs under DEC
VMS Alpha, Solaris, DEC Unix, Debian (PC) Linux and HP/UX. More details,
including how to obtain CIA, can be found at
http://www.iso.vilspa.esa.es/users/expl_lib/CAM_top.html.
We discuss the challenges faced in this multi-site, multi-environment
project throughout the complete software development cycle, outline
the chosen approach, and review the lessons learned.
Starting mid 1994, the ISOCAM Interactive Analysis System
(CIA)47 was developed to support the calibration and operation
of ISOCAM, the infrared camera on board of ESA's Infrared Space Observatory
(ISO)48.
As planned, this system now includes the functionality to perform the
astronomical data processing of data from ISOCAM, and is used within the ESA
ISO Data Centre, by the ISOCAM consortium and by many other institutes.
The system is mainly IDL based, and contains currently 1100 IDL modules with
about 220000 lines of code and comments. CPU intensive tasks are coded
in C++.
Some requirements on CIA were
- expected to change with time (from a calibration/monitoring system for
operational use to an astronomical data analysis system)
- conflicting (flexibility vs. user-friendliness)
- very broad (CIA had to be prepared for the unexpected and to serve as a
development bench for new algorithms, which ultimately will migrate into the
general pipeline)
- unforeseen additional major requirements, e.g., CIA had
also to run on Unix.
The development of CIA had to be spread over multiple development sites.
Furthermore, the schedule for the first version (``launch readiness version'')
was extremely tight -- 9 months from the start of development to acceptance
testing, and the development team also had to build up
IDL49/C++
expertise.
As in every other software project, special attention had to be paid to
software quality control and documentation. The single most difficult and
time-consuming problem was to get the astrometry, e.g., the
correlation of image elements (detector pixels) with astronomical
(sky) coordinates, right. (ISOCAM contains two detectors, which are
read-out electronically in different directions. Also its images might
be re-binned to non-square pixels.)
In order to keep the system on track, requirements were strictly prioritized.
The system is reviewed and, if necessary, pruned from time to time. In order to
fulfil ``expert'' and ``normal'' user requirements, two different types of data
structures were implemented. CIA was coupled with the general automatic data
reduction pipeline via its architecture, not via the code. This eased
development for both systems.
Project management and control was closely entwined with software
configuration control: One VMS system was chosen as the master
development/configuration control system and mirrored to the other VMS
development system. Unix versions are created at monthly or bi-monthly
intervals from the current development version and installed at the other
development sites.
The collaborative and collegial approach within the software team and among
collaborators was essential to overcome all difficulties.
A CIA configuration control board reviews and sets the priorities of the
implementation of software problem reports and software change requests. This,
together with a several-month-long testing period before the release of any
major new version, proved essential to achieving a high quality for the
software.
For the documentation, emphasis was put on the production of an accurate and
helpful User's Manual. CIA's internal help system relies on the information
contained in module headers. Therefore all contributors were continuously
encouraged to maintain them in case of a code change or to modify them in case
of deficiencies spotted.
- 1.
- Management
- the effort to code properly and to maintain a stable system is normally
underestimated by all participants and by their hierarchy.
- quite often it is assumed that coding can be done as a by-product of
algorithmic development. However, it was found that a proper balance in
staffing between astronomers and software engineers is not only essential to
produce a maintainable system but actually more efficient!
- frequent ( monthly) workshops between developers, and bi-annual
users meetings, are essential.
- 2.
- Architecture
- a proper definition of data structures is important. Growing data
structures and flexible export tools are preferable to hard coded data
structures and export tools which have to be upgraded in case of new
requirements.
- link systems with a different primary programming language
only via the architecture, not via the code. Otherwise the
progress of both systems will be slowed down.
- if possible, avoid non-standard data structures depending on pointers. If
non-standard data structures are necessary, provide an intuitive access. (NB:
IDL does not provide this functionality).
- while collaboration is the key to success, some rules have to
be obeyed:
- avoid duplication of already existing functionality.
- enforce consistent naming conventions for routines and for the tag names
within data structures.
- avoid private ports to other operating systems.
- avoid operating-system-dependent data files.
- store the processing history as part of the data instead of using
logfiles.
- 3.
- Implementation
- IDL permits an easy port between systems, compared to classical
languages. (But the support of different operating systems remains
nevertheless an overhead). However, internal IDL changes from version 3.6 to
5.0 for the widget interface led to a major recoding effort.
- the maintenance of module headers is a Sisyphean task which needs
dedicated manpower.
- knowledge of ``expert'' language features like _extra can speed up
the development significantly.
- look for solutions available in other packages (e.g.,
ASTROLIB). But the use of several packages can lead to compatibility
problems (e.g., name conflicts between functions and variables).
- 4.
- Configuration Control & Multiple Development Sites
- good network connections and easy-to-use configuration control systems
are essential.
- the use of mirrors reduces synchronization effort dramatically.
- to avoid bad surprises for users, both a floating development
version and a frozen user's version are necessary.
Footnotes
- ...
S. Guest1
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain,2
- ...
M. Delaney2
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain,3
- ...
B. Altieri3
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
B. Ali4
- University of Rochester, Rochester, USA
- ...
A. Abergel5
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
J.-L.. Auguéres6
- CEA, Saclay, Gif-sur-Yvette, France
- ...
H. Aussel7
- CEA, Saclay, Gif-sur-Yvette, France
- ...
J-P. Bernard8
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
A. Biviano9
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain,7
- ...
J. Blommaert10
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
O. Boulade11
- CEA, Saclay, Gif-sur-Yvette, France
- ...
F. Boulanger12
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
C. Cesarsky13
- CEA, Saclay, Gif-sur-Yvette, France
- ...
D. Cesarsky14
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
R-R. Chary15
- University of California, Los Angeles, USA
- ...
V. Charmandaris16
- CEA, Saclay, Gif-sur-Yvette, France
- ...
A. Claret17
- CEA, Saclay, Gif-sur-Yvette, France
- ...
C. Delattre18
- CEA, Saclay, Gif-sur-Yvette, France
- ...
F-X. Désert19
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
T. Deschamps20
- CEA, Saclay, Gif-sur-Yvette, France
- ...
P. Didelon21
- CEA, Saclay, Gif-sur-Yvette, France
- ...
D. Elbaz22
- CEA, Saclay, Gif-sur-Yvette, France
- ...
P. Gallais23
- CEA, Saclay, Gif-sur-Yvette, France
- ...
K. Ganga24
- Infrared Processing and Analysis Center, JPL and Caltech,
Pasadena, USA
- ...
G. Helou25
- Infrared Processing and Analysis Center, JPL and Caltech,
Pasadena, USA
- ...
M. Kong26
- Infrared Processing and Analysis Center, JPL and Caltech,
Pasadena, USA
- ...
F. Lacombe27
- DESPA, Observatoire de Paris, Meudon, France
- ...
D. Landriu28
- CEA, Saclay, Gif-sur-Yvette, France
- ...
O. Laurent29
- CEA, Saclay, Gif-sur-Yvette, France
- ...
P. Le Coupanec30
- DESPA, Observatoire de Paris, Meudon, France
- ...
J. Li31
- Infrared Processing and Analysis Center, JPL and Caltech,
Pasadena, USA
- ...
L. Metcalfe32
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
K. Okumura33
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain,4
- ...
M. Perault34
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
A. Pollock35
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
P. Roman36
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
D. Rouan37
- DESPA, Observatoire de Paris, Meudon, France
- ...
M. Rupen38
- National Radio Astronomy Observatory, Socorro, USA
- ...
J. Sam Lone39
- CEA, Saclay, Gif-sur-Yvette, France
- ...
M. Sauvage40
- CEA, Saclay, Gif-sur-Yvette, France
- ...
R. Siebenmorgen41
- ISO Data Centre, Astrophysics Division of ESA, Villafranca del
Castillo, Spain
- ...
J-L. Starck42
- CEA, Saclay, Gif-sur-Yvette, France
- ...
D. Tran43
- CEA, Saclay, Gif-sur-Yvette, France
- ...
D. Van Buren44
- Infrared Processing and Analysis Center, JPL and Caltech,
Pasadena, USA
- ...
L. Vigroux45
- CEA, Saclay, Gif-sur-Yvette, France
- ...
F. Vivares46
- Institut d' Astrophysique Spatiale, Orsay, France
- ...
(CIA)47
- CIA is a joint development by the ESA Astrophysics Division
and the ISOCAM Consortium. The ISOCAM Consortium is led by the ISOCAM PI,
C. Cesarsky, Direction des Sciences de la Matiere, C.E.A., France.
- ...
(ISO)48
- ISO is an ESA project with instruments funded by ESA member
states (especially the PI countries: France, Germany, the Netherlands and the
United Kingdom) and with the participation of ISAS and NASA.
- ...
IDL49
- IDL is the trademark of Research Systems, Inc.
© Copyright 1999 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: ORAC-DR: Pipelining With Other People's Code
Up: Software Development and Management
Previous: The Gemini Data Handling System: A Case History
Table of Contents -
Subject Index -
Author Index -
Search -
PS reprint -
PDF reprint
adass@ncsa.uiuc.edu