Next: On the Analysis of Old Objective-Prism Plate Spectra with Modern Systems
Up: Surveys & Large Scale Data Management
Previous: Data Archive and Transfer System (DARTS) of ISAS
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint

Salama, A., Ortiz, I., Arviset, C., Dowson, J., García-Lario, P., Gry, C., Hernández, J., Lorente, R., Matagne, J., Osuna, P., Salgado, J., & Verdugo, E. 2003, in ASP Conf. Ser., Vol. 314 Astronomical Data Analysis Software and Systems XIII, eds. F. Ochsenbein, M. Allen, & D. Egret (San Francisco: ASP), 26

User provided reduced data, catalogues and atlases in the ISO Data Archive

Alberto Salama, Iñaki Ortiz, Christophe Arviset, John Dowson, Pedro García-Lario, Cécile Gry, Jose Hernández, Rosario Lorente, Jean Matagne, Pedro Osuna, Jesus Salgado, Eva Verdugo
ISO Data Centre, RSSD, ESA VILSPA, PO BOX 50727, 28080 Madrid, Spain


The ISO Data Archive contains products obtained by processing the observations data through an automatic processing pipeline. This corrects well a number of instrumental artifacts in an automatic fashion. The final products can however be improved by processing them further, e.g. by means of the Interactive Analysis software packages. The resulting products are called Highly Processed Data Products (HPDP). In a broader sense this definition includes catalogues and atlases. In July 2003, the ISO Data Archive has been enhanced with the functionality for continuous ingestion of new data, catalogues and atlases.

1. Introduction

Five years after the end of the Infrared Space Observatory (ISO) operations, its data continues to be an exemplary resource for scientific exploitation. To date, over 1100 papers based on ISO results have been published in refereed literature. ISO data has yielded an abundance of exciting discoveries and many more are still to be expected. Some papers are based on a systematic reduction of ISO data, producing the so-called 'Highly Processed Data Products' (HPDP). These products include data (images, spectra etc.), which have been processed beyond the automatic processing pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the legacy pipeline products, as well as any resulting catalogues and atlases. In this direction, projects have been undertaken by the ISO Data Centre (IDC), in collaboration with the national instrument data centres, for systematic data reduction of specific instrument modes, that will produce homogeneous sets of HPDP. The ISO Data Archive (IDA) has been enhanced to host these new products. Version 6, released in July 2003, has the functionality for continuous ingestion of new data, catalogues and atlases, subject to screening by the ISO Data Centre. All datasets are queriable and retrievable, in a user-friendly way, and are well separated from the Legacy Off Line Processing Pipeline products. This is one of the core activities of the current phase of ISO, the Active Archive Phase, running until the end of 2006.

2. The ingestion of Highly Processed Data Products in IDA

Reduced Data, Catalogues and Atlases can be ingested in the archive at any time. Simultaneously with the ISO Data Archive V6 release, a campaign was launched to get from the scientific community the results of the reduction work. Simple procedures have been built which minimise the work of the authors for the actual ingestion, still preserving some homogeneity in the presentation to the archive user, by which a designated IDC contact astronomer solicit relevant information from the author and does the actual ingestion. This happens via a web form filled in by the IDC astronomer, with administrative and technical information about the Highly Processed Data Products and a list of relevant files. The metadata (author, title, abstract, etc.) is stored in the IDA database and all the files related to a particular observations are stored in a directory named with the observation identifier, plus a running number if multiple. Catalogues are ingested in the archive by providing the RSSD Archive Development Team with an associated ASCII description file containing all column descriptions, following a simple format specification. The ingestion of HPDP is thus semi-automatic, requiring very little human intervention for it to be visible on-line worldwide.

3. HPDP query and retrieval from the ISO Data Archive

Simultaneous with the IDA 6 release, the ISO home page had been slightly modified to allow for an enhancement of the HPDP visibility. The HPDP has previously been called User Reduced Data and did not include user provided catalogues and atlases. On the ISO Home Page the ISO Data Archive is easily accessible, as well as the list of HPDP currently stored. The list is generated on-the-fly from the IDA database.

Once getting the IDA Java applet, the user may open the dedicated Highly Processed Data Products Panel (see Figure 1). A description of each HPDP set can be obtained by clicking on the title. HPDP of interest can be selected via the check box on the left. Combinations with other query panels are also possible. For HPDP associated with a catalogue, the button "Search catalogue" allows to obtain a query panel for the specific catalogue. This is generated on-the-fly from the database (see an example in Figure 2). A description of each catalogue column can be obtained by clicking on the column title, coloured in blue (as for any other field of the IDA).

The general `Execute query' button will return a list of observations matching the given selection criteria.

In the query results panel, as for any other product, the HDPD can be retrieved directly on disk by pushing `Retrieve' and selecting `HPDP'. For more information and features, the HPDP button at the left of any given observation may be clicked (if highlighted). This will lead to a new panel (see an example in Figure 1).

An abstract will be given of all HPDP sets containing the observation. Six buttons will be displayed on the left:

  1. Data
    Clicking this button will allow to download the data directly.

  2. Documentation
    This is generally information stored in the IDA in the form of a web page, launched by the default browser, or a PDF file, with links to specific documents or, as applicable, to the paper(s), via the ADS service.

  3. Postcard
    An illustration (plot, image) of the reduced observation. This will be obtained by automatically launching the user's default browser.

  4. Atlas
    This is defined as an illustration of the observations together with other information provided in the HPDP set, such as external SEDs, stellar models, etc.

  5. Catalogue
    A new panel will be launched, pointing to the raw containing the observation. The whole catalogue may be saved on disk or printed. A search facility is also provided (see an example in Figure 2).

  6. Add Related
    Clicking this button will move all observations pertaining to the given HPDP set to the shopping basket for retrieval in one go.

Figure 1: The IDA applet with the HPDP query panel open (left) and an example of HPDP panel for a given observation (right)
\epsscale{0.4} \plottwo{P1-3_1.eps}{P1-3_2.eps} \end{figure}

Figure 2: Examples of HPDP catalogue query panel and catalogue display
\epsscale{0.4} \plottwo{P1-3_3.eps}{P1-3_4.eps} \end{figure}

4. Conclusion

A mechanism is fully set to allow continuous ingestion of user provided reduced data, catalogues and atlases in the ISO Data Archive. This mechanism permits some homogeneity in their presentation, although the products are quite heterogeneous by definition. A number of special projects focussed on reducing data from selected observing modes have already been carried out and we have started to gather HPDP sets from the community. Systematically reduced data will continue to be ingested in the IDA throughout the Active Archive Phase. This is a valuable asset and will boost the scientific exploitation of ISO data by the general astronomical community by providing readily useable scientific data products.

The ISO Data Archive is subject to more upgrades throughout the Active Archive Phase, such as boosting interoperability aspects (see for example Osuna et al., 2004) or enhancing the quality information of the ISO data.


Kessler, M.F. et al. 2003, The ISO Handbook, Vol. 1, ESA-SP 1262

Osuna, P., Arviset, C. & Salgado, J. 2004, this volume, 129

© Copyright 2004 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: On the Analysis of Old Objective-Prism Plate Spectra with Modern Systems
Up: Surveys & Large Scale Data Management
Previous: Data Archive and Transfer System (DARTS) of ISAS
Table of Contents - Subject Index - Author Index - Search - PS reprint - PDF reprint