We describe the roots of the CPL code, the development process of the CPL and the problems we had to overcome to reach the current stable beta-release of CPL.
As major components of the Data Flow System (DFS), used to
operate the Very Large Telescope (VLT), the pipelines are operation-critical.
They must run in an efficient and stable manner to guarantee high-quality data
output needed for
checking the quality of the observations and monitor the health of
They are also instrument-specific
reduction tools and thus their number keeps growing with each new instrument
installed at Paranal.
At the time of writing,
observational data from seven instruments are automatically
processed by their individual pipelines. The reduction software used in the
pipelines was written either at ESO, or outside by the instrument
As a result, the pipelines currently in operation have been written on
the basis of three different software packages, use different interfaces to
the pipeline infrastructure and, not surprisingly, contain
considerable overlap and duplication of functionality.
The number of instruments commissioned at the observatory continues to
increase, and the operation and maintenance of these pipelines is the
responsibility of ESO with essentially the same number of staff and
As a solution to that dilemma, the concept of a library of the reduction software which is needed for typical tasks of pipelines was formed: the Common Pipeline Library (CPL). Such a common software library should greatly help in solving the problems mentioned above - our silver bullet. It would standardize the two main issues of instrument pipelines: writing the data reduction tasks used (the recipes), and interfacing these recipes with the pipelines.
The concept of the CPL was first proposed in 2001 and since then a team inside the DFS group of ESO's Data Management Division (DMD) has been working on the project. The CPL design and development could not start from a clean slate due to
The CPL would provide all the functionality needed for building the data reduction tasks of pipelines for VLT instruments. It would not be oriented towards an interactive data analysis system, but should be optimized for automatic, batch oriented, pipeline processing. The main users of the CPL would be the groups building pipelines for the VLT, i.e. the DFS group of DMD at ESO and the instrument consortia, and not the general user community. Furthermore, we would only implement concepts which had proven to be commonly used in the operational pipelines so far. Also, it was assumed that the CPL user community, experienced application programmers of instrument pipelines, were knowledgable in writing C-code and familiar with object-oriented (OO) concepts.
The CPL is intended to serve as the basis for all future instrument
pipelines of the VLT, i.e. in particular for the second generation VLT
By using a standard set of components (controlled by ESO)
the operation of CPL based pipelines should become more uniform and robust,
and, especially, the maintenance of them should be simpler and more efficient,
independent of the original authors of the code - whether they be
ESO or outside consortia.
The first pipeline to use CPL would be the GIRAFFE pipeline, and thus serve as the testbed for the CPL implementation and its usability. Existing pipelines would be converted on the basis of available resources in parallel with the actual development work. The CPL shall be released in December 2003.
All the instruments on ESO telescopes (Paranal, La Silla) produce their data in FITS format. Consequently, the internal data format of the CPL is also FITS, and CPL accesses FITS data files via the QFITS library which provides all FITS related functionality needed for pipeline processing. The QFITS software library has been developed at ESO, is specifically tailored to the needs of the VLT pipelines, and adapted to the DICB concept of ESO (i. e. standard, ESO defined dictionaries for the FITS keywords of all data files produced by VLT instruments). Chosing QFITS for the CPL gave us full control of the I/O software, facilitating enormously any changes and updates to the FITS I/O code which became necessary due to the CPL development work. The QFITS source code is available independently of CPL as a stand-alone FITS library.
CPL software is written in standard ANSI-C, employing object-oriented concepts, with emphasis on platform independency and portability. The code is maintained using CVS and distributed as "tarfiles" at regular intervals, and for the generation of the documentation (on-line help, reference manual) we employ the doxygen package.
CPL is streamlined for smooth execution of standard tasks used routinely in the operational pipelines at the VLT. The software supports all the data types as CPL Objects (in the style of C++) which were commonly needed in the instrument pipelines, e.g.:
The CPL does not just provide interfaces for accessing all data structures (e.g. like ESO-MIDAS) but goes farther and contains already all the functions needed to support the basic operations typically needed in pipeline processing. Available functions of the CPL are e.g.:
The CPL also provides a framework for executing recipes in a standardized way via dynamically loadable software modules ( plugins). The concept of plugin interfaces helps in
Requiring recipes to be implemented as Pluggable Data Reduction Modules (PDRM) enforces a certain coding standard which guarantees that software developed by external consortia can be faster integrated into an ESO pipeline and validated. It also reduces the need for in depth knowledge of the recipe implementation right from the beginning. Furthermore, all recipes will have the same look and feel across different instrument pipelines.
Once a given recipe complies to the plugin interface it can be executed by an external application which takes over tasks like command-line parameter processing, collecting input data and distributing data reduction products.
Merging two different software packages into a unified library proved to be more difficult and cumbersome than expected. The eclipse library and the ESO software written for VIMOS were built with quite different ideas about how such code should be written and were also in different stages of development. A very pragmatic and flexible approach favouring code production which was very user responsive, quickly updated and easily accessed by outside users contrasted with a more formal and static, but thus traceable, way of building up software to be delivered to outside users assuming full responsibility for that code. We had to make many compromises, cutting edges and abandoning high flying objectives in order to get the CPL out in time.
The acceptance of CPL by the instrument consortia will determine if this library will solve most of the problems we're currently facing with the rapid increase of pipelines which have to be supported on Paranal.