The Subaru Telescope is an optical infrared telescope with 8.2m monolithic mirror located at the summit of Mauna Kea, Hawaii, funded 100% by the Japanese government, Ministry of the Education, Culture, Sports, Science, and Technology (MEXT). The Subaru Telescope began operation in January 1999, and began the Open Use program for astronomers all over the world in December 2000 (Kaifu 1998; Noumaru 2002; Iye 2002).
The essence of the Subaru Telescope can be summarized as follows.
The construction of the dome structure at the summit of Mauna Kea, Hawaii, began on July 6, 1992, and the telescope was brought in 1997 December. Meanwhile, the Subaru Headquarters at Hilo, Hawaii (Hilo Base Facility) opened in April 1997. The 8.2m monolithic mirror blank was transported to the summit November 1998, and the mirror emerged after a successful onsite aluminization on November 8, 1998. All eight facility instruments have successfully survived the commissioning process in the following two years to open a new era of astronomy in December 2000; complete open use for all astronomers in the world.
The Subaru Telescope is characterized as a complex system of various mechanisms; telescope structure, enclosing dome structure, air flow and temperature control, and many instruments and the huge amount of data produced by those instruments.
An operational model of the Subaru Telescope was defined as follows.
The three main components of the core system have been abstracted; an observation control system to support the whole observation process, an archive system to support data management, and a data analysis system to support data analysis including automatic pipeline analysis. Through this operation, the quality control of observed data is considered to be a most important concern.
The control system of the telescope and dome structure was developed in close conjunction with the construction of the telescope and dome by Mitsubishi Electric Co. Ltd.(Tanaka 1998), the main contractor of the construction of the Telescope from 1992-1998. The first generation instruments, three in visible wavelengths and five infra-red instruments had been planned to be developed by each instrument team after 1995 (Iye 1998; Iye & Yamashita 2000).
When we started a project to develop a software system for the telescope operation in 1992, none of those instruments was existing, nor was the telescope control system. The telescope control system and instrument control systems were treated as plug-in applications to the core system and an application interface layer was defined. It is advantageous to complete the core software system during the construction of the telescope itself, because the software system could then be used for data acquisition in the final adjustment of the telescope construction.
The milestones were planned and performed as follows.
We named the core software system of the Subaru Telescope data flow operation as the Subaru Telescope Software Trinity System(STsTS) to symbolize that none of those three software components could be considered removable from the system.
The Subaru Software Trinity System(STsTS), consisting of the Subaru Observation Software (SOSS), the Subaru Telescope Archive System (STARS), and the Data Analysis System Hierarchy (DASH) has been developed as another systematic software system of the telescope operation in Japan (Morita 2002). Performing effective observation with a user-friendly environment is the primary purpose of SOSS at the summit of Mauna Kea, 4200 m in the altitude, where there is 40% less oxygen in the atmosphere in comparison with that at sea level.
Observed data is kept in the tape library storage as an archive managed by STARS. DASH is waiting for all observed frames to be treated from series of observations, and invokes an automatic data analysis pipeline process. A supercomputer system STN-I and STN-II has been installed to support STsTS operation at the Hilo Base Facility.
The SOSS is designed to perform effective operations on the Subaru Telescope (Kosugi 1997; Sasaki 1996; Sasaki 1997). The SOSS system consists of five subsystems; Observation Control System (Sasaki 1998), Data Acquisition System (DAQ, Noumaru 1995), Quick Look Analysis (QDAS, Kosugi 1996), Data Base system, and Tool Kit (Noumaru 1997). The Observation Control System controls the interface with the Telescope control system, in a loose network connection on the Ethernet, and the Tool Kit defines the interface with the instrument control systems (OBCP) that has been developed independently by the instrument development teams. Even for testing facility instruments and private instruments, the Tool Kit has been provided and adopted as a standard interface between SOSS and the instruments. Simulator software of the telescope control system is provided in the Tool Kit so that OBCP could have done testing even without the actual connection of the telescope control system to SOSS.
During the commissioning procedure of eight facility instruments, the testing dedicated for the software interface between SOSS and OBCP took a few days at the simulator laboratory in Hilo Base Facility and another couple of days at the summit telescope site, and the introduction of Tool Kit Interface to instruments were quite successful.
The actual observation process continues based on the Observation Procedure, that is prepared by observers based on the observation plan in advance. A supporting interface for creation of Observation Procedures is provided. The Observation Procedure includes the Observation Data Set (ODS) detailing relationships among various frames (target frames, standard star frames, flat frames, and other calibration frames) with IF-THEN rules. The ODS is a key component in data flow in STsTS operation (Kosugi 2000; Kosugi 2002) as described in Section 4.
The ODS is written with Abstract Commands which are defined to be a common language among various kind of instruments (Kosugi 1998). The Abstract Commands are expanded by the Observation Control Subsystem into specific commands for each instrument. Execution of ODS could be dynamically optimized by adopting an optimization process (Sasaki 2000) based on SPIKE. Using Abstract Commands, observers can use the same language such as ``Get_Object'', ``Begin_Exposure'' and so on for all instruments attached to the Subaru Telescope.
The operation of the SOSS runs on the closed system at the summit site in order to maintain a secure environment as well as stable operation. A synthesized GUI for observers or operators (even in the case of usage during the engineering & maintenance period of the telescope or instruments) are prepared. A quick look analysis of observed data is supported by the QDAS subsystem which is developed on DASH & SASH (see Section 3.3) environment. Monitoring functions of various kinds of status information of telescope, dome condition, and environmental conditions has been equipped.
The SOSS system has been designed to work as an autonomous software system and interfaces with instruments and telescope are implemented independently, so it is rather simple to run SOSS remotely from Hilo Base Facility or even from Mitaka Headquarters in Japan as far as a critical condition of the interface can be satisfied. Since the interface is defined on the basis of command & status protocol, the condition is written as short enough round trip time among relevant processes. Testing of remote operation has been successfully carried on from Hilo Base Facility and it will be open to Open Use in near future. We are planning to realize the remote operation from Japan in near future.
SOSS consists of 3200 modules, 690,000 steps approximately.
STARS supports online registration of observation data in close relation with SOSS, and offline data retrieving with DASH as well as a WWW interface for astronomers. The design of STARS began in 1997 to achieve the following features (Takata 1996; Takata 1998; Takata 2000 ASP; Takata 2000 SPIE):
Oracle 7 had been adopted initially as the database management system in STN-I, and it is running with Oracle 8 on STN-II. Several terabytes of raw data have been archived so far, and will be increasing by a few terabytes per year, due to the enhancement of data production rate by improved instruments.
In STARS, all observation proposals are assigned with UNIX group attributes which is common among SOSS and DASH. Each observer can handle data through a WWW interface of STARS during and after the observation, by accessing both servers at Hilo or at Mitaka with suitable authentication according to the UNIX group assignment. Each item in the FITS header of frames is categorized suitably for retrieving.
An interface between STARS and DASH is provided so that STARS can handle not only the observed data, but also environmental data, Observation Data Set (see Section 4), and telescope logging data. The primary purpose of STARS is to provide fast and automatic retrieval through the DASH pipeline processing. The baseline of the design is to use the high performance computing power of STN-II for the analysis particularly as an automated pipeline process, and no bulk transfer of observation data to the other site is supported so far. Various kind of removable media instead, such as DLT, DAT, ATL, and so on, are available for remote users.
As for the communication between Hilo and Mitaka, STARS has been designed to support loosely coupled remote database management from the beginning. The faster the network bandwidth between Subaru Hilo Base Facility and Subaru Mitaka Headquarters, the closer the relation between two databases (Takata 2002). As of December 2001, there has been established STM1/OC3 cable network between Hilo and Mitaka Headquarters, and a closely coupled operation of STARS and MASTARS, the replica of STARS at Mitaka, began operation as a standard communication. In Japan, an academic 10 Gbps backbone network, Super-SINET8 is utilized to deliver data among researchers in universities and institutes.
STARS consists of 600 modules, 90,000 steps approximately.
The DASH project, based on the Object Oriented Method and CORBA, began in 1996 (Mizumoto 1998 ASP; Mizumoto 1998 SPIE; Yagi 1998), and completed in March 2002 (Mizumoto 2000; Yagi 2000; Yagi 2002).
The basic concept of DASH can be summarized as follows.
In order to develop the project, we adopted an Object Oriented Method (OOM) for research target oriented purpose. Prototyping began in 1996 to show how the concept of DASH can be implemented on a distributed computer environment. A metaphor of Restaurant Model has been developed to clarify how the astronomer needs to treat or analyze data. The model can be easily mapped into OOM modeling for programmers who were belonging to the software house, not to the astronomical observatory. The model has three tiers; 1) user interface (waiter and dish) 2) process agents (chef and manager) 3) database (warehouse), where the restaurant metaphor is shown in parentheses.
A unit of processing in DASH is defined as a PROCube (an abbreviation of PROcess Cube). The PROCube contains all information required for the data analysis as frame categories in the X-axis, number of frames in the Y-axis and reduction flows in Z-axis. Reduction in the Z-direction runs with various engines, that need to be wrapped for existing generic modules (e.g., eclipse), or newly developed for each of Subaru's instruments.
Furthermore, in order to support a typical application of the DASH pipeline for the mosaic CCD frames, the Hierarchy which contains PROCubes as components are implemented. Thus, there are three classes of reduction procedure, Engine, PROCube, and Hierarchy in the DASH pipelines, where PROCube contains Engines, and Hierarchy contains PROCubes.
The logging and management information of PROCube execution is stored in the PROCube itself to make reproduction of the result of analysis possible. Since engines are stateless, all parameters necessary for the execution of engines need to be kept in the PROCube.
The concept of PROCube can be useful even without a complete set of database and engines with CORBA naming services. Then, a single user interface of DASH (it is named as SASH) can be detached from the DASH system to make available to observers offline analysis locally on her/his lap-top PC with the same interface as DASH. This is actually applied to the quick-look analysis subsystem(QDAS) of SOSS during the observation. JavaIDL is running as an Object Request Broker for SASH.
As for base of the DASH, the Object Director of Fujitsu INTERSTAGE (initially CORBA 1.0, then CORBA 2.3) has been implemented. JAVA2 (initially JDK 1.2.0, then JDK 1.3.1) has been used for the user interface. As browsers, ESO-skycat and SAOimage-DS9 are implemented.
Subaru is equipped with eight standard facility instruments and observational data taken with five of those instruments (Suprime-Cam, CIAO, HDS, OHS/CISCO and FOCAS) are now ready to be analyzed by dedicated pipelines constructed on the DASH platform. We are opening a new production phase of pipeline packages for numerous variations of observation modes, and will be adopted to the big survey project planned in collaboration with several other telescopes.
DASH consists of 4000 modules, 110,000 steps approximately.
The amount of observing data produced by the Subaru Telescope had been estimated to be 50GB/night in the beginning of the development of STsTS, and actually, more than 30GB of data/night was produced in the year 2001 by the Suprime-Cam imaging observations. For handling huge amount of data with STsTS, we have introduced a super computer system, the Subaru Telescope Network with hierarchical storage capability up to several petabytes. The first system, STN-I, was installed in March 1997 (Ogasawara 1998), with 150TB storage, and successfully having supported the STsTS operation, then upgraded in 2002 March (Ogasawara 2002 SPIE 4844-44), STN-II with 600TB tape library. A satellite system at Mitaka Headquarters with 120TB storage has been installed.
SOSS is operating on the summit subsystem of STN that is connected by OC12 optical fiber network with the Supercomputer system at the Hilo Base Facility. Gigabit Ethernets are installed as backbones on each subsystem, and Storage Area Network with fiber channel switches are supporting reliable and secure data transfer among networks.
STARS and DASH are running on the supercomputer system in Hilo, as well as on the satellite system located in Mitaka. A high speed OC3 network over the Pacific Ocean has been in use since December 2001 (Ogasawara 2002 SPIE 4845-02).
Thus, STN-II is supporting full function of STsTS on distributed locations among the summit telescope site, Hilo Base Facility, and Mitaka Headquarters. The network connections among sites are operated by Subaru and the STN-II system is installed from Fujitsu Ltd. as a limited term operational contract9. Real time quality control using nearly tera-flops computing power to find suitable parameters for proceeding observations is planned and will be introduced on STsTS in near future.
According to the observation procedure defined in SOSS, all information of data frames in principle can be described in the Observation Dataset (ODS). In a modern CCD observation, a target frame needs to be verified after several calibration processes. Since the quality of calibration frames directly reflects the quality of the target frames, it is important to gather enough qualified calibration frames during observations. By adopting ODS-rules properly, the observer could get sufficient calibration frames.
As the observation proceeds, frames that match the rule described in the ODS ensemble could be filled and the DASH pipeline will begin automatically to execute proper PROCubes, or Hierarchies. DASH issues a retrieving request to STARS to find relevant frames, either target frames or calibration frames, according to the description of the ODS and fulfilled the data frames in PROCube. Thus, the quality control of the target frame can be achieved automatically by the cooperative function of SOSS, STARS, and DASH through the concept of the Observation Data Set.
After completion of development of the three components of STsTS, we did observe using the Suprime-Cam instrument, the mosaic CCD attached to the prime focus of the Subaru Telescope (Miyazaki 1998). An observation procedure was prepared according to the observation plan and ODS rules were applied. The observation began around 02:00am HST on June 22, 2001 and automatic calibration ended around 09:30am HST to get images of pixels without any human interruption. This was another first light of the Subaru Telescope by STsTS, that is the software first light.
Since there is no direct link at this moment with the environmental data, observers are careful enough to watch the weather conditions. However, we hope it is possible to detect a change of several key variables, e.g., seeing, transparency, point-spread-function and so on, it is not impossible to maintain the quality by the software to a certain degree. It would be advantageous in a survey observation in which homogeneous quality is one of the most important features of the survey data. We are planning to apply the quality controlled data flow with STsTS in several survey projects using the Subaru Telsecope.
Since the data flow operation based on the ODS runs automatically in principle, it will be possible to control the quality by applying suitable observation parameters such as exposure time, interval for calibration frames and so on, by real time, and detailed analysis of observing data. For this purpose, a huge amount of computation power is required, using the supercomputers of STN-II.
Not only a real time response during the observation, but also for preparation of the next observation plan, the STARS archive and DASH can be helpful for retrieving previous results (Noumaru 1998; Noumaru 2000; Ogasawara 2000). Even without actual observations, STsTS is available to find suitable calibration frames from the STARS archive, then re-analyze the target frames afterward, to achieve a required quality level. This is a most important feature of STsTS; to keep the data obtained by the Subaru Telescope precious enough to be fed to the Japanese Virtual Observatories (Mizumoto 2002) as reliable data.
The archive in STsTS is basically raw frames so far, it is strongly desired to prepare a scientific archive that keeps well-calibrated frames. What can be achieved by STsTS is to use CPU power to analyze frames according to the re-defined ODS-rules recursively to get a scientific result repeatedly, and it would be implemented as a basic tool for browsing Subaru data.
The development and operation of STsTS is summarized as follows.
The STsTS project has been a close collaboration with a software house, Fujitsu Ltd., and related corporations. In considering the situation in Japan, such a contract seems to have been the only possible way to develop a complicated software system, with foolproof or need to work basis.
After the three years successful operation of the Subaru Telescope, the next generation of the STsTS components; SOSS, STARS, and DASH will be developed based on experiences of the STsTS operation especially taking the possibility of a much more efficient system to support data flow operation of the Subaru Telescope into account. For example, development of a more user-friendly GUI, a supporting system of remote observations, and more efficient observation including optimization of ODS could be the possible targets of improvement in the future. Current STsTS is running on the Solaris-based UNIX system, and selection of operating system would be another important issue.
Since the development and operation of the Subaru Software Trinity System is a long-term project, there have been many collaborators. We are thankful to Mr. Y. Kosaka and Mr. H. Monzen of Fujitsu Ltd. and Dr. N. Kaifu, the Director-general of National Astronomical Observatory of Japan. The members who were attending discussion and contributed to the basic design of the Subaru Software Trinity System was as following; Dr. T. Kato (Kyoto University), Dr. T. Aoki (University of Tokyo), Dr. H. Baba (ISAS), Dr. M. Taga (Waseda University), Dr. T. Horaguchi (National Science Museum, Tokyo), Dr. M. Hamabe (Nippon Univ.), Mr. N. Koura, Mr. S. Honma (Fujitsu Ltd.), Mr. M. Furuichi, Mr. T. Yamamoto (Fujitsu America Inc.), Mr. T. Kusumoto (FFC Ltd.), Mr. I. Akiyama, Mr. Y. Kusama, Mr. S. Iwai, Mr. K. Ooto, Mr. Y. Morita, Mr. N. Watanabe, Mr. K. Ukawa, Mr. H. Nakamoto, Mr. T. Yagasaki (Systems Engineering Consultants Co. Ltd.), Mr. S. Wakabayashi, and Mr. T. Goto (MAIN Corp.).
Without the precious ideas and comments given by those collaborators, and without the stellar effort of them in coding and testing, the Subaru Telescope Software Trinity System could not have been achieved successfully as it is. The project has been supported by the budget for the construction of the Subaru Telescope during 1995-2001, and now goes into the stable era of operation since 2002 Apr. We are grateful for Dr. K. Kodaira, Dr. H. Karoji, Dr. H. Ando, Dr. M. Iye and all Subaru related personnel for supporting and helping the project.
Iye, M. 1998, in SPIE Proc., Vol. 3355, 344
Iye, M. & Yamashita, T. 2000, in SPIE Proc., Vol. 4008, 18
Iye, M. et al. 2002, PASJ December, in Press
Kaifu, N. 1998, in SPIE Proc., Vol. 3352, 14
Kosugi, G., et al. 1996, in ASP Conf. Ser., Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes (San Francisco: ASP), 404
Kosugi, G., et al. 1997, in SPIE Proc., Vol. 3112, 284
Kosugi, G., et al. 1998, in SPIE Proc., Vol. 3349, 426
Kosugi, G. et al. 2000, in SPIE Proc., Vol. 4010, 174
Kosugi, G. et al. 2002, in SPIE Proc., Vol. 4833, in Press
Miyazaki, S. et al. 1998, in SPIE Proc., Vol. 3355, 363
Mizumoto, Y. et al. 1998, in ASP Conf. Ser., Vol. 145, Astronomical Data Analysis Software and Systems VII, ed. R. Albrecht, R. N. Hook, & H. A. Bushouse (San Francisco: ASP), 332
Mizumoto, Y. et al. 1998, in SPIE Proc., Vol. 3349, 173
Mizumoto, Y. et al. 2000, in SPIE Proc., Vol. 4009, 429
Mizumoto, Y. et al. 2003, this volume, 96
Morita, K. et al. 2003, this volume, 166
Noumaru, J. et al. 1995, in SPIE Proc., Vol. 2479, 453
Noumaru, J. et al. 1997, in SPIE Proc., Vol. 3112, 52
Noumaru, J. et al. 1998, in SPIE Proc., Vol. 3349, 195
Noumaru, J. 2000, in SPIE Proc., Vol. 4010, 10
Noumaru, J. 2002, in SPIE Proc., Vol. 4844-02, in Press
Ogasawara, R. et al. 1998, in SPIE Proc., Vol. 3349, 255
Ogasawara, R. et al. 2000, in SPIE Proc., Vol. 4010, 159
Ogasawara, R. et al. 2002, in SPIE Proc., Vol. 4845-02, in Press
Ogasawara, R. et al. 2002, in SPIE Proc., Vol. 4844-44, in Press
Sasaki, T. et al. 1996, in ASP Conf. Ser., Vol. 125, Astronomical Data Analysis Software and Systems VI, ed. G. Hunt & H. E. Payne (San Francisco: ASP), 196
Sasaki, T. et al. 1997, in SPIE Proc., Vol. 3112, 267
Sasaki, T. et al. 1998, in SPIE Proc., Vol. 3349, 427
Sasaki, T. et al. 2000, in SPIE Proc., Vol. 4009, 350
Takata, T. et al. 1996, in ASP Conf. Ser., Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes (San Francisco: ASP), 251
Takata, T. et al. 1998, in SPIE Proc., Vol. 3349, 247
Takata, T. et al. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, ed. N. Manset, C. Veillet, & D. Crabtree (San Francisco: ASP), 157
Takata, T. et al. 2000, in SPIE Proc., Vol. 4010, 181
Takata, T. et al. 2002, in SPIE Proc., Vol. 4844-52, in Press
Tanaka, W. et al. 1998, in SPIE Proc., Vol. 3351, 478
Yagi, M. et al. 1998, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, ed. David M. Mehringer, Raymond L. Plante, & Douglas A. Roberts (San Francisco: ASP), 375
Yagi, M. et al. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, ed. N. Manset, C. Veillet, & D. Crabtree (San Francisco: ASP), 510
Yagi, M. et al. 2002, in SPIE Proc., Vol. 4847-33, in Press