This paper describes several aspects of Keck observatory software, concentrating on interfaces, infrastructure and technologies rather than on technical details. First of all, KTL is introduced, using the telescope control system as an example and illustrating how it was possible to move from the old pre-EPICS Keck 1 control system to the new EPICS control system with hardly any changes to the KTL keyword interface. Next, software for AO and the Interferometer (which uses AO) is described, with an emphasis on automation and the management of complexity. Finally, some new developments are discussed, notably the increasing use of Java and the move towards the use of CORBA.
Both telescopes now run identical control systems, implemented using the Experimental Physics and Industrial Control System (EPICS, Dalesio et al. 1991). However, the original Keck 1 control system was not based on EPICS; the EPICS control system was written for Keck 2 and then retrofitted to Keck 1 in March 1997.
We require each subsystem to provide access to all its functionality via keywords (possibly with the exception of low-level engineering access). Indeed, subsystem acceptance procedures are normally implemented as scripts which access the subsystem only via keywords.
Keywords are not used for large data arrays. Typical uses of arrays are for binning factors (two elements) or readout windows (four elements). Such arrays are small enough that the value can be encoded as a string in a FITS file if necessary. Larger arrays are occasionally used, e.g. the AO system uses several 349-element arrays for telemetry items (the AO deformable mirror has 349 actuators).
It is often necessary to pass structured data between subsystems. For example, an astronomical target specification includes an (RA, Dec), a coordinate frame, an equinox, an epoch and proper motions. Rather than defining a record structure which would then have to be known on both sides of the interface, we define a keyword for each scalar item and another keyword which, when written, tells the telescope to go to the new target.
% show -s dcs telescop instrume utc telescop = Keck II instrume = LRIS utc = 02:50:10.41 h % modify -s dcs telfocus=12 setting telfocus = 12.000 (wait)
In more complex cases a configuration file must be created (Table 3). Use of generic clients can dramatically reduce development time (because configuration is quicker than programming) and increase reliability (because code gets more widely used and is therefore better tested).
Use of csh is adequate for simple cases but for more complex or event-driven scripts there is increasing use of ktcl and kidl, which are, respectively, KTL's Tcl/Tk and IDL1(Interactive Data Language, not Interface Definition Language!) interfaces. Both ktcl and kidl are implemented as dynamically loadable shareable libraries and both support all KTL facilities, including monitors.
Many instrument specialists and observers write csh keyword scripts. This is encouraged by the simple keyword model, the generic clients, and general familiarity with shell programming.
This was early 1994. We knew that Gemini had adopted EPICS and that UKIRT were looking at it, so we decided to find out more. We were impressed by the system but were worried that its adoption would force our team to make too many changes in their ways of working. For example,
In the event, after an appropriate review process, we decided to proceed with EPICS for the Keck 2 control system. The rest is history.
Given an EPICS system, it is necessary only to create a configuration file in order to access it via KTL. The configuration file, which is read at run-time, defines the keyword names and indicates which EPICS channels they correspond to. Via such a configuration file we were able to define a KTL interface to the new EPICS telescope control system which was almost identical to the old non-EPICS interface. This meant that the KTL GUIs from the non-EPICS system were able to run almost unchanged with the EPICS system.
It goes without saying that such generic tools are only possible if control system objects are accessed by name and if any structures used are known to both client and server (structures where members are accessed by name are not really structures, they are aggregations of scalars!).
For at least the first year of our EPICS development, we made every effort to compartmentalize knowledge. One of us handled EPICS builds, another knew all about device drivers, a third was an expert in the Capfast database generation tool, and so on.
We had sufficient resources, and our group was of a sufficient size, to permit this approach. Getting started with EPICS can be particularly difficult for a very small group.
The only practical solution to this (short of remedying the omission) is to have a local EPICS expert who can fill in the gaps for the rest of the group. This is what we did.
This is really a fact of life for systems such as EPICS, which are created by volunteers at many sites. There are two display managers because they were written at two different sites and, when development started, neither met (nor promised to meet) the other site's needs.
The fact remains that the core system and many of the tools are of the very highest quality. They are used by many groups for many different applications on a wide range of Unix and VME hardware and are therefore much more reliable than one would expect from comparable tools produced for a single project.
CORBA allows a client written in one language to communicate with a server written in another. If the AO WUI's use of RMI were replaced with CORBA, the server could be written in C++ and it would not be necessary to use Java native methods to interface to the legacy systems.
IDL and services CORBA applications use IDL (Interface Definition Language this time!) to specify interfaces in a language-independent way. Some interfaces are pre-defined and come as part of the CORBA implementation. These are referred to as ``CORBA services''.
One such service is the ``Property Service'', which allows a set of named properties and property values to be associated with an object. Such properties and property values are conceptually pretty close to keywords and keyword values.
COKE We are extending the Property Service to create a Keyword Service that we call ``COKE'' ( CORBA KEyword). We are using the Property Service at two levels:
The Property Service does not support the delivery of events to clients when property values change. We are using the TAO Real-time Event Service to add event support to COKE.
The resulting three layer CORBA model is illustrated in Figure 2. Not surprisingly, the main difference from Figure 1 is the replacement of RMI with IIOP (Internet Inter-ORB Protocol) and the use of the COKE interface.
As should be clear from the tone of this section, the COKE Keyword Service is still under active development.
The full-up interferometer system consists of the interferometer itself (delay lines, fringe tracker etc.), two 10m Kecks (each with an NGS AO system), and four 1.8m Outrigger telescopes (each with a fast tip/tilt mirror). It is far more complex than any of our existing systems, and its automated operation has always been a requirement. The big worry, of course, is that, with so many subsystems, we will not be able to achieve acceptable system reliability.
We are dividing the sequencing task between two different tools, described below.
Interferometer sequencer This is JPL's responsibility. It understands about interferometer observing modes and calibration sequences. It talks to the telescope sequencers, one per telescope.
The interferometer sequencer will be implemented using NASA's ``Remote Agent'' (RA). This is a planner and sequencer which was originally developed to increase spacecraft autonomy, has recently flown on NASA's DS1 mission, and which will be used not only for the Keck Interferometer but also for NASA's future space-based interferometers.
Telescope sequencer This is Keck's responsibility. There will be an instance of this sequencer per telescope. It is responsible for a single telescope and its subsystems.
The telescope sequencer will be implemented using a Unix/KTL version of the EPICS sequencer. This is a natural extension of our use of the same tool for lower-level sequencing within the telescope and AO control systems. The telescope sequencer will also be useful for non-interferometry applications.
We do not see this keyword model being superceded as we adopt new technologies such as Java and CORBA. As we have described, we are working on a CORBA Keyword Service (COKE). It is quite possible that COKE, rather than KTL, will become our interface standard.
Finally, we have attempted to explain why it is that EPICS has been a success at Keck.
Conrad, A., & Lupton, W. 1992, in ASP Conf. Ser., Vol. 52, Astronomical Data Analysis Software and Systems II, ed. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes (San Francisco: ASP), 203
Dalesio, R., Kraimer, M., & Kozubal, A. 1991, in Proceedings of the International Conference on Accelerator and Large Experimental Physics Control Machines, KEK, Tsukuba, Japan, 278
Lewis, H., Lupton, W., Tsubota, K., Honey, A., & Quady, S. 1996, in Optical Telescopes of Today and Tomorrow, ed. A. Ardeberg (SPIE), 2871, 970
Lupton, W., & Conrad, A. 1992, in ASP Conf. Ser., Vol. 52, Astronomical Data Analysis Software and Systems II, ed. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes (San Francisco: ASP), 315
Lupton, W., Lewis, H., Tsubota, K., Honey, A., & Quady, S. 1996, in Optical Telescopes of Today and Tomorrow, ed. A. Ardeberg (SPIE), 2871, 977