Up: BoFs and Demos
Previous: SyNAPS: System for Networking Astronomical Publication Services
Table of Contents - Subject Index - Author Index - PS reprint -

Mehringer, D. M. & Plante, R. L. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, eds. N. Manset, C. Veillet, D. Crabtree (San Francisco: ASP), 703

DaRT: A Java Tool for Retrieving Archived Telescope Data

D. M. Mehringer
Astronomy Department, University of Illinois Urbana-Champaign

R. L. Plante
National Center for Supercomputing Applications, University of Illinois Urbana-Champaign


DaRT is a client-side Java graphical user interface used for simultaneous retrieval of multiple documents and/or archived scientific instrument data sets which can be accessed via the World Wide Web. DaRT is currently used worldwide by astronomers to retrieve data sets from the BIMA Data Archive. DaRT can be configured to support data set retrieval from multiple data archives; an archivist can configure DaRT to support her archive simply by creating a short text file. Optionally, DaRT can support more features of an archive if more properties are added to this configuration file and/or by supplying an archive-specific Java class.

1. Introduction

Observational data from the Berkeley-Illinois-Maryland Association (BIMA) Array are transferred in real time from the telescope at Hat Creek, California to the National Center for Supercomputing Applications (NCSA) on the University of Illinois campus. After a project has been archived at NCSA, the investigators use web-based forms to access and download these data from the BIMA Data Archive (Plante & Crutcher 1997).

The original method employed to download data from the archive (which has been in place for several years) was to invoke a client-side program written in C which would silently download the data using HTTP. This method had several drawbacks. For example, the program didn't inform the user on the status of the downloads and, when errors occurred, messages were usually not decipherable by the user.

We wanted to make the data retrieval process more user-friendly. Java seemed the best choice for developing a client-side graphical user interface because: (1) Java applications are easily ported to multiple platforms which is necessary for our diverse user base; (2) light-weight (Swing) components allow for the development of powerful graphical user interfaces (GUIs); and (3) it is simple to develop Java network applications. Our efforts have resulted in a client-side GUI tool called DaRT ( Data [or Document] Retrieval Tool). DaRT is not BIMA-specific; it can be configured to support any archive from which data can be retrieved via the Web. All that is required for this configuration is for the archivist to create a short text file and perform a small amount of server-side development. In addition, DaRT can be used to download any set of web documents, and there is a command line switch which will cause DaRT to download files non-interactively, thus allowing retrieval of web documents in batch mode which is useful for retrieving dynamic documents using e.g., crontab or at jobs on Unix systems. DMM uses DaRT in this mode to download airfares from http://travelocity.com/ daily :).

2. Using DaRT

DaRT can be launched from the command line or from a web browser. In the latter case, the user configures her browser to launch DaRT if the MIME type application/x-multiget is returned by a server. Although DaRT can be started with no command line arguments, usually at least the name of the multi-get file to load is provided:

% dart $<$filename$>$

This file simply contains the list of URLs (one per line) which point to the requested documents. When DaRT is launched from a web browser, the multi-get file is returned by the server immediately after the MIME type.

Figure 1 is an example of how the DaRT GUI appears after a multi-get file containing URLs to BIMA data sets has been loaded.

Figure 1: Example of the DaRT GUI after BIMA data set URLs have been loaded.

The GUI table contains information on the various data sets. Some of these metadata (e.g., for the BIMA archive, the project and the date) are embedded in the URLs; the archive-specific configuration file contains information on what metadata are embedded in the URLs. In addition, some of the metadata (the size and the location) are retrieved by DaRT from the server via a CGI script. To retrieve metadata in this manner, the archivist must write a archive-specific Java class (DaRT will function normally without such a class; the only function of this class is to provide methods to retrieve additional metadata). The checkboxes in the first column are editable by the user; the corresponding data set will be retrieved only if the box is checked. The Current File text field shows the name of the file which is currently loaded into the GUI; it can be edited by the user so that another file can be loaded (equivalently, the user can choose File $\rightarrow$ Open... to perform this function). The Destination Directory text field shows to which directory the retrieved data sets will be written and is also editable ( File $\rightarrow$ Destination performs the same function). The non-editable Status text field displays status messages.

To download data sets, the user simply clicks the Download button. Figure 2 shows how the GUI appears during file retrieval.

Figure 2: The appearance of the DaRT GUI during file retrieval.

DaRT provides information on the status of each data set being downloaded in the Status column of the table. During the phase where a file is being written to local disk, the corresponding progress bar is continuously updated to indicate how much of the file has been transferred. If an archive delivers tar files (such as the BIMA Data Archive), this can be indicated in the configuration file. The retrieved files can then be unpacked by DaRT (whether to do so is specified by the user). Retrievals can be aborted by clicking the Stop button.

Other features of DaRT include user-specified preferences, an error indicator (located just to the right of the Stop button), an internal error log, a detailed help facility, and a facility to send the error log or general comments to the author and/or the archivist.

3. Configuring DaRT for Other Archives

DaRT is easy to configure to work with other archives. The minimum that an archivist must do is to create a simple text file containing three properties of her archive. These properties are the archive's user support email address, the general format of the URLs which point to the archive's data sets, and a list of the titles for the DaRT table columns. The archivist has the option of specifying additional properties such as the local directory structure under which the retrieved files are written and if her archive delivers tar files.

The archivist must do a small amount of server-side development as well. She must write a CGI script that will return a list of URLs to the data sets which the user requests. Immediately before this list is sent to the client, the server should send the application/x-multiget MIME type which will launch DaRT from the user's web browser.

Finally, the archivist has the additional option of writing an archive-specific Java class to determine or retrieve additional information on the requested data sets that is not embedded in the URLs. We have written such a class for the BIMA Archive which fetches the sizes and locations of data sets. This class simply makes a URL connection to a CGI script which runs on our server.

The DaRT User and Archive Administrator Manuals contain detailed examples on how to configure and use DaRT. Both can be found at


Plante, R. L., & Crutcher, R. M. 1997, in SPIE Proc., Vol. 3112, Telescope Control Systems II, ed. H. Lewis (Bellingham: SPIE), 90

© Copyright 2000 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Up: BoFs and Demos
Previous: SyNAPS: System for Networking Astronomical Publication Services
Table of Contents - Subject Index - Author Index - PS reprint -