Observational data from the Berkeley-Illinois-Maryland Association (BIMA) Array are transferred in real time from the telescope at Hat Creek, California to the National Center for Supercomputing Applications (NCSA) on the University of Illinois campus. After a project has been archived at NCSA, the investigators use web-based forms to access and download these data from the BIMA Data Archive (Plante & Crutcher 1997).
The original method employed to download data from the archive (which has been in place for several years) was to invoke a client-side program written in C which would silently download the data using HTTP. This method had several drawbacks. For example, the program didn't inform the user on the status of the downloads and, when errors occurred, messages were usually not decipherable by the user.
We wanted to make the data retrieval process more user-friendly. Java seemed the best choice for developing a client-side graphical user interface because: (1) Java applications are easily ported to multiple platforms which is necessary for our diverse user base; (2) light-weight (Swing) components allow for the development of powerful graphical user interfaces (GUIs); and (3) it is simple to develop Java network applications. Our efforts have resulted in a client-side GUI tool called DaRT ( Data [or Document] Retrieval Tool). DaRT is not BIMA-specific; it can be configured to support any archive from which data can be retrieved via the Web. All that is required for this configuration is for the archivist to create a short text file and perform a small amount of server-side development. In addition, DaRT can be used to download any set of web documents, and there is a command line switch which will cause DaRT to download files non-interactively, thus allowing retrieval of web documents in batch mode which is useful for retrieving dynamic documents using e.g., crontab or at jobs on Unix systems. DMM uses DaRT in this mode to download airfares from http://travelocity.com/ daily :).
% dart filename
This file simply contains the list of URLs (one per line) which point to the requested documents. When DaRT is launched from a web browser, the multi-get file is returned by the server immediately after the MIME type.
Figure 1 is an example of how the DaRT GUI appears after a multi-get file containing URLs to BIMA data sets has been loaded.Open... to perform this function). The Destination Directory text field shows to which directory the retrieved data sets will be written and is also editable ( File Destination performs the same function). The non-editable Status text field displays status messages.
To download data sets, the user simply clicks the Download button. Figure 2 shows how the GUI appears during file retrieval.
Other features of DaRT include user-specified preferences, an error indicator (located just to the right of the Stop button), an internal error log, a detailed help facility, and a facility to send the error log or general comments to the author and/or the archivist.
DaRT is easy to configure to work with other archives. The minimum that an archivist must do is to create a simple text file containing three properties of her archive. These properties are the archive's user support email address, the general format of the URLs which point to the archive's data sets, and a list of the titles for the DaRT table columns. The archivist has the option of specifying additional properties such as the local directory structure under which the retrieved files are written and if her archive delivers tar files.
The archivist must do a small amount of server-side development as well. She must write a CGI script that will return a list of URLs to the data sets which the user requests. Immediately before this list is sent to the client, the server should send the application/x-multiget MIME type which will launch DaRT from the user's web browser.
Finally, the archivist has the additional option of writing an archive-specific Java class to determine or retrieve additional information on the requested data sets that is not embedded in the URLs. We have written such a class for the BIMA Archive which fetches the sizes and locations of data sets. This class simply makes a URL connection to a CGI script which runs on our server.
The DaRT User and Archive Administrator Manuals contain detailed
examples on how to configure and use DaRT. Both can be found at
Plante, R. L., & Crutcher, R. M. 1997, in SPIE Proc., Vol. 3112, Telescope Control Systems II, ed. H. Lewis (Bellingham: SPIE), 90