| prev toc next |
This talk summarises the key features of the project EGEE (Enabling Grids for E-Science in Europe), for the FP6 call
The STScI HST Mission Data Management Systems, HST-DMS, encompass the Development, Integration & Test and Operational environments to process the science and engineering data from the telescope, ingest it into the archive and distribute raw and calibrated data to GO's, mirror sites and the astronomical community at large.
Over the past year, we have re-architected the HST-DMS to create a flexible, scalable and highly reliable solution centered around
The Atlasmaker project is using Grid technology, in combination with NVO interoperability, to create new knowledge resources in astronomy. The product is a multi-wavelength, scientifically trusted image atlas of the sky, made by federating many different surveys at different wavelengths. The atlas is implemented as virtual data - the data is intimately linked with the software that makes it. Computations take place on the US Teragrid, either as on-demand instantiations or backfill, and computed data stored in a distributed virtual file system based on SRB (Storage Resource Broker).
Atlasmaker uses Montage, a new and rigorous NASA code for mosaicking images. The code has been parallelized and scripted into the Atlasmaker package, and outrigger services built and deployed, for high-performance, wide-area computation on Teragrid. Connection to arbitrary image archives is through the NVO publishing protocol (SIAP). The protocol allows for multiple retrieval mechanisms: if the input data is on an SRB system, it can be retrieved that way, or else through HTTP. Code has also been built for the creation of atlases -- coherent collections of mosaicked images that lead directly to multi-wavelength imagery. We expect these atlases to be a new and powerful paradigm for knowledge extraction in astronomy, as well as a magnificent way to build educational resources. As the Teragrid matures, we expect to be computing large numbers of mosaics, each a reprocessing of a particular image survey to a particular page from an atlas. The results will be stored back in a single virtual file system managed by SRB, but physically located at multiple places on the Grid.
The Sloan Digital Sky Survey science database is approaching 1Tb in size. A small fraction of queries submitted to the DB server often take hours or days to run either because they require non-index scans of the largest tables or request very large result sets. However, such queries inevitably slow down the vast majority of queries that would normally execute in seconds or minutes. A job submission and tracking system has been developed with multiple queues. Execution time in queues are strictly limited.
Since the exponential growth in the size of online datsets has outstripped the increase in network bandwidths (and this discrepancy is only likely to get worse in the future), the transfer of very large result sets from queries over the network is another serious problem which must be addressed. Statistics suggested that some form of local storage would alleviate this problem as, in many cases, users do not need to download results immediately. They would prefer to store results locally to allow further cross matching and filtering. The system allows local space in the form of MYDB. Selects may be performed into MYDB. Tables in MYDB may be extracted to files (FITS,VOTABLE) and then transfered using FTP/HTTP to the users machine. Tables may also be shared with other users through a groups mechanism.
For the Virtual Observatory we need to extend MYDB to the notion of VOSPACE - a collection of local spaces (on the Grid, MYDB and other systems) where data may be replicated, registered, transfered, and shared inside a VO Grid-enabled environment.
| prev toc next |