| prev toc next |
DRACO (Datagrid for Research in Astrophysics and Coordination with the virtual Observatory) is the Grid project of the Italian astrophysical community and has the aim of providing the scientific community with a distributed multi-functional environment allowing the use of specialized (observational, computing, storage) Grid nodes.
DRACO provides the framework through which the Italian astrophysical community participates in the international Virtual Observatory effort. In particular, access to the following datasets will be provided: * the prototype TNG Long-Term Archive (LTA) and soon to the full-fledged service; * the ASDC archive of high-energy data, and in general the DIANA system; * a set of distributed radio data (EVN catalogue, Tirgo 2MASS archive, ...).
Contributions on the following topics are furthermore envisaged: * visualization and analysis allowing the possibility of 3D graphical representation; * "machine learning", which encompasses the use of neural networks, genetic algorithms, fuzzy-C sets; * high-energy data management tools, allowing easy integration of high-energy data with the rest of the VO.
Browsing and providing access to large local and distributed datasets is an important aspect of enabling Virtual Observatories. We present an example implementation of a meta-data tree in the Astrophysical Virtual Observatory prototype tool. This is a dynamically built "meta-data tree" containing information on image datasets, based on the IDHA data model. This meta-data representation of the GOODS dataset in the AVO demo, allows efficient data browsing and selection. This capability is being developed in the framework of the CDS Aladin image browser, and AVO prototypes.
Interoperability is one of the fundamental aspects for the Virtual Observatory and an essential requirement for the multi-wavelength astrophysical research. LAEFF is developing a multi-mission data server to provide an easy, efficient and uniform access to astronomical databases. The XMM, EURD and INES archives are already accessible from the system. Other archives like ISO will be accessible in the near future.
A prime goal in this project is the implementation of science-driven tools. In this sense, MMDSL provides an excellent framework to check the performance of utilities before its inclusion in the VO. This is a compulsory step in order to properly scale the tools to the huge flow of data expected from the VO network. At present, MMDSL allows for comparisons between the observed spectral energy distribution of a stellar object with synthetic models. Moreover, for some stellar objects, the user can automatically retrieve their physical parameters (absolute magnitude, effective temperature, surface gravity, metallicity) by making use of updated photometric calibrations.
In this paper the main functionalities and contents of the system are presented.
Although most astronomical archives store image metadata in fairly standard ways (typically in FITS and using conventions that include the WCS standard) the history of archiving spectral data from different instruments is much less successful. The recent FITS WCS proposal for wavelength transformations is only one of the steps needed to use archived spectra interoperably.
We present requirements for a standardized 1-dimensional spectral data model for use in the virtual observatory. Such a model would be a special case of a more general n-dimensional data model. We discuss the different kinds of spectra and the different observables used, as well as the appropriate instrumental calibrations. The problem of describing spectral data is closely related to the problems of defining bandpasses and photometric calibrations, as well as of abstract instrument descriptions such as spectral responses and efficiencies.
This work was supported by NASA under contract NAS8-39073.
The location and access methods of astronomical resources (catalogs, observation logs, and data archives) and associated computational services (e.g., data processing pipelines, source extraction services, theoretical simulations) in the Virtual Observatory will be determined by querying dynamic resource registries. These registries function as a sort of yellow-pages, providing descriptive information (metadata) about the resources in order to locate information and services in response to user queries. The metadata also needs to describe the provenance of the information, provide some indication of the data quality, quantity, and type, and guide users to information appropriate to their needs (i.e., research-oriented data archives vs. educational resources).
We describe the content and structure of the resource metadata being developed for the international VO. We have implemented several prototype registries based on these metadata definitions, and will share 'lessons learned' concerning metadata integrity and consistency. We also describe the challenges we expect in registry maintenance in an inherently distributed environment.
An important element of the Virtual Observatory is the development of a Data Model which permits the relationships between data values and physical properties to be unambiguously represented. We are developing a data model design that can support representation, analysis and display of data collected on different types of instruments.
Our design contains a high-level framework of general purpose components that provide a consistent protocol for accessing data which are recorded and organized in different ways. Software is, in addition, enabled to inspect the organization of the data, which will permit the creation of tailor-made, efficient processing tools. From this framework, we have focused on detailing a subset of components required to meet selected science objectives on spectral and image data.
As part of this design, we are including a unified framework for organizing standard data transformations. This framework covers mappings between units, between coordinate systems and between data values that are denoted using interchangeable properties such as frequency and wavelength.
Acknowledgements This material is based upon work supported by the National Science Foundation under Cooperative Agreement No. AST-0122449. This project is supported by the Chandra X-ray Center under NASA contract NAS8-39073
GAVO, the German Astrophysical Virtual Observatory, is publishing the ROSAT Bright Source Catalogue, for which it has developed a simple cone search (SCS), following recommendations and standards of the International Virtual Observatory Alliance (IVOA). In an extension of this work, we report on our experience in trying to execute simultaneous simple cone searches on a variety of other published astronomical catalogues. The individual search results are fed into a catalogue matcher developed by GAVO. The matcher attempts to perform a “fuzzy join” based on sky positions and their uncertainties. A central component in handling these kinds of catalogue data is a table utility that allows manipulating tabular data. We describe current features of the GAVO architecture that support such simultaneous queries, and outline some requirements for future versions.
As part of the NVO framework development initiative a prototype Astronomical Registry was designed for serving resource metadata across the internet to the world community. While this registry incorporates many VO standard Cone Search and Simple Image Access (SIA) services it provides mechanisms for publishing custom archive services with associated metadata as well. The registry is mirrored at two sites, Space Telescope Science Institute and Johns Hopkins University, and additionally harvests resources at Caltech and NCSA OAI repositories. Web services and forms were implemented for independent higher level application integration with the registry such as the NASA Data Inventory Service (DIS). These interface methods provide fundamental add, edit, remove features and include standard SQL query support. This registry is built with .NET technology integrated with MS SQL Server Database, IIS Web server, and C# product code.
The hierarchical tringular mesh (HTM) is a discrete foundation for describing location, size and shape on the celestial sphere. Indeces derived from HTM descriptors are used in a relational database for managing spatial information. Some of the new features available in the current implementation support operations such as the ability to perform searches based on arbitrary polygons, convex hulls of polygons or any region bounded by great or small circles. These functions are reached through a language that is implemented as an extension to MSQL Server 2000 relational database engine. The heart of the HTM tools can also be used through various interfaces in several langugages, like C++, C# and Java. An extensive XML specification for describing spatial structures to support spatial queries is also under development.
The Astronomical Data Query Language (ADQL) is a proposed standard query language for the interoperability of the International Virtual Observatory. The data servers in the International Virtual Observatory could be searched using an ADQL query. The servers would return VOTables as a result of the query.
The development of SkyQuery and JVOQL which perform queries on distributed databases revealed that a standard underlying query protocol is required to access individual data servers. In the Virtual Observatory Query Language (VOQL) architecture proposed in the International Virtual Observatory Alliance, ADQL is a simple underlying protocol which would allow data servers to join the International Virtual Observatory. Higher level language, Virtual Observatory Query Language (VOQL), will be built on top of ADQL and other services.
AQDL is the XML equivalent of a scaled down SQL grammar. The only operation permitted is a 'select'. A 'circle' clause has been added to SQL to facilitate the astronomical queries. The schema of ADQL is defined as XSD and the query described in ADQL will be passed in parse tree form as XML.
All data servers joining the International Virtual Observatory would implement a standard IVOA Query Interface Web Service. This Web Service can be used to find out the details about the content of the data servers, the tables contained, their columns etc..., and to perform the queries on that data servers and return VOTables as the results of the ADQL queries.
The Japanese Virtual Observatory (JVO) project has been commenced as an observatory's project since April 2002. The JVO aims to provide federated astronomical databases (especially SUBARU, Nobeyama and ALMA) and data analysis environment through the Grid technology. We defined a unified query language to access astronomical databases, the JVOQL, and constructed a prototype of the JVO to confirm if the JVOQL really works on a federated database. The prototype consists of 1) User Interface (WWW), 2) JVO Portal (JVO Controller) to accept users' requests described by JVOQL, and to parse them into individual query requests, 3) a registry which contains metadata on individual databases, 4) database servers. In this prototype we installed the Suprime-Cam data of the Subaru telescope and 2MASS data. The system consists of five servers (Solaris and Linux) under a GbE environment, and we used the Oracle and PostgreSQL as database management software. We adopted Globus Tool Kit 2 (GTK2) as an underlying middleware in this prototype to see its effectiveness in VO type application, especially for remote operations and file transfers, and UDDI for looking up services. We found that the system worked well, however, it took very long to authenticate in each Grid process. Thus we replaced the GTK2 with NMI (NSF Middleware Initiative) since NMI has a binary module to produce authentication keys. We also shortened the polling interval of the job-manager from 30 seconds to 3 seconds. As a result, the serious problem has been partially resolved -- the elapsed time for a query became about a half compared with the previous version. Since the authentication process is run every time when a globus command is issued, it would be necessary to implement a module to cache the authenticated key in the system for further acceleration. We plan to develop the second version of the prototype by utilizing the Grid Service of the OGSI. In the near future JVO will be connected with other VOs in the world through the International Virtual Observatory Alliance.
Astronomers commonly analyze astronomical data by imposing different views on the data, frequently viewing it in image form or as multi-dimensional plots. The ability to correlate the data in these views in order to see manifestations of patterns across views would be a powerful tool. The Mirage data visualization application offers this functionality. In order to increase the value of this tool to astronomers, we have added two features to Mirage, namely a module for viewing FITS images, and the ability to load VOTable data. During the process of adding the VOTable functionality to Mirage, we also developed a separate Java package, called the IVOAClient package, which can easily be integrated into any other Java application to provide the ability to load VOTable data via Cone search queries submitted to any Cone services published in a Registry, or by a direct SDSS CAS query.
We describe the usage of VO Enabled Mirage, the process of writing a data view module which can be incorporated into Mirage, the process of programming a FITS viewer in Java using JSky, and the process of integrating the IVOAClient package into any Java application.
The Astrophysical Virtual Observatory (AVO) prototype is a suite of tools that serves as a testbed for scientific and technical use cases (http://www.euro-vo.org/twiki/bin/view/Avo/SwgDownload). For instance, it already supports the Simple Image Access (SIA) protocol, hierarchical browsing based on the IDHA data model and fully exploits the potential of Unified Content Descriptors (UCD). Its components communicate in VOTable format - an XML dialect for Astronomy.
Each of the software components of the AVO prototype is developed and maintained by a different team in a different country. All modules work stand-alone, but can be integrated as shown at the demo booths of CDS and ESO/ST-ECF. The AVO prototype consists of a browser backed by image and catalogue servers, a web service for the identification of sources in FITS images and a utility for analyzing the spectral energy distribution.
Finding an easy way of sharing knowledge and experience in a geographically dispersed project team is not easy. The "TWiki" is a Web-based collaboration platform. It looks like a normal Internet web site, but everybody can change pages or add content by just using a browser. A revision control keeps track of changes.
Several Virtual Observatory projects use this type of web site to share info among project members and across projects. This is an experience report on the usage and maintenance of TWiki sites in Astronomy.
Even if at first it seems almost the opposite of how the Web and online communications "normally" work, it is intuitive and aspires to the Zen ideals known as Wabi-Sabi: "it finds beauty in the imperfect and ephemeral and constantly evolving".
Many of the problems that we are trying to address in solar physics require access to large amounts of data from as wide a spectral range as possible. Analysis can only start following the identification of events, features and phenomena and the location and retrieval of the required data. The rapidly increasing volumes of data, and the desire to share data with other communities has led to several projects intended to create virtual solar observatories to facilitate access.
There are three main initiatives: the European Grid of Solar Observations (EGSO), funded by the European Commission; the US Virtual Solar Observatory (US-VSO), funded by NASA; and the Sun Earth Connector (CoSEC), funded by NASA under the International Living with a Star (ILWS) program. EGSO is leading the effort to coordinate these activities and where possible share resources.
We will outline the three projects and describe how the the combined effort will result in a facility that will better match the needs of the community. Interaction with related communities are discussed, including similarities and differences with the IVOA and interoperability.
XML is the lingua franca in the Web (and Grid) services world and so will play a major role in the construction of the Virtual Observatory. Its great advantages are its flexibility, platform-independence, ease of transformation and the wide variety of existing software that can process it. An obvious disadvantage in its use as an astronomical data format is its verbosity; the number of bytes taken up writing the XML tags can easily outnumber those constituting the actual astronomical data. This becomes prohibitively inefficient when large amounts of data are stored in XML, and the developers of VOTable sought to circumvent this, by allowing for the use of binary data, either in the VOTable document itself or in an external file linked from it. The verbosity of XML in this regard is a problem in many other disciplines, and computer scientists are developing more generic solutions to that found in the VOTable specification. In this paper we describe several of these projects currently underway in Edinburgh, which focus on the compression and querying of XML, and a technology for representing the structure of a binary file in XML, enabling it to be read as if it were XML
UCDs (Unified Content Descriptors) are metadata that have been used in the VizieR catalogue service since they were first developed in the ESO/CDS data mining project. Because the catalogues currently described by UCDs cover many different domains, UCDs can be used to describe a large fraction of the astronomical concepts. The different Virtual Observatory (VO) projects, collaborating in the IVOA (International Virtual Observatory Alliance) have therefore decided to support UCDs, and to collaborate to define a new set of standard metadata. This new set of UCD will be more coherent and complete than the old one. It uses a new syntax, and a list of
The Data Inventory Service (DIS) is an early service developed as part of US National Virtual Observatory (NVO) project. This service makes astronomical data easier to find, browse and use, buiding on emerging VO standards. This simple web tool provides information on any region of sky in which a user is interested. A user simply enters the position. A large number of resources are searched on behalf of the user who can immediately view and compare images, find ground and space observations, look for Galactic or extragalactic objects and generally get a sense of what is known about the given piece of the sky.
While other systems have allowed for searching multiple astronomical resources -- the CDS VizieR service, NED and SIMBAD, SkyView, and the existing Astrobrowse tools -- users can see several important differences in the DIS. Remote resources are queried on behalf of the user and results are cached, so that users can easily browse the returned datasets. Image, observation and object data are all clearly organized into simple wavelength and object-type categories. Data analysis tools that can accept standard image (FITS) and table (VOTable) formats can launched immediately from the service. Currently OASIS and Aladin are supported. and others will be added.
The internal differences are just as important. All access to remote services is through the standard Cone and Simple Image Access protocols. Most important, the Data Inventory Service uses dynamic queries to a VO service registry to find services of interest. Any data provider can register a service, and this service will immediately be added to the DIS.
The combination of distributed data resources, multiple analysis envirnoments connected through standardized protocols and described by a dynamic registry makes the DIS a flexible and powerful guide for scientists using on-line astronomical resources.
Currently the DIS is available at http://heasarc.gsfc.nasa.gov/vo/data-inventory.html and is hosted at NASA's High Energy Astrophysics Science Archive Research Center. This work has been partially funded through NSF Cooperative Agreement AST0155449.
In this poster we present our proposal for a common data model, built up from simple building blocks which are
Started at CDS one year ago, the work around Web Services is in full exploitation phase. Several services are now available via SOAP: a name resolver for Simbad-VizieR, a GLU tag resolver, a UCD resolver, a UCD tag list, an Aladin image Access, etc. We have also created WSEA (Web Service Easy Access), a light client-side Java SOAP API which provides a very easy way to use SOAP in Java standalone applications as well as in applets. This API is not restricted to CDS Web Services and is user friendly since it requires only very basic skills in Web Services technologies. WSEA has not been created from scratch but is an overlay on kSOAP which is small, fast and free. Other works related to Web Services are ongoing at CDS and will be published on the Web Service portal http://cdsws.u-strasbg.fr/ (information, examples, help, etc.).
MySpace is a component of AstroGrid, the Virtual Observatory system being developed in the UK. It provides AstroGrid users with scratch space for storing temporary datasets, typically the results of queries submitted to large databases, and other transient files. The novel feature of the MySpace system is that the scratch space is geographically dispersed, with stores at the various sites hosting AstroGrid archives, but the user can access and navigate it seamlessly and easily, with the network details of the individual stores being hidden. MySpace is a fully integrated component of the AstroGrid system, written in Java and communicating via Web services. It is under active development and its current state and future plans are described. Functionality similar to that of MySpace seems likely to be a common requirement for a Virtual Observatory system, and the experience gained with MySpace should be applicable elsewhere.
In the Virtual Observatory (VO), a registry helps users locate resources,such as data and services, in a distributed environment. A general framework for VO registries is now under development within the International Virtual Observatory Alliance (IVOA) working group. We present a prototype of one component of this framework: the publishing registry. The publishing registry allows data providers to expose metadata descriptions of their resources to the VO environment. Searchable registries can harvest the metadata from many publishing registries and make them searchable by users. We have developed a prototype publishing registry that data providers can install at their sites to publish their resources. The descriptions are exposed using the Open Archive Initiative (OAI) Protocol for Metadata Harvesting. Automating the input of metadata into registries is critical when a provider wishes to describe many resources. We illustrate various strategies for such automation, both currently in use and planned for the future. We also describe how future versions of the registry can adapt automatically to evolving metadata schemas for describing resources.
The IVOA data access layer (DAL) working group is currently defining a Simple Spectral Access (SSA) protocol for accessing 1D spectra and SEDs. To help plan this effort a little survey was conducted. This poster summarizes the results from a data provider as well as data consumer point of view.
It turns out that a wide variety of spectroscopic data is readily available on-line. Data structures and qualitative aspects, however, vary substantially. A preliminary analysis shows that an SSA service will need to transform archived data into a common, well defined structure prior to delivery.
The Astrogrid registry stores metadata for every resource available to the Astrogrid virtual observatory. Resources can be defined as data archives, data storage areas, data modelling and astronomical algorithms, communities, and other registries to name some implemented examples. Astrogrid is developing its registry schema in cooperation with the IVOA registry workgroup. Each registry entry describes a resource in terms of identity, curation, content, and service access metadata. An Astrogrid user or component can conduct nested searches on the registry using XQuery, an implementation of the XML query language. The Astrogrid registry is currently stored as an XML document, but as development continues the registry will migrate to database storage in DB2 augmented by XML Extender.
Virtual Observatory related tools use a new standard for data transfer called VOTable format. The VOTable format is the variant of xml format that enables easy transfer of data over the web. We describe a streaming interface that can bridge the VOTable format with the FITS and ASCII formats, which are commonly used by astronomers, through a user friendly graphical interface. A streaming interface is important for efficient use of memory because of the large size of catalogues. The tools are developed in JAVA to provide a platform independent interface. We have also developed a stand-alone version that can be used on local machine to convert data stored in ASCII & FITS format. The streaming writer is successfully being used in voplot. We demonstrate usefulness of the tool by converting huge FITS & ASCII data into VOTable format on machines that have only limited memory.
Graphical interfaces are extremely useful tools in data mining as well as statistical analysis of huge datasets. We describe a graphical visualization tool, VOPlot, that can be interfaced with data in VOTable format to form a valuable aid in scientific discovery.
The VOPlot toolkit, developed by us, is available as either a stand-alone tool or as a web-based applet fully integrated with VizieR catalogue service. The VOPlot tool is very versatile, and can be used to perform a number of graphical tasks and simple statistical estimations.
VOPlot can produce output in postscript and VOTable formats and can be easily integrated with packages that require visualization support. It has full two-way communication with Aladin, and has been adopted in packages like LEDAS. We describe VOPlot and recent enhancements to it, and demonstrate its usefulness in specific science applications.
Eventually the IVO will contain many thousands of resources arranged in many different organizational structures and with data of varying age and precision. Often, the scientist will not want to be bothered with the details of each resource to be queried, but needs all of the relevant data to come back with error bars for intercomparison, weighted averaging, or extrapolation to the present time. Often, query will include a certain physical property, but a resource describes the data with a different term or else the property is derivable from several observables. Often the request will be complex, so that the requestor needs to focus on the astronomical objects and their properties and not on data center concepts also. Often the request will include complex mathematical and logical functions, so that the scientist needs a natural language layout that is easily comprehensible yet rich in expressions. For these user scenarios the IVO has decided to research into an Object Query Language that focuses on astronomical objects, their classes, their physical properties and their aggregations into clusters of various types. A near natural language rendition, an XML encapsulation, and a symbolic representation of this language will be presented. We will present solutions to challenges in an implementation that has many mundane details taken care of under the hood, so to speak.
| prev toc next |