SPIDR middleware for WDCs

[4] SPIDR is a type of Grid for environmental data. We define a Grid of environmental data sources to be a set of web services following the same contract for dynamic service registry, metadata and data request interfaces, as well as output metadata scheme and data model. This is inline with the general Grid approach towards virtualization of data, services, and interfaces [Zhao et al., 2006]. "Behind'' the web service we can store the environmental data in a file system as binary files or images, in a relational database as rows of observations, or as another web service possibly with a different service contract. Each storage method and structural organization of a dataset will require a specific implementation of our "virtual'' data source web service, but for the user of the Grid all of them will look like the same Common Data Model (CDM) apart from the specific environmental data contents, such as parameters, stations, grid-coordinates and observation time intervals.

[5] We have been developing this concept of a virtual environmental data source for some time already, starting with distributed web services and portals for the space physics, meteorological and simulation communities. There are two main reasons why are moving to Grid middleware and infrastructure, which is more complicated for development and support as compared to a "pure'' web-services approach implemented in the "standard'' Apache Axis or Microsoft.NET web-services container:

The availability of a scalable Virtual Organization proxy mechanism for individual users based on digital certificates used by Grid for secure access to multiple distributed resources compared to the local portal user-password authentication [Foster et al., 2001];
A data request and/or processing from large environmental archives may take quite a while even if we specifically optimize the database structure and the processing algorithms for this type of request, and a synchronous web-services call mechanism is not always appropriate to handle data requests which involve a long time delay [Barkstrom et al., 2003].

[6] The SPIDR system concept is similar to several emerging technologies for data access in the environmental sciences. Notable among these are Unidata's Thematic Real-time Environmental Distributed Data Service (THREDDS) [Domenico et al., 2006], the Environmental Scenario Generator (ESG) from the USAF [Kihn et al., 2004], and the Coordinated Data Analysis Web (CDAWeb) from NASA (http://cdaweb.gsfc.nasa.gov).