WMO Hydrological Observing System (WHOS)
- Xiaoxia Chen
- Igor Chernov (Unlicensed)
- Enrico Fucile
Introduction
In hydrology, the numerous and varied activities and applications have led to widespread heterogeneity in resources and procedures, which has hindered cooperation among the different actors and stakeholders. The goal of the WHOS is to fully implement the concepts of:
Water Data Catalogue by federating heterogeneous data providers and publishing a number of interfaces in support of advanced data discovery and access;
Water Data as a service by a broker mediating data formats and services in support of interoperability between data providers and users;
Enriching Water Data by appropriate hydrological attributes which make data discoverable according to a plethora of operational and scientific purposes; and
Community for Water Data and Tools by removing the digital divide in hydrology with the aim of building a community interacting with the offered architectural functionalities.
Project description
WHOS can be defined as a collection of components that work together to store, index, access and distribute hydrological information. WHOS is built around seven fundamental components:
Data
Format
Service
Mediator
Broker
Ontology
Client
WHOS defines a new paradigm in hydrology: a “datagram” reshaping specific data exchange using all these seven components.
The WHOS components enable the linkage of service providers with service consumers. In particular, the catalogue services facilitate the discovery of data sources, while the mediator and broker components facilitate the connection to data services for its request and access. When data providers are harvested by WHOS, data services are included in the registry. When data users search for available services of interest, WHOS improves data discovery and access with advanced interoperability, where every single component serves an important role in the designed architecture. Accordingly, data providers and users play the primary role in the exchange of information and in the development of a distributed knowledge base having large volumes of hydrological data in a federated architecture for e-monitoring.
WHOS Broker
A key component of the WHOS architecture is a WHOS Broker, realized by leveraging the Discovery, Access and Semantic Brokers technology.
WHOS Broker discovers and accesses the data from heterogeneous data providers. It then harmonizes the accessed data and re-publishes with various web services and data models required by different users' applications.
WHOS Broker works as a mediator between provider’s service and consumer’s application, suitably transforming the data on-the-fly and without storing any data (except for caching purposes).
WHOS Broker supports various service interfaces and Application Programming Interfaces (APIs) that data providers and hydrological application builders can already leverage (e.g. OGC SOS, OGC CSW, OGC WMS, ESRI Feature Service, CUAHSI WaterOneFlow, DAB API, USGS RDB, OAI-PMH/WIGOS, THREDDS). New API and service protocols are continuously added to support new data providers web services and user applications with the aim of improving their interoperability. This is due to the WHOS Broker being a modular and flexible framework designed considering the change/evolution of standards through time.
Each of the three WHOS brokers - Discovery, Access, and Semantic - has a set of components.
The Discovery Broker has three components: Accessors, Profilers, and a Registry.
Each accessor has a connector and a mapper for metadata. The connector's role is to query or harvest metadata published by the data provider by means of an exchange protocol. The mapper's role is to harmonize the data provider's metadata by means of ad hoc metadata mapping.
Each Discovery Broker profiler has a web-request transformer, a mapper for metadata, and a formatter for metadata sets. A web-request transformer's role is to query or harvest requests coming from a user application by means of an exchange protocol. The role of the mapper for metadata is to profile the previously harmonized metadata, according to a specific metadata model required by a user application. The role of the formatter for metadata sets is to create the response envelope with mapped records tailored to a user application.
To allow automatic search for services that provide requested metadata, the Discovery Broker has a Registry component containing details on data providers' web services. A Registry also allows WHOS to publish customized data views tailored to users needs.
The process of metadata mapping
A metadata source provides its original metadata which consist of the metadata elements defined by data providers.
The original metadata are mapped by the WHOS to create Harmonized Metadata, holding a description of the original metadata.
The Harmonized Metadata consist of metadata elements that were mapped by the broker as two types of metadata elements:
Core metadata elements that are defined by the ISO 19115 and WIS Metadata Standards.
Extended metadata elements defined by a particular community (e.g., WIGOS Metadata Standard for the hydrometeorological community). The Extended metadata elements are also enriched by Augmented metadata elements by means of WHOS Hydrological Ontology. Semantic metadata augmentation, for example, allows users that are searching for a specific variable in a specific language, to also automatically discover related terms for the variable, including in multiple languages.
The process of metadata harmonization makes possible a more harmonized data discovery allowing seamless querying of hydrological information from various heterogeneous data providers, and harmonization of query results to a common metadata model.
The publication of customized data views
Consider four data providers represented by four institutions sharing data with WHOS. Now consider some users who need information from specific stations operated by institution 1. WHOS packages data from only these needed stations and shares them for these users as View 1. If some users need all data shared by one specific institution, WHOS packages data from that specific institution and shares them as View 2. In a transboundary setting where users may need data from multiple stations in different countries represented by different institutions, WHOS packages these data and makes them available as View 3. To share all data available through WHOS, there is View 4. There are many other views that can be created depending on users needs (for example, if users desire only the precipitation variable).
The Access Broker has three components: Accessors, Profilers, and a Workflow orchestrator.
A subcomponent of the accessors is a data downloading component. Its role is to access data published by the data provider by means of an exchange protocol.
A Workflow orchestrator role is to build a process where data transformers components: (1) harmonize original data published by data providers and (2) convert the harmonized data to the data model required by the user application.
Each Access Broker profiler has a web-request transformer and a formatter for data. A web-request transformer's role is to respond to data access requests coming from users applications by means of exchange protocol. The role of the formatter for data is to create the response envelope with previously mapped records tailored to the user application.
The process of data mapping
A data source provides its original data which are encoded according to the data model chosen by data providers (e.g., WaterML 1.0).
Through a data mapping procedure, the original data are mapped by the Access broker to harmonized data encoded according to NetCDF standardized data model, which is the WHOS Broker default data model.
The process of data harmonization makes possible further data conversion from the data model chosen by data providers to the data model required by users' applications (e.g., from WaterML 1.0 to WaterML 2.0), thus ensuring the interoperability of data with the user applications.
WHOS Broker also uses the WHOS Hydrological Ontology for two different but related purposes:
(1) Improved data discovery through semantic query augmentation.
A user can make a query through WHOS for a specific concept expressed in a specific language.
For instance, a user would like to query for discharge data in French - “débit”:
By applying a normal discovery, only datasets that have been documented using French language (e.g., only coming from the French NHSs) will be found by the user.
Instead, by applying a semantic discovery, the WHOS-Broker would get all the synonyms for “débit” from the WHOS Hydrological Ontology, including “discharge” and “streamflow” (in English), “portata” (in Italian), “caudal” (in Spanish), etc. Here, a combined query is executed by the WHOS Broker, matching many more records from additional NHSs (e.g., Spanish, English and Italian ones).
(2) Harmonized presentation of results through semantic metadata augmentation.
Records coming from the NHSs can be enriched by WHOS Broker (at harvest time, or during result presentation), through the use of WHOS Hydrological Ontology.
E.g., a variable metadata element describing a specific published dataset coming from a French NHS may have been documented as “débit”.
WHOS Broker consults the WHOS Hydrological Ontology to search for concepts matching the textual information, in this case finding the “Discharge, stream” concept stemming from Ontology of the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI).
Further, it can also find an equivalent concept from the WMO Codes Registry - "River discharge”. These will be presented to the user, enabling a harmonized presentation of results.
Depending on user preferences, the user application may be displaying the concept definition coming from one ontology or another, as well as in the preferred language.
WHOS prototypes, portals, web services and supported tools
Two regional WHOS prototypes located in the La Plata Basin in South America (WHOS-Plata) and in the Arctic Region (WHOS-Arctic) have now reached their final stage of implementation.
The implementation of regional prototypes has demonstrated how the WHOS implementation benefits the participating countries. Some of the main benefits are as follows:
shared data are more discoverable, accessible, interoperable and reusable;
uniform and up-to-date discovery and access to shared data through a single interoperable web service, thus overcoming challenges associated with data discovery and access from heterogeneous data sources;
no additional development is needed on the data user side, for the reuse of shared data with various hydrological tools and applications;
as a result of a uniform discovery and access to shared data, national and regional forecasting and modelling systems can be more easily built;
making use of the WHOS Hydrological Ontology, WHOS expands and augments metadata in support of building trust in shared data and their more accurate use;
WHOS development, maintenance and support are provided by the WMO community.
WHOS data are organized and published in different WHOS Views. WHOS Views are the data subsets defined by a given area, institution, temporal extent, and/or variable, e.g. WHOS-Arctic.
Each WHOS View can be accessed by means of various tools and applications. For this, WHOS makes available web service endpoints implementing various standard communication protocols and metadata/data models that are supported by different tools.
Currently, WHOS also makes available three data portals allowing users to easily leverage common WHOS functionalities such as data discovery and data access, on the web by means of common web browsers.
WIS 2.0 Principles in the project
Principle 1: WIS 2.0 adopts Web technologies and leverages industry best practices and open standards
WHOS supports various service interfaces and Application Programming Interfaces (APIs) that data providers and user application builders can already leverage (e.g., OGC SOS, OGC CSW, OGC WMS, ESRI Feature Service, CUAHSI WaterOneFlow, DAB API, USGS RDB, OAI-PMH/WIGOS, THREDDS)
New API and service protocols are continuously added to support new data providers web services and user applications with the aim of improving its interoperability. This is due to the WHOS-Broker being a modular and flexible framework designed considering the change/evolution of standards through time
Principle 2: WIS 2.0 uses Uniform Resource Locators (URL) to identify resources
WHOS makes use of URLs to identify resources such as datasets, sites, concepts from ontologies, through several of its published service interfaces
WHOS supports publishing of metadata and data on Internet through various web services available for machine-to-machine access
WHOS supports harvesting of digital data and metadata resources from various data providers that publish their resources on Internet through web services
WHOS-broker provides various standard web service interfaces and APIs (e.g., OGC SOS, OGC CSW, OGC WMS, ESRI Feature Service, CUAHSI WaterOneFlow, DAB API, USGS RDB, OAI-PMH/WIGOS, THREDDS)
New APIs and service protocols are continuously added to support new applications
WHOS publishes data access interfaces supporting (1) subset functionality to retrieve only the data subsets that are of interest to the users (e.g., subset by temporal and spatial extent) and (2) down-sampling functionality to reduce the data size
WHOS already supports publication of web services implementing publish-subscribe message pattern such as OGC Publish/Subscribe interface standard
WHOS supports such requirements through its Data Cube component that acts as cache/store mechanism (component planned to be released by 2022)
Principle 8: WIS 2.0 will adopt direct data exchange between provider and consumer
WHOS-Broker works as a mediator between provider’s service and consumer’s application, suitably transforming the data on-the-fly and without storing any data (except for caching purposes)
Principle 9: WIS 2.0 will phase out the use of routing tables and bulletin headers
WHOS can support publication of web services implementing different message patterns (including Publish/Subscribe) and be configured to achieve desired information flow
WHOS already publishes different customized catalogue web service interfaces (e.g., OAI-PMH/WIGOS, OGC-CSW, CUAHSI HIS Central) that can be queried to obtain metadata records describing datasets and the services to access them
WHOS will also publish discovery metadata in the Global Information System Centers (GISCs) allowing WMO users to access a range of services on WHOS. By publishing in GISCs, the discovery metadata will also be available to the GEOSS community through the GEO Data Portal that includes all WMO discovery metadata
WHOS-Broker also publishes an OpenSearch 1.1 service which is supported by the main commercial web browsers (other indexes can be easily implemented as needed)
Questions & Answers
What are the functions of a broker?
The broker acts as a mediator and its main functions are making available data discovery and access from heterogeneous data providers to heterogeneous user applications (such as portals, models and apps). And doing this in a way that is evolvable (e.g., able to cope with changing standards and additional systems) with minimal efforts from data providers or users.
What programming language is used for WHOS and what are the technical components or layers?
The WHOS broker is mainly a Java software application, although other languages are also sometimes applied to connect to specific sources. The main components are accessors (one for each data provider type), profilers (one for each client application type) and the query distributor.
What is the license associated with the code?
The WHOS Broker is provided as a cloud-based Software-as-a-service (e.g., like the Google search engine) for better performances. It is based on code provided free of charge for education, research and non-commercial usage.
The code will be released as open-source with a CC-BY-NC kind of license to support local deployment and personalization (Community Edition). The release is planned for the end of 2021.
Is there a repository that can be accessed?
It's planned to publish a repository for the source code of the community edition during 2021.
Who is involved in the development and maintenance of the code?
The Institute of Atmospheric Pollution Research of the National Research Council of Italy (CNR-IIA) designs and develops the code, taking into account the feedback and requirements coming from data providers, users and WMO community.
On which platforms is deployed?
The deployment is based on Docker containers, to enhance interoperability and deployment on the most common cloud infrastructures.
Is there an updated cycle and change management?
The development follows an agile approach with adaptive planning, evolutionary development, and continual improvement.
The release of the Community Edition will include a bug tracking system to collect feedback from the community.
Metadata in WHOS and who is updating them?
The metadata is published by each data provider. WHOS broker acts only as a translator that maps the original metadata model to the WHOS internal one (based on ISO 19115 plus extensions). Then, additional metadata profiles (e.g., WIGOS) are available upon request to users. For each metadata profile, an initial mapping is configured by CNR-IIA and is iteratively refined with the feedback of the specific data provider or specific data user.
Project Team
Silvano Pecora, Italy (Lead)
Juan Bianchi, Argentina (Co-lead)
Andy Wood, United States of America
Artem Shevchenko, Russian Federation
Marcelo Jorge Medeiros, Brazil