Input to WIS 2.0 from demonstration projects

Owned by Enrico Fucile
Last updated: Sept 29, 2021 by Jeremy Tandy (Unlicensed)
3 min read

 

Data Discovery

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161447937

  • Build out topic hierarchy with Project

    • Expressed in metadata

    • Queryable via OARec API

  • Build out a data identification scheme / granularity

  • Metadata provisioning via a basic catalogue or API provisioning

    • OGC API –Records

    • STAC Items

  • Key performance indicators

  • Documentation / cookbooks for onboarding, migration, publication, and use

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161284132

  • Proposals on the schema of service metadata, defining:

    • key elements: extending, updating

    • metadata format: JSON

  • Service catalogue (prototype) and demo services

  • Data reduction services, providing operational data

    • Data customization

    • Data visualization

    • WebGIS

Data Exchange

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161447953

  • The use of MQP for notifications fills a gap in standards; provides a necessary element for the replacement of GTS by WMO members

  • MQP notifications permit the standardization of large dataset exchanges currently done exclusively with bilateral agreements (NetCDFpilot)

  • MQP notifications could be adopted by wider groups later

  • Continued standardization and pilots recommended for now

Contributions, suggestions, technical specifications, standards used in the project that should be adopted as standard or recommended practises in WIS2

  • pubSub messages

  • CF-netCDF as meteorological data format

  • Web services for the provision of product data (like ERDDAP)

WMCP version1.3 vs. WMCP WIS2

  • pubSub messaging: single, separate attributes for

    • message broker URL

    • Exchange

    • Topic

(so that scripts could use metadata to get pub Sub values)

GTS2WIS JSON Specification

  • Integrity Value requires download of data

  • How is this value set by data provided using a Web Service?

Authentication/ Authorization concepts

  • How to register for pubSub messaging

  • Who should be allowed to publish messages

  • Access control to restricted data

  • User Federation

  • Ongoing discussion about ‘openness’ of the solution

    • Strong preference for Open Source, but this might limit potential to re-use existing capability

  • Platform agnostic (within reason) solution proposed.

  • Cloud-first principle with EWC considered as a host for centralised capability

    • NMHSs are free to follow national strategy including public cloud (e.g. AWS)

  • The key MQP technology expected to replace GTS has been used by a few NMHSs to date. Some countries (esp. LDCs ) may struggle with related incorporation into operational systems.

  • This project found some different nature between MQPs and direct cloud-storage downloading. (details later)

  • MQPs system is very real-time in nature while direct downloading is simple and the hurdles to migrate maybe low.

  • Big data (e.g. NWP, satellite) is likely to be suitable for simple downloading, as the data is to be produced on schedule.

  • As an alternative or transitional method, combined approaches may be an option for efficient migration from GTS to WIS2. 

Earth system domains

  • CF-NetCDF as meteorological and oceanographic near real-time data exchange format

  • ERDDAP as a recommended federated web service for the exchange/harvesting of data and metadata in near real-time

  • Relevant to the WIS 2.0 Data exchange of CF-NetCDFprofiles

  • WMO formats have little traction outside the NMHS, need to be pragmatic and support other standards (each for its specific purpose)

  • Improve the usage of CF-NetCDF and ACDD and actively engage in developments

    • It simplifies interaction with external communities

  • Evaluate Zarr as backend for CF-NetCDF

  • Include OPeNDAP as a mechanism for data exchange

    • Used internally in Copernicus services, ESGF etc

  • Connect with external communities on data management

    • e.g. on data management RDA, CODATA, WDS, on refinement of (ESIP)

    • Interact actively with semantic communities/activities (e.g. ESIP, ENVO, ...)

  • Primary challenge on data provider side is too complex standards (e.g. WMO Core Profile), understanding of semantics an how to use this in metadata

    • simplification is needed

  • Identify open source software/approaches that could benefit a larger community and actively contribute to this

    • e.g. pyCSW, THREDDS/HYRAX/pyDAP, MeteoIO, ...

  • The brokering architecture can be suitable for other domains other than hydrology that also have a diversity of data sharing solutions

  • It can be suitable for cross-domain data sharing that works as an interface between diverse systems and WIS2

  • The development is not restricted to metadata and data interoperability, WHOS also supports interoperability of tools for operational needs of NMHSs and other users

  • Existing mechanisms of user authentication should be used to strenghtenthe roadmap for free and unrestricted data sharing according to the new WMO Data Policy

  • The WHOS Broker is based on code provided free of charge for education, research and non-commercial usage. The code will be released as open-source with a CC-BY-NC kind of license to support local deployment and personalization (Community Edition)

LDCs and SIDSs

 

  • Important to foster the adoption of cloud services to provide turn-key solutions.

  • BUFR presents a barrier that is extremely difficult to address. More tools and training needed to support BUFR data exchange.

  • Difficulties are encountered in data exchange from the station to the acquisition centre at national level. Support to improve national exchange is required. 

  • Internet availability and costs can be a limitation in LDCs.

  • Observing station manufacturers provide closed solutions to access data from the stations. Open standards would be helpful at that level.