Input to WIS 2.0 from demonstration projects

Owned by Enrico Fucile
Last updated: Sept 29, 2021 by Jeremy Tandy (Unlicensed)
3 min read

 

Data Discovery

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161447937

  • Build out topic hierarchy with Project

    • Expressed in metadata

    • Queryable via OARec API

  • Build out a data identification scheme / granularity

  • Metadata provisioning via a basic catalogue or API provisioning

    • OGC API –Records

    • STAC Items

  • Key performance indicators

  • Documentation / cookbooks for onboarding, migration, publication, and use

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161284132

  • Proposals on the schema of service metadata, defining:

    • key elements: extending, updating

    • metadata format: JSON

  • Service catalogue (prototype) and demo services

  • Data reduction services, providing operational data

    • Data customization

    • Data visualization

    • WebGIS

Data Exchange

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161447953

  • The use of MQP for notifications fills a gap in standards; provides a necessary element for the replacement of GTS by WMO members

  • MQP notifications permit the standardization of large dataset exchanges currently done exclusively with bilateral agreements (NetCDFpilot)

  • MQP notifications could be adopted by wider groups later

  • Continued standardization and pilots recommended for now

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/155090983

Contributions, suggestions, technical specifications, standards used in the project that should be adopted as standard or recommended practises in WIS2

  • pubSub messages

  • CF-netCDF as meteorological data format

  • Web services for the provision of product data (like ERDDAP)

WMCP version1.3 vs. WMCP WIS2

  • pubSub messaging: single, separate attributes for

    • message broker URL

    • Exchange

    • Topic

(so that scripts could use metadata to get pub Sub values)

GTS2WIS JSON Specification

  • Integrity Value requires download of data

  • How is this value set by data provided using a Web Service?

Authentication/ Authorization concepts

  • How to register for pubSub messaging

  • Who should be allowed to publish messages

  • Access control to restricted data

  • User Federation

https://wmo-teams.atlassian.net/wiki/spaces/WIS2/pages/161284113

  • Ongoing discussion about ‘openness’ of the solution

    • Strong preference for Open Source, but this might limit potential to re-use existing capability

  • Platform agnostic (within reason) solution proposed.

  • Cloud-first principle with EWC considered as a host for centralised capability

    • NMHSs are free to follow national strategy including public cloud (e.g. AWS)

GISC Tokyo cloud project

  • The key MQP technology expected to replace GTS has been used by a few NMHSs to date. Some countries (esp. LDCs ) may struggle with related incorporation into operational systems.

  • This project found some different nature between MQPs and direct cloud-storage downloading. (details later)

  • MQPs system is very real-time in nature while direct downloading is simple and the hurdles to migrate maybe low.

  • Big data (e.g. NWP, satellite) is likely to be suitable for simple downloading, as the data is to be produced on schedule.

  • As an alternative or transitional method, combined approaches may be an option for efficient migration from GTS to WIS2. 

Earth system domains

Open Access to the GTS (Open-GTS)

  • CF-NetCDF as meteorological and oceanographic near real-time data exchange format

  • ERDDAP as a recommended federated web service for the exchange/harvesting of data and metadata in near real-time

  • Relevant to the WIS 2.0 Data exchange of CF-NetCDFprofiles

Global Cryosphere Watch

  • WMO formats have little traction outside the NMHS, need to be pragmatic and support other standards (each for its specific purpose)

  • Improve the usage of CF-NetCDF and ACDD and actively engage in developments

    • It simplifies interaction with external communities

  • Evaluate Zarr as backend for CF-NetCDF

  • Include OPeNDAP as a mechanism for data exchange

    • Used internally in Copernicus services, ESGF etc

  • Connect with external communities on data management

    • e.g. on data management RDA, CODATA, WDS, on refinement of http://schema.org (ESIP)

    • Interact actively with semantic communities/activities (e.g. ESIP, ENVO, ...)

  • Primary challenge on data provider side is too complex standards (e.g. WMO Core Profile), understanding of semantics an how to use this in metadata

    • simplification is needed

  • Identify open source software/approaches that could benefit a larger community and actively contribute to this

    • e.g. pyCSW, THREDDS/HYRAX/pyDAP, MeteoIO, ...

WMO Hydrological Observing System (WHOS)

  • The brokering architecture can be suitable for other domains other than hydrology that also have a diversity of data sharing solutions

  • It can be suitable for cross-domain data sharing that works as an interface between diverse systems and WIS2

  • The development is not restricted to metadata and data interoperability, WHOS also supports interoperability of tools for operational needs of NMHSs and other users

  • Existing mechanisms of user authentication should be used to strenghtenthe roadmap for free and unrestricted data sharing according to the new WMO Data Policy

  • The WHOS Broker is based on code provided free of charge for education, research and non-commercial usage. The code will be released as open-source with a CC-BY-NC kind of license to support local deployment and personalization (Community Edition)

LDCs and SIDSs

Interconnection of GISC Casablanca to the National Meteorological Centres within its area of responsibility

 

WIS 2.0 Malawi Automatic Weather Stations data exchange

  • Important to foster the adoption of cloud services to provide turn-key solutions.

  • BUFR presents a barrier that is extremely difficult to address. More tools and training needed to support BUFR data exchange.

  • Difficulties are encountered in data exchange from the station to the acquisition centre at national level. Support to improve national exchange is required. 

  • Internet availability and costs can be a limitation in LDCs.

  • Observing station manufacturers provide closed solutions to access data from the stations. Open standards would be helpful at that level.