Input to WIS 2.0 from demonstration projects
- Enrico Fucile
- Jeremy Tandy (Unlicensed)
Data Discovery
Discovery Metadata exchange and harvesting
Build out topic hierarchy with Project
Expressed in metadata
Queryable via OARec API
Build out a data identification scheme / granularity
Metadata provisioning via a basic catalogue or API provisioning
OGC API –Records
STAC Items
Key performance indicators
Documentation / cookbooks for onboarding, migration, publication, and use
GISC Beijing Web services catalogue projects
Proposals on the schema of service metadata, defining:
key elements: extending, updating
metadata format: JSON
Service catalogue (prototype) and demo services
Data reduction services, providing operational data
Data customization
Data visualization
WebGIS
Data Exchange
Exploring the use of message queuing protocols for GTS data exchange
The use of MQP for notifications fills a gap in standards; provides a necessary element for the replacement of GTS by WMO members
MQP notifications permit the standardization of large dataset exchanges currently done exclusively with bilateral agreements (NetCDFpilot)
MQP notifications could be adopted by wider groups later
Continued standardization and pilots recommended for now
Experimental WIS 2.0 data exchange for data in WMO CF-NetCDF profiles
Contributions, suggestions, technical specifications, standards used in the project that should be adopted as standard or recommended practises in WIS2
pubSub messages
CF-netCDF as meteorological data format
Web services for the provision of product data (like ERDDAP)
WMCP version1.3 vs. WMCP WIS2
pubSub messaging: single, separate attributes for
message broker URL
Exchange
Topic
(so that scripts could use metadata to get pub Sub values)
GTS2WIS JSON Specification
Integrity Value requires download of data
How is this value set by data provided using a Web Service?
Authentication/ Authorization concepts
How to register for pubSub messaging
Who should be allowed to publish messages
Access control to restricted data
User Federation
EUMETNET Supplementary Observations Data-Hub (E-SOH)
Ongoing discussion about ‘openness’ of the solution
Strong preference for Open Source, but this might limit potential to re-use existing capability
Platform agnostic (within reason) solution proposed.
Cloud-first principle with EWC considered as a host for centralised capability
NMHSs are free to follow national strategy including public cloud (e.g. AWS)
The key MQP technology expected to replace GTS has been used by a few NMHSs to date. Some countries (esp. LDCs ) may struggle with related incorporation into operational systems.
This project found some different nature between MQPs and direct cloud-storage downloading. (details later)
MQPs system is very real-time in nature while direct downloading is simple and the hurdles to migrate maybe low.
Big data (e.g. NWP, satellite) is likely to be suitable for simple downloading, as the data is to be produced on schedule.
As an alternative or transitional method, combined approaches may be an option for efficient migration from GTS to WIS2.
Earth system domains
Open Access to the GTS (Open-GTS)
CF-NetCDF as meteorological and oceanographic near real-time data exchange format
ERDDAP as a recommended federated web service for the exchange/harvesting of data and metadata in near real-time
Relevant to the WIS 2.0 Data exchange of CF-NetCDFprofiles
WMO formats have little traction outside the NMHS, need to be pragmatic and support other standards (each for its specific purpose)
Improve the usage of CF-NetCDF and ACDD and actively engage in developments
It simplifies interaction with external communities
Evaluate Zarr as backend for CF-NetCDF
Include OPeNDAP as a mechanism for data exchange
Used internally in Copernicus services, ESGF etc
Connect with external communities on data management
e.g. on data management RDA, CODATA, WDS, on refinement of http://schema.org (ESIP)
Interact actively with semantic communities/activities (e.g. ESIP, ENVO, ...)
Primary challenge on data provider side is too complex standards (e.g. WMO Core Profile), understanding of semantics an how to use this in metadata
simplification is needed
Identify open source software/approaches that could benefit a larger community and actively contribute to this
e.g. pyCSW, THREDDS/HYRAX/pyDAP, MeteoIO, ...
WMO Hydrological Observing System (WHOS)
The brokering architecture can be suitable for other domains other than hydrology that also have a diversity of data sharing solutions
It can be suitable for cross-domain data sharing that works as an interface between diverse systems and WIS2
The development is not restricted to metadata and data interoperability, WHOS also supports interoperability of tools for operational needs of NMHSs and other users
Existing mechanisms of user authentication should be used to strenghtenthe roadmap for free and unrestricted data sharing according to the new WMO Data Policy
The WHOS Broker is based on code provided free of charge for education, research and non-commercial usage. The code will be released as open-source with a CC-BY-NC kind of license to support local deployment and personalization (Community Edition)
LDCs and SIDSs
WIS 2.0 Malawi Automatic Weather Stations data exchange
Important to foster the adoption of cloud services to provide turn-key solutions.
BUFR presents a barrier that is extremely difficult to address. More tools and training needed to support BUFR data exchange.
Difficulties are encountered in data exchange from the station to the acquisition centre at national level. Support to improve national exchange is required.
Internet availability and costs can be a limitation in LDCs.
Observing station manufacturers provide closed solutions to access data from the stations. Open standards would be helpful at that level.