Page Comparison

1

Managing provision of high-volume Core data sets 

High-volume core data sets mechanism

(Jeremy) Summary:

1. We have technical mechanisms to (a) limit the cache size, (b) inform data providers when data is not cached, and (c) allow data providers to assert what Core data shouldn’t be cached.

2. We need the “big” data providers to be “reasonable” and share the burden of distributing their data. The key question from my perspective is what being “reasonable” means. We can iterate towards this by:

   1. Indicating in the Guide the daily data volume that Global Cache (GC) operators are expected to cache (for UK/USA this is 100GB)

   2. Asking [“big”] data providers to prioritise which of their Core data should be distributed via the Global Cache and which they will serve directly themselves – perhaps based on their understanding of which data products are most widely used but ultimately their decision; with the aim of limiting the GC volume to the agreed daily amount (e.g., 100GB) We would expect some on-going discussion between big data providers and GC operators to get this right.

3. should be written in the Guide somewhere. Suggest this is in the section for DCPCs?

(Remy) In Notification Message, data producers in property (True or false) to cache or not. TB data should be cached. Cache operators, for those risks posing impact the system, should them cache the data? Proposal is to opt more agile approach, that is recommended data (not need to cache) to use the notification message to inform the data is not cached.

(Tom) Data not cached, NM still still points to the originating nodes.

(Simon) from data provider’s perspective, this mechanism (recommended data not being cached) will make the wis2 less attractive. This will make satellite data similar to the current GTS operation, very little satellite available.

(Jeremy) Current GTS, no data exchanged, no visible at all. But WIS2, recommended data will be visible from GB in Notification Messages. Achievable levels, big data producer, service operators to have a dialogue about the mechanism .

(Remy) data provider, data users to collaborate to decide what data is useful. It is important to decouple the roles of the GC and data providers.

(Simon) Community like CSMS can work with the end users to decide what other data can be shared in WIS2.

---

(Xiaoxia) actions: to draft name convention for the centre ids.

(Tom) delineate the Global services, one for NC and one for DCPC

1. Global Cache operators can cache more data than the current volume on GTS. Communities like CGMS, can work together with end users to determine what data to be cached in Global Cache.

Topic Hierarchy (TH) of merging country and centre-id timeline

• (Rémy)

  • As discussed in the ET-W2AT in Washington (13-15 November), there should be a window for all the WIS2 nodes to be able to migrate from the current TH to the new one.

  • Three Global Cache Operators (DWD, JMA and SYNOPTIC) confirmed that they are able to handle both THs.

  • Proposal: to send an email to all WIS2 node operators in Pilot Phase informing this migration to the new TH starts from 4th December and window is set to be 3 months.

    • For WIS2 node operators using wis2box (new release set to be on 12 December),

    • Action: Secretariat (Xiaoxia) to draft a WIS2 node naming convention

WIS2 node naming convention

• (Tom) For WIS2 node naming, it is good to delineate the centre ID for Global Services rather than using NC or DCPC.

• (Jeremy)

  • for overseas/remote territories, using TLD, such as Saint Helena (SH), the good practice is the center id

• • being “sh-metoffice”

Another case: Two Met Office sites in UK, Exeter and Aberdeen, using the same center id “metoffice”. There is a need to distinguish them further in the center id.

• • for

• • particular cases, such as USA and UK

• • jointly providing Global Service, the name?

• • (Rémy) Not really matter as GC name is used internally.

• (Rémy)

  • Need to consider the differences among France, overseas France, and

• • “Half” France

• (Simon)

• confirmed that EUMESAT will start using the new

• TH from 4th December

2

Checkpoint on writing sections of Guide and Transition document - any outstanding questions

(Tom) Requests a link for the reference to responsibility matrix for each chapter in Guide to WIS.

Images in the publication

• (Tom) suggested placing raw and editable images in the GitHub Guide Repository under the “image” folder https://github.com/wmo-im/wis2-guide/tree/main/guide/images

• (Rémy) asked for guidance for the use of SVG format

• (Enrico) Secretariat will handle the images to ensure a consistent style for publication. For technical specifications, including WCMP2, Notification Message and Topic Hierarchy, they could be added as separate appendices to Manual on WIS (WMO-No.1060, Vol-II).

3

Discuss outstanding issues pertinent to upcoming Satellite and Cryosphere workshops

Satellite workshop

(Enrico) Will present some slides at the workshop. One question from this community is: Does data should be sent to WIS2 or not? According to Manual on WIGOS: data shall be exchanged on WIS. It is an obligation, at least for Core Data.

(Simon) EUMETSAT delegation at the workshop is to ensure understanding of data policy-what core data and recommended data are. It is more on legal part, not technical discussions.

Cryosphere workshop

(Tom) For Cryosphere, Portal, federation, ESGF, Oystein is not in favor of using MQTT protocol but for WIS1 metadata harvesting. A meeting to discuss how we federate with other communities is needed.

(Enrico) There are other components, e.g. huge data (snow data), are sending data to ECMWF, which could be easily to transmit to WIS2.

4

AoB

(Simon) Shared some takeaways from the MQTT training course last week.
- Decoupling the message from data surprise the MQTT community.

- MQTT 5 feature: shared subscription, may have potential applications.

-(Rémy): (1) Benefits: redundancy + scalability.

(2) Challenge: Additional Topic Hierarchy (TH) is needed above the current TH and needs authentication.

(Simon) transport community was surprise to learn that we did the JSON standardization.