2022-01-24 ET-W2AT Meeting

1. Opening

Jeremy

Topology was discussed at last weekly meeting. Good progress like the use of MQTT1 and MQTT5 protocols.

• Rémy reminded that for a long time have been pushing the idea of "shared service approach" to how GISCs support WIS.

• He highlighted that having 15 GISCs all doing the same thing is not efficient

• Many GISCs spent effort on operating the infrastructure; not the important task of supporting centres in their AoR

2. Presentation by Remy

Remy

Remy presented WIS2.0 MQP Architecture and Data Exchange Topology :

• Core services: Global broker, Cache, Monitoring

  • a small number of “Global Brokers” (3 or 4)

  • a small number of cache

  • a small number of monitoring centers

• Topology: NC/DCPC

• MQP:

• Auto discovery of the services

• GISCs must collaborate with other GISCs, contribute to running core services ….

Open

3. Discussion

All

Thorsten

• MQP: all NCs to run the broker ?

  • (Remy) Transition is for 5/6 years. Some NCs run a broker gradually to get data passed between the GTS and WIS2.

  • (Rémy) WIS2 architecture is not only for GTS data, but covers all data and services

  • Peter WIS2node could provide this functionality

  • (Henning) What is the criteria for running a global broker if it’s limited to 3 or 4 and 15 want to be?

  • (Rémy) no limitation for the number of brokers. There is plenty of jobs to go around cache, gateway, metadata. We tried to run a vGISC in Europe, it didn’t work, but we can take one shared service each. In Europe we image that we can do that and a similar in Asia? It’s easy to share out functions, but it’s difficult to share the operational service

  • Jeremy, can private sector run a broker or cache or other core service? Maybe as a delegated authority from a GISC?

  • Rémy, yes a GISC can delegate to private company to do that

  • Rémy, If Meteo-France approached the Secretariat with an offer to run core business?

  • Enrico, It’s not Secretariat decision, It’s up to the Members

  • Rémy, the perfect situation is where private sector offers to run a service "out of the box", like CDN … in this case, there's no bespoke function, so we (WMO) would just consume their service]

    [but what about longevity - how long do we need a private sector agent to commit?]

  • (Thorsten) agree that supporting AoR is primary importance, but a list of services is needed for WIS2 architecture.

  • (Jeremy) shared service model agreed?

  • (Enrico) we need a list of the optional services that GISCs can run, and develop a structured model to allocate these optional services

  • (Remy) two operational core services at least. One in Europe, and an instance of all services in Asia.

  • Kai- pretty optimistic that the shared service will work. Less optimistic that every NC will run its own broker and data store. Perhaps a GISC may need to host these functions

  • Jeremy- lots of options about how GISCs might support these: a GISC may offer to host the MQP broker and data store / data server for an affiliated centre - or help the centre build and operate this service themselves, this is a better option than stating technical requirements like "GISC _must_ run the data store / data service / MQP broker". in many cases, an NC or DCPC will be more than capable of doing this themselves

  • Peter- cache and broker go well together, don't split them. We're probably adding complexity (in the topology), but the risk is low because countries can self-organise to fill gaps

  • Timo, data isn't in the message - it has to be got from somewhere, isn't this the cache?

    what about the storage [for third party data transfer] - maybe NCs can't provide this data reliably, so the GISC should offer this?

  • Rémy- the global broker republishing messages from the cache is just the same as the global broker republishing messages from an NC … a cache is just a bigger NC, but doesn't publish its own data … global broker doesn't push the data]

  • Peter- cache runs a broker and downloads data from NCs, it publishes messages about the data it's hosting; the global broker subscribes to messages from the cache-broker

  • Rémy- the cache is more like a "Data Collection" centre - in old WIS1 parlance]

  • Jeremy- this is similar to the point Kai made previously - I think a GISC should be responsible to ensure data from NCs are available according to an SLA, but not dictate a technical solution - it would be up to the GISC to determine the best way to achieve this, ranging from providing the service for the NC to use, or helping the NC build & operate their own service

  • Kai- a WIS Centre will need to talk to several GISCs - each offering different core services, so we'll need register of which GISC is offering which core service << relates to the "marketing portal", and could be implemented in WIS2 in a box

  • Enrico: have we got the direction or NC-GISC interaction right? Is it the GISC that subscribes to the NC or the other way around where GISCs publish info about their service offering (via the "marketing portal"?) and NC / DCPC decide which GISC they want to use for a given core-service offering?

    I think it's the GISC offering services to NCs - which is where the registry might come in

  • Remy- we don't mean GISC here - we mean "GISC running core-service x". This is about the "welcome package" for new centres, the "marketing portal" will provide info about WIS2 and also practical information about how to connect a node to the WIS2 ecosystem e.g. a configuration file. the need to split cache and broker: cache is only for weather data; and broker in WIS2.0 is for all data

  • Tom - each node can advertise the services they offer through discovery metadata - just like data. Re-use what we're already using.

  • Hassan -The curent status is 15 GISCs, choosing who can run core services may not be easy. Let’s start by asking for volunteers to run the global brokers

  • Jeremy- agree! Ask for volunteers to operate core services as first part.

  • Does the concept of AoR still fit?

  • Enrico: AoR is geographic, if GISC doesn’t support it’s affiliated centres then those centres never get any help. We need a mechanism that allows a centre to get help elsewhere if the principal GISC don’t provide it. Concept of AoR should be dynamic, and may be be outside of Manual.

  • Jeremy: mechanism is needed for the implementation of AoR

  • Timo: AoR is the main reason people get stuck with bad services and that observations don’t get out

  • Jeremy: but observations distribution will be based on core-service providers … and AoR is much more about helping Centres leverage the WIS2 ecosystem]

  • Timo- at the technical level, the AoR is an impediment [which the shared service approach should resolve]; but the "soft" stuff is a different concern].

  • Remy: Although AoR concept is outdated, we should now only focus on important pillars

  • Kai: some GISCs provide service to AoR. During transition period, AoR is a key element.

  • Enrico: Secretariat now have the regional officers to help during transition. and if AoR is about non-technical responsibilities, then it should be fine

  • Question: metadata catalogues; how is all the information shared? (to be discussed next time)

  • Question: Center running a cache

  • Timo: to make sure discovery metadata work is the core service of WIS2.0

  • Hassan: learn from the architecture of YouTube providing search, download, notification…

4 Wrap up

Remy

• The team agreed on a workable way forward for shared service approach, we need to dig a bit further, e.g. does an NC always run a broker, or the data flows between  global broker, NC and cache, but we're making progress.

• Once we've resolved this, we can present to GISC community to see what they think, and also, then take another look at the transition

Next week:

1. Finalise data and message flows between NC/DCPC and GISCs offering various core-services (global broker, cache, etc.)

2. finalise the list of core services