2022-05-23 ET-W2AT Meeting

1

Weiqing QU

Weiqing presented his slides on Transition to WIS 2.0-GTS Gateway

Open

• Questions:

  • What does Gateway do? (Expected output from GTS Gateway: observation, warning, forecast, chart)

  • what do you need to do to run a GTS Gateway? How should a GTS Gateway work?

    Objective: provide the bulletins [to WIS2] that are currently shared on GTS

  • Start by talking about what we exchange today

    Examples: Surface Observation (TAC, BUFR), Satellite Product (BUFR), Warning Product, Forecast Product (TAC, GRIB), [SigWx] chart (BUFR)

  • The task is to make sure this information continues to be shared with all Members

Enrico> the example of forecast data (TAF) is not supposed to be exchanged in GTS; and formally, ICAO say they're not interested in migration to WIS2.

Jeremy> key point: we need a separate plan for ICAO data]

Wiequing> continue with his slides and presented how the gateway should work:

Open

• The middle bit (GTS Bulletin Production) needs to be developed

• Some MSS already have the ability to batch up reports into bulletins; but this existing MSS isn't suitable

•  Batching reports into bulletins. Example: BoM bulletin SNAU15 contains 14 reports

• Receivers expect to receive data from 14 stations, based on the Vol C1, data could be delayed or missing etc.; how to manage this

• Complexity of the bulletin production system depends on:

  • how data is organized in the Global Cache

  • how closely we need to follow existing standards for how bulletins are created according to Vol C1

•  Organizing data in the Global Cache? using the topic structure: e.g. ……./weather/surface-observation/SYNOP_xxxxx.txt

2

Discussion

Jeremy > this isn't the physical organization of the cache; the topic hierarchy is used as a logical structure << noted

 Jeremy > in WIS2 we aim to have a single message / report per file] << noted.  Key problem is how the individual data objects are identified; e.g. the 'data-id', so these individual data objects can be mapped into bulletins as per Vol C1 with GTS header. If the data provider publishes data objects with multiple reports/messages, as per Vol C1, then all that's needed is putting a GTS header on it. In the case of BoM, the BUFR generation (and grouping) is upstream of the MSS.

 Enrico > the MSS has the capability to merge messages into bulletins; there's also simple software available for merging TAC and BUFR that could be used

Jeremy > can we use MSS to pick up files of individual reports deposited by WIS2node into a single bulletin, and then to publish that bulletin onto the GTS?

 Weiquing> Yes - but lots of MSS systems won't be able to do all of this without customization. If data providers can provide data objects with the same content defined in Vol C1 this will make the "GTS Bulletin Production" easier. We want to make it simple for data providers - so unlikely that we'll force people to batch up reports/messages. Current MSS can all do this

Jeremy> Organizing all this (e.g. the mapping from files to headers, amending routing tables etc.) will require a small team. it's a lot of effort.

 Extra components needed:

1. subscribing to Global Brokers, retrieving data from Global Cache

2. mapping data objects (according to 'data-id') to GTS headers

3. batching up multiple data objects into bulletins

 Then can use existing MSS software to publish onto GTS.

 Only need a couple of global centers to operate such a GTS Gateway.

 What's the alternative?

Can we get Centers to publish to a GTS topic structure, batching up files as per Vol C1. Essentially, requiring that data producers publish bulletins the same as today (in addition to publishing 'natively' to WIS2) - but pushing those data-objects to a local broker/http-server to be picked up by the WIS2 system

Remy > the proposal covers packaging of data; what about the point-to-point connections required for distributing data on GTS? Could avoid the RTH problem; and directly push bulletins to all the subscribing countries. Potentially requiring direct connections to 193 countries; albeit public internet could be used. Maintaining these routing tables will be on-going effort. Could start by pushing to RTH and then take over RTH role one-by-one as they decide to retire; we could "inject" information at a few points into the GTS/IMTN, which in turn push data to RTHs. Using RTHs would be a good method to keep the human effort manageable, avoiding the need to establish direct connections with countries in lots regions, using lots of languages etc. 

Remy > am not expecting "any of us" to turn off their MSS before 2030 because they are main Hubs on the GTS/IMTN. IMTN centers will have the largest routing tables"; they should be able to best cope with data from many locations Exeter, Toulouse, Tokyo, Washington, Offenbach, Melbourne etc. There may need to be some changes - but those would be limited to a small number of centers. Without this, we'll be many years getting connections to 193 countries established before we can get started on WIS2 migration 

Hassan > agrees that we can leverage the existing GTS connections to individual countries and leverage existing RTH connections, but what we need is the mapping between WIS2 and GTS

 Remy > 3 challenges:

1. re-injecting data into the GTS (e.g. IMTN centers) so that data will flow as it does now

2. one-to-one mapping between GTS header and a data-object (e.g. GRIB file); mapping between the two

3. repacking data such as SYNOPs into GTS bulletins as described in Vol C1 (Vol C1 is out of date)

Weiqing > for data producers, you could require data publishers to publish bulletins as they are today, or you could allow data publishers some flexibility; which would mean that the Gateway needs to do this, which adds complexity

 Enrico > the packaging can be done; there's software to do this, but you need a store and forward mechanism, and implement logic to deal with late / missing data etc.; this latter bit is the hard part. Also need to discuss who would run this gateway until the end of the transition

 Jeremy > how onerous would it be for data publishers to continue to package reports/messages into bulletins?

 Weiqing > they could do it - it's what they do today, but then that stops them decommissioning existing systems [so what's the motivation to migrate?]

 Remy > distinguish between the "packaging feature" we may ask people to maintain after migration to WIS2, and the full-blown GTS connection with point-to-point links. You'd need to keep packing reports/messages into files, but you can decommission the telecoms parts. The question is whether the "GTS bulletin factory" is feasible

Weiqing > this would be non-trivial

Remy > what do others think? Kai? Kentaro?

3

Kai Wirt

Kai > appreciates the great presentation from Weiqing. my few slides are a summary of the detailed presentation provided by Weiqing

Open

• GTS to WIS2:

  • Necessary and Easy

  • Send GTS Bulletins to WIS2 as any other file

  • Topic derived from TTAAii

  • WIS2 Nodes can subscribe to GTS Topics

• Problem: All Bulletins need to be available

• Solution: Needs to be done by one of the main RTHs. Possible updates to routing necessary

• WIS2 to GTS:

• Idea 1

  • Nodes migrated to WIS2 keep sending out data on GTS

  • No change needed

• Problem: Centers are required to run two systems (at least until 2030)

• Idea 2:

  • Use WIS22GTS Idea 2 until enough nodes are migrated

  • Then announce a GTS shutdown date

  • After that date WIS2 nodes can stop sending out data on GTS

Reason: It‘s easier to run a deprecated system without changes            than to install something new and adjust the routing. This work is better invested in migrating to WIS2

Take a 2-step approach?

Idea #2 means that we don't need to invest in big gateways

4

Discussion

Kai > not thought about the "halfway house" of centers keeping the packaging, but using WIS2 protocols

Remy > question is how expensive is the "big gateway" system

Kai > it will be a new operational system; this will be expensive

Remy > in DWD and MF the MSS is packaging the bulletins, so we're just talking about doing this on a larger scale?

Weiqing > doing this on a global basis is a huge job

Kai > when a center moves to WIS2, their data would be missing on the GTS. So we're trying to take away pressure on the migration. It's not creating bulletins for the whole world - only those that have migrated to WIS2

 Remy > Kai's proposal is good. It takes pressure off the "brave" Centers. It would be good to reward the brave (early adopter) Centers by allowing them to turn off systems. Not forcing them to run two systems in parallel

Kai > yet, still the "lazy" (follower) Centers may not migrate at all because they still get their data

Remy > thinking from a political perspective. Our objective is to get the transition as short as possible

Jeremy > can we engage RTHs (or GISCs) to do the packaging?

Remy > we need two things:

1. software to package up messages/reports in bulletins and implement the logic to deal with late/missing data

2. people to operate this software: NC/DCPC? RTH? GISC? Gateway?

How much effort to build this software? 1-month, 1-year, 10-years?

 Weiqing > difficulty is knowing how much flexibility people will have in publishing data objects

 Kai > between 1-month and 1-year. But the issue is that we're already busy supporting the WIS2 migration ourselves - not building new software for the packaging.

 Kenji > Number of bulletins exchanged over the GTS is around 500k. If bulletins are decomposed we're talking about 10M messages. This is why the granularity doesn't seem good to me.

• New data will be ingested to WIS2 only, this will help migration. Kai's idea #2 is a good proposal.

• The Malawi project uploads data to WIS2 and GTS - can we replicate this?

• Small NMHS will(?) use their systems for 10-years or more; they may be challenged to do the migration before the 2030 deadline

 Remy > Responding to Kenji's comments on packing

The one-message-per-file is something we've been pushing for some time in WIS2, I don't think it's an issue looking at the system performance. You mention the user perspective; they might want to receive messages already packed. The foundation would be one-message-per-file, but some countries may offer a repackaging service as a "Supplemental" service; creating new products. At the foundation - WIS2 is one message per file

 Jeremy > Next week we take two more views at this problem

1. Peter Silva

2. WMO Secretariat (Enrico / Hassan)