2024-03-11 WIS2 Monitoring and ET-W2AT Meeting

1

Jeremy: At the stress test in Japan in May, all the GS operators implemented the metrics?

Rémy: currently, GB (Brazil, France), GC(Germany) already provide the metrics.

Jeremy: Max from Synoptic is now preparing the metrics (GC-USA & UK).

• topic hierarchy for alerts

  • monitor/a/wis2/# will only be open to WIS2 Nodes using authentication (a unique user and password)

notice board of global monitor, not to ask WIS2 users

Jeremy proposed to discuss the connection between alert and ticketing system at the stress test in Japan in May. However, Rémy emphasises that the objective of stress test in Japan focuses on checking if Global Services are operating a decent level and doing proper jobs.

(Kai): who raise the alert? Preference: automatic alerting using alert manager. to define what GM should alert. Other GS should not create such alerts.

(Rémy) GM raises the alert. GS may raise alerts but this is not a must. Criteria for GM to raise an alert.

Key question: what is the threshold of the metrics to raise an alert for GM? Kai proposed to go through all the metrics offline to define the

(Kai) If GM and GS raise a same alert, it is good to detect the same thing.

Involving China (CMA) in the GM

(Jeremy) In May, we may do the performance test. To get China on board to do the GM.

(Action) Xiaoxia to contact CMA colleagues for their GM plan.

(Maaike) what to communicate with the GM (China)

(Rémy) we will finalize 1) what are the potential alerts and the next step is 2) the GM to implement the alert. The meeting is to define the overall architecture to keep track of the alerts we want to raise.

2

Metrics hierarchy https://github.com/wmo-im/wis2-metric-hierarchy/tree/main/metric-hierarchy

• Global Discovery Catalogue Metrics https://github.com/wmo-im/wis2-metric-hierarchy/blob/main/metric-hierarchy/gdc.csv

Tom proposed to start with data schema with extensibility and then refine it.

To look at the metrics, the levels of the metrics, the threshold at which alerts are raised (triggered by GM).

(Max) reset period for the metrics, example as: total download errors

• (Jeremy) total number of metadata records, a rolling 24 hour? 00 UTC, set it as 0.

• (Kai) no requirement on reset for open metrics. no difference between a rest and continuous counting. For DWD, reset at 00UTC.

• (Rémy) metrics are published as raw data.

Levels of metrics/threshold for alerting

• Info/warning/broker?

• “wmo_wis2_gb_connected_flag”

  • severity level= info

(Max) to share the currently existing standard that we can follow https://docs.python.org/3/library/logging.html#levels

Level and duration

• Info

• warning (30 minutes)

• error (30 minutes)

• critical (an hour): a complete loss of connectivity

Alert manager rules

(Kai) Do we agree to use Alert Manager? To install the rules. For each listed metric, we create rules of severity levels, by grouping of type of global services instead of metrics. JMA may have some available. (Secretariat) no metrics received from JMA.

(action) GB: Rémy; GC: Kai and Max; GDC: Tom