3rd Joint meeting between TT-WISMon and TT-WDQMS-2022-Feb-15

2

Review discussion and high-level architecture

Timo

Timo makes an introduction about the previous discussions and agreement. The presentation can be found here.

• Reviewed architecture:

  • Central component and distributed Sensor Centers

  •   Sensor centers extract metadata and report to central component

3

Finalize “loosely” vs “tightly” coupled approach

ALL

• The team agreed loosely coupled approach using MQP and hourly reporting

• File-exchange format (which metadata fields)

  • Agree to use JSON, since it allows to easily add fields and different fields by file-type

  • Need to agree mandatory fields by file-type

  • Need to agree how to distinguish WIS2 from GTS traffic

    • Approach 1: separate files for WIS2 and GTS

    • Approach 2: indicate “source” using a metadata field

• How the sensor centers connected to central component

  • Kai: both are ok. To start with the monitoring, the frequency will be hourly. For sensor centers, to generate the reports when the data is received; for monitoring centers, they can aggregate the reports they receive and decide what to display.

  • Cristiano: ECWMF is introducing the MAP-AMQP protocols. In the short time, for monitoring, preference is “loosely”, in the long time, real time monitoring is possible. How does DWD currently deal with the data when facing the power outrage. (Kai: message is lost)

  • Hassan: In WIS2 vision, Real-time monitoring is needed and valuable.

  • Takuma: 6-hour interval is enough. To start with “loosely” coupled is possible.

  • Cristina: hourly seems reasonable. The interval depends on the action be taken for the monitoring.

  • Francilly: tightly coupled is better for more efficient monitoring and GBON requirement.

• Decision

  • The team agreed: sensor centers transfer aggregated information on an hourly basis

4

Data exchange format (Sensor Center and Monitoring Center)  

ALL

• Kai shared thoughts about the report from message, topic structure, reporting format for different data format, etc. The presentation can be found here.

• Summary of Discussion

  • Software that could be used by sensor centers

  • DWD (Kai) provides a containerized version of his prototype that could be jointly developed. Containerization makes reliable handling of WMO file-formats easier. (For example, the current implementation using the ECCodes image)

  • Software is intended to be periodically run on a directory of files

  • Could be developed into a “pipeline” where files are ingested into the system and processed by metadata extraction, aggregation and publication components

  • Pursue modular approach. One routine or container per file-type

• Discussions:

  • agreed: sensor centers generate reports

  • Pub/Sub from SC(Sensor Center) to MC (Monitoring Center)

  • reporting format for different data format (BUFR/GRIB/TAC…)

    • Timo: JSON is better, adding new fields.

    • Kai: software to generate report should be open source that SC can use

    • Cristina: is quality part of the report? (Kai: No. Observation unreadable, report will be marked as missing observation.)

    • Takumu: docker is difficult to apply into the operational server.

    • Timo: different routines for different data type? (Kai: Yes.)

    • Cristina: (Timo: Is it possible to extract metadata from different files from GTS ) With right software tools to read, data can be extracted and filtered.

  • Topic Structure

    • to be moved to discuss at the next meeting

• Next Steps

  • DWS will setup GTS&WIS2 pub notifications

  • Other SC to test?

  • Is MC agreed?

Decision:

• To discuss who can run the software as SC. and test the software.