Exploring the use of message queuing protocols for GTS data exchange

Owned by Xiaoxia Chen
Last updated: Jun 22, 2021
2 min read

Introduction

Weather data distribution consists of the timely distribution of both small (a few kilobytes) and large (several gigabytes) datasets. It is usually using the one-to-many data model. Since the 1990s, the protocols used for data exchange between National Meteorological Services and other operators are, as defined by WMO, FTP, and, more recently, SFTP. The use of WMO FTP is, however, becoming an issue for data exchange due to:

  • Confidentiality: files are being exchanged without encryption

  • Integrity: it is not possible to identify the data source as messages/files are not signed

  • Availability: the protocol in itself does not provide mechanisms in case of telecommunication link failures

Performance: when delivering many files especially over high latency networks, FTP is not efficient and does not make use of all available bandwidth.  The WMO FTP is used for GTS data exchange and in order to provide all GTS/WIS features. NMHSs have implemented Automated Message Switching Systems (AMSS) to satisfy the WIS-GTS requirements of message routing using FTP as the exchange protocol. The emergence of new open standard message queuing protocols provides an opportunity to improve and simplify the systems eliminating the need for routing procedures.

Project description

The project aims to experiment with the international exchange of GTS data using publication/subscription (pub/sub) protocols such as Advanced Message Queuing Protocol (AMQP) and Message Queuing Telemetry Transport (MQTT) protocols. Several Global Information System Centres (GISC), Data Collection or Production Centres (DCPC), and National Centres (NCs) will be involved in the project in order to leverage the existing national experience on the use of pub/sub solutions in the context of international collaboration.

Project Lead

Peter Silva (Canada)