Principle 5: WIS 2.0 encourages NCs and DCPCs to provide 'data reduction' services via WIS that process 'big data' to create results or products that are small enough to be conveniently downloaded and used by those with minimal technical infrastructure

WIS 2.0: encourages NCs and DCPCs to provide 'data reduction' services via WIS that process 'big data' to create results or products that are small enough to be conveniently downloaded and used by those with minimal technical infrastructure

BENEFIT:

Using ‘data reduction’ Web services to process high volume, complex data remotely, Members’ agencies and institutions can deliver high-value, high-quality services to their governments and citizens helping them more effectively meet their national mandates without the need to invest in and operate their own data management infrastructure[1].

Real-time delivery of data and products in support of the World Weather Watch programme remains a core requirement for WIS. Current data-exchange methods permitted on the GTS[2] require manual intervention in response to each user request for real-time data delivery, e.g. to set up and configure a new data delivery path. This practice will not scale to meet the demand for real-time data from across the meteorological community.

Modern messaging protocols, such as those that underpin social media platforms like WhatsApp and Twitter, solve this problem by automating how the relationship between data provider and consumer is established. Data providers create a channel ('message queue') and categorize which data are published to that channel. Data consumers determine which channels contain data of interest and, assuming they have the necessary access rights, subscribe to them. Once a subscription to a channel is established, data published to that channel are then automatically sent to the subscriber. This is known as the publish-subscribe messaging pattern ('pubsub'). Using such modern messaging protocols, there is no manual configuration burden on data providers to add new subscribers.

Note that these modern messaging protocols may also be used to send notifications to subscribers. For example, to alert subscribers that new data or products are available for them to access or download at their convenience.

[1] Provision of data processing services in this way supports the WMO mantra the "no Member be left behind" and that "no Member stands alone" - through cooperation, all Members should access to the necessary capability to work with the predicted increase in data volumes

[2] File Transfer Protocol (FTP) and Secure File Transfer Protocol (SFTP); see Manual on GTS (WMO-No. 386), Attachment II-15