Architecture

Owned by Enrico Fucile
Last updated: Aug 02, 2021 by Jeremy Tandy (Unlicensed)
4 min read

Evolution not revolution:

Incremental evolution is a necessary part of the transition to WIS2.

We recognise that Members have different levels of capability and are able to move at different speeds. Our aim is not to preclude anyone implementing a more advanced solution - we're just aiming to make the "entry ticket" to participating in WIS2 as simple as possible.

We can converge on optimal solutions over time as participating centres grow their capability and see the benefits of more sophisticated approaches. We can support this evolution by providing Guidelines and best practices that illustrate implementation, exemplars that can be copied, and open-source software that can be adopted.

Issues to consider:

  1. With an incremental, evolutionary approach we will need clear requirements in the Technical Regulations to ensure that evolving solutions remain interoperable.

    1. [Remy] Yes. That is the tricky part of the more “dynamic” / agile method. We have to make sure that everyone is staying on the right (but not entirely defined) track. However, I believe that the risk is small. Considering WMO Members, only a handful of us may be tempted to do advanced stuff. We could e.g. in one/two years identify a new generation of pilot projects with advanced option and manage the risk like that.

  2. Do we need to provide a roadmap for solutions that move toward a target (optimal) state?

    1. [Remy] As part of the paper describing “step 1” (i.e. the minimal, simplest start to adoption of WIS2) we could show/illustrate step 2 and 3. So, we will also need a more visionary paper [for INFCOM2?].

Shared services among GISCs

There are two possible approaches when considering shared services:

  1. Discuss the feasibility of a shared service model for GISCs, then define an architecture that leverages the option of shared service provision,

  2. Define the technical architecture, then look for options around shared service provision.

We recommend that shared service provision is an option we want GISCs to investigate. The key non-technical issues to resolve include governance and cost-recovery / funding.

We should request that TT-GISC begin developing options for how they might organise the community of GISCs to provide shared services. TT-GISC will need to balance the role of GISCs in supporting affiliated centres in their Area of Responsibility, global reach (e.g. ensuring that every region is adequately served), geopolitics (where some Members are unable to directly connect to IT Systems provided by other Members), and efficiency/cost effectiveness when determining how GISC Services are allocated.

For WIS2 to function, GISCs will have to provide a number of services. Let's refer to these as "GISC Services". These may be offered by every GISC, or shared between GISCs according to a scheme approved by TT-GISC.

Working with their Regional Association, a GISC may choose to delegate GISC Services to other centres in its Area of Responsibility. For example, an existing RTH may take responsibility for GISC Services relating to real-time messaging.

Scope:

The real-time component part of WIS2 will be using pub/sub and file servers to fetch the data.

There are four aspects of the WIS2 architecture that impact all participating centres:

  1. Providing access to data (e.g. publishing files via HTTPS or SFTP, or using Web services like the OGC API)

  2. Metadata and catalogues

  3. Real-time notifications via pub/sub messaging

  4. Quality of Service (QoS) and performance monitoring

Priority:

Priority aspects to consider ahead of INFCOM2 are: (2) Metadata and catalogues, and (3) Real-time notifications via pub/sub messaging.

See: Plan - WIS 2.0

Architecture topics:

Metadata and catalogues

Metadata and catalogues are already familiar in WIS; here we aim to simplify but these components already exist.

See: Metadata and Catalogues for WIS2

Real-time notifications via pub/sub messaging

However, use of pub/sub messaging is completely new. Understanding how this aspect will work in WIS2 is an obvious priority. Where to run brokers? What is the overall broker architecture? One single redundant broker in the cloud for everything. Probably not… mandate a broker per NC, per DCPC, per GISC. Maybe - but is it efficient? So the purpose of the “broker” component is to address this kind of question.

See: Pub/sub messaging for WIS2 and Message Queue Protocols (MQP)

[new, 2-Aug-2021] WIS2: topic hierarchies & dataset classifications (meeting notes)

Providing data access

Data access (aspect #1) is a crucial part of WIS2. This may occur in one of three modes (listed in order of increasing complexity):

  1. Data is published in files for download.

  2. Data access via interactive Web services

  3. Hosted processing on a cloud-platform - where the data is too large to (efficiently move) and data processing ‘algorithms’ are deployed and run close to where the data is stored.

The WIS2 Principles assert that we will use Web architectures. For simple file-based data publishing, this means publishing data on a server using the HTTPS and/or SFTP protocols. It’s still a Web service - just a very simple one.

WIS needs to provide low-latency, resilient data access to a global community. In WIS1, this was achieved using the 24-hour cache maintained by GISCs. WIS2 provides opportunity to review this approach. The underpinning question is “where do we store the data”? A unique gigantic 24h cache for all our data? Certainly not! A distributed cache of data? One option is to use existing solutions for file distribution on the Web: Content Delivery Networks (CDN).

However, incorporating a CDN in the WIS2 solution may be considered too revolutionary for many - it's maybe step 2 or step 3 along the evolutionary path. The first step is to get the community to accept data publication via HTTPS or SFTP.

But - we don't need to define this part of the architecture yet. We can leave this for after INFCOM2.

The crucial point for data publishers (NCs and DCPCs) and data consumers is that data files are identified via URL and made available via an HTTPS or SFTP server.

See: File-based data distribution, CDN for WIS2

Quality of Service / performance monitoring

[very draft - more content required here]

Could use a simple JSON-format announcement for service performance based on something like Prometheus such as is used in WIS1.