...
GISCs may establish agreements within their Area of Responsibility to support additional message queue protocols, e.g. when republishing messages from affiliated centres. Whatever the protocol used by the affiliated centre, GISCs shall publish using one or more [?] of the approved protocols.
Note: OpenAPI has callbacks (using Webhooks). These are being considered for OGC API – Records subscription workflows. See https://github.com/opengeospatial/ogcapi-records/issues/111
For discussion: use of AsyncAPI (a protocol-agnostic abstraction layer for asynchronous APIs)
...
It is not clear what benefit an abstraction layer such as AsyncAPI provides. Message queues are already unfamiliar to many Members; the AsyncAPI abstraction layer adds further complexity that seems unwarranted.
protocol agnostic; auto-generation of client libraries from AsyncAPI endpoint description
Should we investigate innovative solutions for event streaming? It's just packet-based streaming – perhaps it's possible to repurpose technology from video streaming? The key challenge here is making sure that client applications are available. In such circumstances, it would be prudent to deliver the client application (or a libraries from which a client application can be built) alongside the stream.
Discussion:
Peter Silva:
I don't understand AsyncAPI enough to recommend it. I have played with it a bit, just enough to really understand that I don't understand. I do not think topics are channels, and that is probably the main blocker. It might work eventually, but I think AsyncAPI will slow things down if introduced right now. As far as I can tell, they are conceptually incompatible and conflicting.
I have made some minor trials with asyncAPI, and so far, I have nothing even conceptually functional. Firstly, while multi-protocol, it does not support AMQP in any published way that I could see. I had to get an MQTT driver working to have something to work with ... so I had to circle around after doing that. I tried again once I had a working MQTT driver, but the conceptual model for AsyncAPI is quite different and I don´t see how to mesh them at all yet, but I'm very much a neophyte with it.
As far as I can tell (though I am by no means an expert) the topic hierarchy used in AsyncAPI is used to identify entry points, rather than a tree to be interpreted by the routines.
The use of the word "channel" above is a hint to a conceptual mismatch. The various initiatives and protocols use the similar words with conflicting meanings. For example, a "topic" in Kafka, roughly matches an "exchange" in AMQP, is entirely absent in MQTT, and matches "channel" in asyncAPI... and does not match the purpose of a topic hierarchy in either MQTT or AMQP (identification of subsets of information made available on a single channel.)
With AsyncAPI, as far as I can tell so far, you would need to identify a channel per dataset, ending up with tens of thousands of channels (one per topic), and you can only get that data from that channel. So every subscriber has to conceptually subscribe to large sets of channels (end points.)
In contrast, the methods used so far have meant connecting to a single end point, and providing it a list of topic of interest. This is more aligned with how MQP's naturally work.
I might be mistaken about AsyncAPI. I have not spent enough time with it, but what I have proposed before has always been the result of years of experience and prior operational deployment. I only propose what I know works, and I don't know that AsyncAPI can keep its promises yet.
Someone who knew AsyncAPI well might be able to have a single end point that takes care of a whole topic sub-tree, which would better match what I think is needed, but so far I don't know how to do that (one seems to have to associate and endpoint with every single topic in a hierarchy.)
...
Jeremy Tandy:
AsyncAPI is intended to be an abstraction layer for documenting Event APIs. And any abstraction layer intended to work with a wide variety of systems will usually have some issues in mapping capabilities of individual systems.
The benefit is it would force you to define your pub/sub requirement in a technology agnostic way, in the long term this could help to reduce technical debt because you are not building a system definition around the technical capabilities of a particular version of a piece of software.
AsyncAPI gives you a clean way of cleanly separating the channels, from the operations (pub/sub), from the actual servers – independent of the implementation. As this specification can be used to generate code, this opens up the opportunity to generate compliance tools, mock testing clients, etc. without needing the infrastructure or test servers (higher productivity for developers and consumers). Using AsyncAPI also means protocols like web sockets and Kafka (which are commonly used) are on the table, as well as any future protocols that might become available/useful.
Not sure where the confusion about channel vs. topic subscription comes from? An AsyncAPI channel is manifest as a server endpoint ( wss://api.xxx.com/market_data ) but you then subscribe using parameterisation ( wss://api.xxx.com/market_data/AAPL for Apple’s stock price). The AsyncAPI specification describes the endpoint and the subscription – generators create the stubs/clients for you.
...
Replay / catchup
WIS2 has two requirements for distribution of messages:
...
Need to determine how users will discover these services.
Do we need 24-hour message persistence? If you haven’t consumed the messages and with a TTL of 24h, they will still be available. But, do we need 24h depth? So, it could be solved by recommending having a TTL of 3? 6? hours
Discussion:
Peter Silva:
I do not see a requirement for any additional replay service as I think the existing proposal covers it (though perhaps a bit too implicitly.)
...