2021-06-03 Meeting notes

Round table introductions

Dave

Axel Anderson, Hermann Asensio, David Berry, Xiaoxia Chen, Mark Curtis, Dean Lockett, Daniel Michelson, Kevin O’Brien,  Antje Schremmer, Kai Wert

Overview PPT of demonstration project

Open

Antje, Kai, Hermann

• Historically, whenever new data types introduced or shared on the GTS/WIS new TTAAii headers were required.

• These are being phased out with the move to WIS2.0, using message queues (AMQP / MQTT) to advertise / share data and to make data exchange easier.

• With WMO CF-NetCDF it was decided to go straight to WIS2.0 rather than introducing new headers.

• The project has several aims:

  • Testing of new WMO CF-NetCDF profiles as a practical example, including feedback on those profiles;

  • WIS2.0 example using message topic trees;

  • Improvements to technical specifications for both WIS2.0 / message queues and profiles.

• DWD has volunteered to lead the project, including setting up the MQ infrastructure and management of queues.

• There are two different areas that need to be agreed:

  • Exchange and feed of observational data into DWD systems (push, pull or message queue);

  • Provision of DAR metadata to DWD for inclusion in metadata catalogue.

• When new data are received / obtained by DWD a message will be broadcast over the message queue advertising the availability of the data. The data will then be accessible via a number of different protocols (e.g. ftp, email, http).

Questions / discussion

all

• Scope / aims of project:

  • Is the aim of project to only test the distribution of data on the WIS2.0 or is it wider than that, including checks of the data integrity, security and usability?

    • The primary aim is to test the distribution of the data but the testing of the profiles and usability of the data is also a significant element of the project;

    • Data security / access control will be less of a focus, only open data will be included in the pilot project (noting there may be some issues with radar data depending on source).

• Data integrity:

  • Some discussion on integrity. This touched on the usability of the data in the profiles as well as the validation of the files containing the data. There are already CF validation tools that it may be possible to customise or adapt to validate the WMO CF profiles.

• Timescales:

  • We should be able to share ocean / marine data using the appropriate CF profile this autumn / fall, with DWD and Kevin O’Brien working bilaterally to make it a reality. The planned data to be exchanged will be shared on the GTS (in BUFR) under open access so this simplifies data policy aspects. The ERDDAP systems are also set up to make the conversion to the required CF profile straight forward. This includes the publication of the discovery metadata.

  • A longer lead time is required for the radar data, the data is more complicated in both terms of instrumental metadata and data policy. The US currently has an open access policy for its radar data, Canada is working towards this for level 2 data but not quite there. Australia is further behind, with some data made available / open for research applications but with several days delay.

  • Spring 2022 was suggested as an optimistic timescale for the sharing of weather radar data.

• Metadata:

  • The discussion on metadata covered both the DAR metadata and also instrumental metadata.

  • For the DAR metadata to be useful there needs to be an element of human interaction / generation, with meaningful title, description and keywords included in the DAR metadata. The automatic generation of this information is less than ideal and an early discussion is needed over what the minimum metadata should be.

  • It was noted that the DAR metadata is also exposed via search engines such as Google and Bing but for the benefit of this is be maximised the metadata does need to be comprehensive and well described.

  • For the marine data much of the DAR metadata will be included in the NetCDF files or accessible through the planned ERDDAP service. From this, and the NetCDF files, it should be possible to automatically generate the DAR metadata (noting that the required manual input will already have occurred by this point).

  • Instrumental metadata was also discussed, noting that there is some natural variability between data producers and different instrument setups.  A subset of the critical metadata needs to be agreed but this may be less of a focus for the project. This may be more of an interest for the CF-NetCDF task team and some work has already occurred trying to define this within the TT.

Next steps

• As per above, DWD and Kevin to continue bilateral discussion on marine data:

  • August forms a good timescale for experimental exchange due to Saildrone missions over the summer;

  • DWD (Climate) volunteered to act test user;

  • It should also be possible to identify and engage another 2 – 3 test users.

• Work will continue on radar profile but effort under this project will be minimal initially. Lessons will be learnt from the exchange of the marine data and used when sharing the radar data.

• Next group meeting will be planned for late August (Doodle poll to follow)