Extend multi-plotting to multiple plots and flexibility on overlay and contour choices

[ukmo-huw-lewis]

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Ensure rose-suite.conf.example has been updated if new diagnostic added.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[github-actions]

Total coverage: 91% (HTML report)

Name                                                              Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------------------------
src/CSET/__init__.py                                                 93      2     12      0    98%
src/CSET/_common.py                                                 149      0     52      0   100%
src/CSET/cset_workflow/app/fetch_fcst/bin/fetch_data.py             115     27     24      0    79%
src/CSET/cset_workflow/app/finish_website/bin/finish_website.py      70      0      4      0   100%
src/CSET/cset_workflow/app/parbake_recipes/bin/parbake.py            29      0      8      0   100%
src/CSET/cset_workflow/app/send_email/bin/send_email.py              25      0      4      0   100%
src/CSET/cset_workflow/lib/python/jinja_utils.py                     17      0      6      0   100%
src/CSET/extract_workflow.py                                         47      0     16      0   100%
src/CSET/graph.py                                                    43      0     14      0   100%
src/CSET/operators/__init__.py                                       89      0     26      0   100%
src/CSET/operators/_atmospheric_constants.py                          9      0      0      0   100%
src/CSET/operators/_colormaps.py                                    205      4     54      4    97%
src/CSET/operators/_stash_to_lfric.py                                 3      0      0      0   100%
src/CSET/operators/_utils.py                                        183      8     74      6    95%
src/CSET/operators/ageofair.py                                      141      7     64      5    94%
src/CSET/operators/aggregate.py                                      76      1     22      1    98%
src/CSET/operators/aviation.py                                       60      0     18      0   100%
src/CSET/operators/collapse.py                                      154     12     72      5    91%
src/CSET/operators/constraints.py                                   107      7     46      2    93%
src/CSET/operators/convection.py                                     37      4     10      2    87%
src/CSET/operators/ensembles.py                                      27      0     14      0   100%
src/CSET/operators/filters.py                                        66      2     30      0    98%
src/CSET/operators/humidity.py                                       89      0     32      0   100%
src/CSET/operators/imageprocessing.py                                56      0     16      0   100%
src/CSET/operators/mesoscale.py                                      17      0      2      0   100%
src/CSET/operators/misc.py                                          146      0     54      1    99%
src/CSET/operators/plot.py                                          929    169    318     59    78%
src/CSET/operators/power_spectrum.py                                 97      3     30      3    95%
src/CSET/operators/precipitation.py                                  93      0     50      0   100%
src/CSET/operators/pressure.py                                       41      0     12      0   100%
src/CSET/operators/read.py                                          409     38    178     14    89%
src/CSET/operators/regrid.py                                        122     18     64      1    83%
src/CSET/operators/scoreswrappers.py                                 47      6     12      3    85%
src/CSET/operators/temperature.py                                   121      0     32      0   100%
src/CSET/operators/transect.py                                       62      0     24      0   100%
src/CSET/operators/wind.py                                           17      0      4      0   100%
src/CSET/operators/write.py                                          15      0      6      0   100%
src/CSET/recipes/__init__.py                                        101      0     28      0   100%
---------------------------------------------------------------------------------------------------
TOTAL                                                              4107    308   1432    106    91%

[ukmo-huw-lewis]

Summary of changes:

• Updated plot.py to allow overlay_cube and contour_cube to be optional inputs to plot.spatial_multi_pcolormesh_plot (note this is also required functionality for Extend functionality for MetDB observations and related point-based observation recipe support #2104).

• Added new tests to tests/operators/test_plot.py to ensure successful call and output to plot.spatial_multi_pcolormesh_plot with different combinations of overlay_cube and contour_cube.

• Added new general recipes multi_contour_spatial_plot_sequence.yaml and multi_overlay_spatial_plot_sequence.yaml to support instances of contour-only or overlay-only calls to plot.spatial_multi_pcolormesh_plot. Updated existing recipe multi_surface_spatial_plot_sequence.yaml to have consistent title behaviour - enables greater control of output plot titles when generating different combinations of variables.

• Updated workflow GUI and meta help information to accept
  a) lists of input variables for multi-variable plotting, allowing users to trigger multiple multi-variable outputs from same workflow run
  b) flexibility to leave contour or overlay variable names as "" or None to trigger overlay-only or contour-only outputs.

• Updated cset_workflow call logic in loaders/spatial_field.py to support increased flexibility described above.

Example workflow output: https://wwwspice/~huw.lewis/CSET/CSET_SEA_DET_multiALL_mix/

Example outputs from command line using a) multi_contour_spatial_plot_sequence.yaml, b) multi_overlay_spatial_plot_sequence.yaml and c) multi_surface_spatial_plot_sequence.yaml.

Marking as ready for review.

[Claudio Sanchez (mo-claudiosanchez)]

Summary of changes:

* Updated `plot.py` to allow overlay_cube and contour_cube to be optional inputs to `plot.spatial_multi_pcolormesh_plot` (_note this is also required functionality for [Extend functionality for MetDB observations and related point-based observation recipe support #2104](https://github.com/MetOffice/CSET/pull/2104)_).

* Added new tests to `tests/operators/test_plot.py` to ensure successful call and output to `plot.spatial_multi_pcolormesh_plot` with different combinations of `overlay_cube` and `contour_cube`.

* Added new general recipes `multi_contour_spatial_plot_sequence.yaml` and `multi_overlay_spatial_plot_sequence.yaml` to support instances of contour-only or overlay-only calls to `plot.spatial_multi_pcolormesh_plot`. Updated existing recipe `multi_surface_spatial_plot_sequence.yaml` to have consistent title behaviour - enables greater control of output plot titles when generating different combinations of variables.

* Updated workflow GUI and meta help information to accept
  a) lists of input variables for multi-variable plotting, allowing users to trigger multiple multi-variable outputs from same workflow run
  b) flexibility to leave contour or overlay variable names as `""` or `None` to trigger overlay-only or contour-only outputs.

* Updated `cset_workflow` call logic in `loaders/spatial_field.py` to support increased flexibility described above.

Example workflow output: https://wwwspice/~huw.lewis/CSET/CSET_SEA_DET_multiALL_mix/

Example outputs from command line using a) multi_contour_spatial_plot_sequence.yaml, b) multi_overlay_spatial_plot_sequence.yaml and c) multi_surface_spatial_plot_sequence.yaml.

Marking as ready for review.

Hi, the example plots are linked to private-user-images.githubusercontent.com, I cannot open them.

[Claudio Sanchez (mo-claudiosanchez)]

The _FIELD variables in rose-suite.conf are now a list with multiple fields, wouldn't it be clearer to rename them as _FIELDS?

[Claudio Sanchez (mo-claudiosanchez)]

Rewriting the comment ...

This functionality allows the user to produce several "multi" coloured plots and to add contours and/or masked coloured fields. The number of plots (and their base fields to be coloured) is declared in the MULTI_BASE_FIELD variable via a python list, the MULTI_CONTOUR_FIELD, MULTI_OVERLAY_MASK_CONDITION and MULTI_OVERLAY_MASK_VALUE variables should have to have the same number of elements. Some of these elements could be empty strings or set to "None" to ignore the contouring or the masking + overplotting.

Few suggestions

1. Update the help panel for SPATIAL_MULTI_VARIABLE to reflect the above and how to use this new "multi-field" & "multi-plot" functionality.
2. As suggested in the previous comment make _VARIABLE and _FIELD variables prural.
3. Is there an error trap to check MULTI_CONTOUR_FIELD, MULTI_OVERLAY_MASK_CONDITION and MULTI_OVERLAY_MASK_VALUE have all got the same number of elements than MULTI_BASE_FIELD, if they got less the lines below in src/CSET/loaders/spatial_field.py will fail.

    for index, _ in enumerate(conf.MULTI_BASE_FIELD):
        ...
        "OVERLAY_MASK_VALUE": conf.MULTI_OVERLAY_MASK_VALUE[index],
        "VARNAME_CONTOUR": conf.MULTI_CONTOUR_FIELD[index],

There is a more complicated alternative, though it may be cleaner for the user: Following the MODEL_COUNT in CSET, domain variables in the RAS or region/resolution/configuration in the RNS and ENS. There would be several panels for "multi-plot" and a macro to read in values from each panel. There is only one BASE variable for each panel and, if fields names are provided in their respective variables, contour and masking+overplotting.

[ukmo-huw-lewis]

Claudio Sanchez (@mo-claudiosanchez) - many thanks for helpful review comments.

Branch updated with:

1. updated metadata for GUI help on multi-variable panel
2. changes to rose-suite.conf variables to plurals, and related updated to loaders/spatial_field.py
3. updated for loop in loaders/spatial_field.py to use zip(<all_multi_vars, strict=True). This provides a more implicit test that all variables are present in correct numbers, but not sure if other loader scripts have any more direct "if then error" type protections.

Note if attempt to run with incorrect inputs (e.g. MULTI_CONTOUR_FIELDS=["var1", "var2", ] for 3-plot case, following error results in parbake tasks, which should be sufficient to alert user to check inputs:

  File "<path_to....>/src/CSET/loaders/spatial_field.py", line 571, in load
    for base, overlay, mask_condition, mask_value, contour in zip(conf.MULTI_BASE_FIELDS, conf.MULTI_OVERLAY_FIELDS, conf.MULTI_OVERLAY_MASK_CONDITIONS, conf.MULTI_OVERLAY_MASK_VALUES, conf.MULTI_CONTOUR_FIELDS, strict=True):
                                                              ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zip() argument 2 is shorter than argument 1

To respond on general logic of approach, as also considered something like "NPLOTS=#" and then create separate panels etc, but think we need to stick with implementations as simple as possible, and the python_list approach mimics other panels.
Therefore also think useful/necessary to insist on equal number of inputs for all multi-plot inputs, even if results in repeated variables (e.g. if wanting 3 flavours of plot with same base_variable, enter base_variable 3 times). This gives simplest flexibility to design multiple different combinations with more explicit tracking for user and workflow logic.

e.g. updated https://wwwspice/~huw.lewis/CSET/CSET_SEA_DET_multiALL/ now shows example with MULTI_BASE_FIELDS=["temperature_at_screen_level", "temperature_at_screen_level", "relative_humidity_at_screen_level"]

Please check updated branch when time allows. Thank you.

[Claudio Sanchez (mo-claudiosanchez)]

Thanks, looks good. I'd personally add an example in the help panel of SPATIAL_MULTI_VARIABLE. Hope it can be added between the approval and the "squash and merge"