Writing of filled recipe fails with TypeError: '<' not supported between instances of 'str' and 'tuple'

[schlunma]

Describe the bug

The recipe

documentation:
  title: Test
  description: Test
  authors:
    - schlund_manuel

preprocessors:
  global_ocean:
    mask_landsea:
      mask_out: land
    area_statistics:
      operator: sum

diagnostics:
  test:
    variables:
      fgco2:
        short_name: fgco2
        preprocessor: global_ocean
        project: CMIP5
        mip: Omon
        exp: esmHistorical
        ensemble: r1i1p1
        additional_datasets:
          - {dataset: CanESM2, start_year: 1960, end_year: 2005}
          - {dataset: IPSL-CM5A-LR, start_year: 2000, end_year: 2000}
    scripts:
      null

when ran with this configuration

rootpath:
  CMIP5:
    /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ: DKRZ
    /work/bd0854/DATA/ESMValTool2/download: ESGF

fails with the following error:

2024-06-27 11:11:33,746 UTC [2721773] ERROR   Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 533, in run
    fire.Fire(ESMValTool())
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/miniforge3/envs/esm/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 413, in run
    self._run(recipe, session)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 455, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 130, in process_recipe
    recipe.run()
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/recipe.py", line 1090, in run
    filled_recipe = self.write_filled_recipe()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/recipe.py", line 1128, in write_filled_recipe
    recipe = datasets_to_recipe(USED_DATASETS, self._raw_recipe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 341, in datasets_to_recipe
    dataset_recipe = _datasets_to_recipe(datasets)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 83, in _datasets_to_recipe
    recipe = _move_datasets_up(recipe)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 92, in _move_datasets_up
    _move_one_level_up(diagnostic, 'variables', 'additional_datasets')
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 119, in _move_one_level_up
    dataset_mapping[name] = {
                            ^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 120, in <dictcomp>
    _to_frozen(ds): ds
    ^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 105, in _to_frozen
    return tuple(sorted((k, _to_frozen(v)) for k, v in item.items()))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 105, in <genexpr>
    return tuple(sorted((k, _to_frozen(v)) for k, v in item.items()))
                            ^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/from_datasets.py", line 103, in _to_frozen
    return tuple(sorted(_to_frozen(elem) for elem in item))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'str' and 'tuple'
2024-06-27 11:11:33,763 UTC [2721773] INFO
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

See also ESMValGroup/ESMValTool#3693 (comment).

[schlunma]

The problem only appears with the new "DRS per rootpath" feature of v2.12. I couldn't reproduce it in v2.11 (though that does not mean it's not there).

The problem basically boils down to this: the code tries to sort the following object when creating the filled recipe using sorted:

[(('ensemble', 'r0i0p0'), ('institute', 'IPSL'), ('mip', 'fx'), ('product', ('output1', 'output2')), ('short_name', 'sftof')), (('ensemble', 'r0i0p0'), ('institute', 'IPSL'), ('mip', 'fx'), ('product', 'output1'), ('short_name', 'areacello'))]

The problem here is that the first element contains ('product', ('output1', 'output2')) and the second one ('product', 'output1'). Comparing those two fails with this TypeError: '<' not supported between instances of 'str' and 'tuple'.

The two entries for product come from this file. I have no idea, however, why for one dataset product is output1 and for the other it's ('output1', 'output2'). There is no data available in /work/bd0854/DATA/ESMValTool2/download/cmip5/output2, so data availability does not get taken into account (I think).

I am not quite sure how to proceed. The obvious way of fixing this is via using a key for sorted, but since the input to this sorted call can be arbitrarily nested this is not at all trivial.

The much easier solution is to drop sorted. This works fine (I also tested the produced filled recipe).

@ESMValGroup/technical-lead-development-team opinions?

[schlunma]

Ah, I think I finally understand what's happening there: after using this rootpath

rootpath:
  CMIP5:
    /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ: DKRZ
    /work/bd0854/DATA/ESMValTool2/download: ESGF

with deleting some downloaded files the tool eventually tries to get the files for the IPSL dataset from the following sources:

2024-07-02 15:41:13,359 UTC [581496] DEBUG   esmvalcore._recipe.recipe:270 Using input files for variable fgco2 of dataset IPSL-CM5A-LR:
  /work/bd0854/DATA/ESMValTool2/download/cmip5/output1/IPSL/IPSL-CM5A-LR/esmHistorical/mon/ocnBgchem/Omon/r1i1p1/v20120430/fgco2_Omon_IPSL-CM5A-LR_esmHistorical_r1i1p1_185001-200512.nc
with files for supplementary variable sftof:
  /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ/IPSL/IPSL-CM5A-LR/esmHistorical/fx/ocean/fx/r0i0p0/v20120430/sftof/sftof_fx_IPSL-CM5A-LR_esmHistorical_r0i0p0.nc
with files for supplementary variable areacello:
  /work/bd0854/DATA/ESMValTool2/download/cmip5/output1/IPSL/IPSL-CM5A-LR/esmHistorical/fx/ocean/fx/r0i0p0/v20120430/areacello_fx_IPSL-CM5A-LR_esmHistorical_r0i0p0.nc

You can see that the two fx files come from the two different rootpaths. Since only one of them provides the product facet in the DRS

CMIP5:
  input_dir:
    DKRZ: '{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{version}/{short_name}'
    ESGF: '{project.lower}/{product}/{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{version}'
    ...

it will be replaced by the proper value for ESGF (areacello), but stays the original tuple (output1, output2) for DKRZ (sftof).