Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perfmetrics recipe fails in multi model mean step #2559

Open
schlunma opened this issue Oct 17, 2024 · 5 comments · May be fixed by #2564
Open

Perfmetrics recipe fails in multi model mean step #2559

schlunma opened this issue Oct 17, 2024 · 5 comments · May be fixed by #2564
Assignees
@bouweandela
Labels
bug Something isn't working
Milestone
v2.12.0

Comments

@schlunma
Copy link
Contributor

schlunma commented Oct 17, 2024 •
edited
Loading

Describe the bug

Running

esmvaltool run --search_esgf=when_missing ~/ESMValTool/esmvaltool/recipes/recipe_perfmetrics_CMIP5.yml

with the current ESMValCore main branch fails with a iris.exceptions.CannotAddError: Duplicate coordinates are not permitted. error. I think this is related to #2518. @bouweandela could you please take a look? Thanks!

Traceback 
2024-10-17 15:20:05,968 UTC [334034] ERROR   esmvalcore._main:630 Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_task.py", line 895, in _run_task
    output_files = task.run()
                   ^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_task.py", line 290, in run
    self.output_files = self._run(input_files)
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/__init__.py", line 710, in _run
    self.products = _apply_multimodel(
                    ^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/__init__.py", line 624, in _apply_multimodel
    result = preprocess(products - exclude, step, **settings)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/__init__.py", line 419, in preprocess
    _run_preproc_function(
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/__init__.py", line 364, in _run_preproc_function
    return function(items, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 815, in multi_model_statistics
    group_statistics = _multiproduct_statistics(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 640, in _multiproduct_statistics
    statistics_cubes = _multicube_statistics(
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 620, in _multicube_statistics
    result_cube = _compute_eager(cubes, operator=agg, **agg_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 487, in _compute_eager
    combined_cube = _combine(input_slices)
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/preprocessor/_multimodel.py", line 416, in _combine
    cube.add_aux_coord(concat_dim)
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/site-packages/iris/cube.py", line 1494, in add_aux_coord
    raise iris.exceptions.CannotAddError(
iris.exceptions.CannotAddError: Duplicate coordinates are not permitted.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 618, in run
    fire.Fire(ESMValTool())
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 446, in run
    self._run(recipe, session, cli_config_dir)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 501, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_main.py", line 143, in process_recipe
    recipe.run()
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_recipe/recipe.py", line 1158, in run
    self.tasks.run(max_parallel_tasks=self.session["max_parallel_tasks"])
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_task.py", line 811, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_task.py", line 857, in _run_parallel
    _copy_results(task, running[task])
  File "/home/b/b309141/repos/ESMValCore/esmvalcore/_task.py", line 884, in _copy_results
    task.output_files, task.products = future.get()
                                       ^^^^^^^^^^^^
  File "/work/bd0854/b309141/micromamba/envs/esm/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
iris.exceptions.CannotAddError: Duplicate coordinates are not permitted.

Recipe: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/recipe_perfmetrics_CMIP5.yml
Info log: main_log.txt

(The full debug log is too large to upload [39MB])
@schlunma schlunma added the bug Something isn't working label Oct 17, 2024
@schlunma schlunma added this to the v2.12.0 milestone Oct 17, 2024
This was referenced Oct 18, 2024
Merged
Draft
@bouweandela
Copy link
Member

Could you share a bit more than the stack trace? At least the error message coming from the preprocessor that lists the inputs and output file would be very helpful.

@schlunma
Copy link
Contributor Author

This is the info log:

main_log.txt

@valeriupredoi
Copy link
Contributor

had a look at Manu's log file -> it appears iris is complaining the multimodel stat aux coord already exists, before adding it? It'd be good to print out the cube's aux coords before we try shove in that aux coord, I smell an iris overly cautious criterion there 🐀

@bouweandela
Copy link
Member

bouweandela commented Oct 22, 2024 •
edited
Loading

I'm seeing a similar issue when concatenating FGOALS-f3-L data in examples/recipe_easy_ipcc.yml with the development branch of ESMValCore and Iris, and that seems to work fine with ESMValCore v2.11.0/Iris 3.10 so I suspect it could also be related to something that changed in the fixes.

@bouweandela
Copy link
Member

It looks like it is the combination of a statistics operator that supports lazy statistics (mean) and one that does not (median) in computing variable sm in the recipe. Non-lazy operators will hopefully soon be a thing of the past, but I'll put up a pull request to fix the issue.

@bouweandela bouweandela linked a pull request Oct 23, 2024 that will close this issue
Draft
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bouweandela bouweandela

Labels
bug Something isn't working
Projects
None yet
Milestone
v2.12.0
3 participants
@bouweandela @valeriupredoi @schlunma