E3SM_diag preprocess for land variables

[BunnyVon]

I'd like to take advantage of zppy to preprocess the E3SM output for E3SM_diags. My focus is on land variables this time so didn't use zppy end-to-end. (However, it's been a while since the last time when I checked about zppy function for land analysis. Has zppy incorporated analyzing land variables with E3SM_diags now? )

I have a few questions related to this purpose:

1. Does [ts] in the zppy .cfg file control the preprocessing for E3SM_diags?
2. Assuming the answer to Q1 is positive, in my zppy .cfg file, I have

[ts]
active = True
years = "1870:2013:48","1870:2013:144",

strangely, only the part with 48 years interval was processed.

(base) [ac.sfeng1@chrlogin2:/lcrc/group/e3sm/ac.sfeng1/E3SM_output/20230717_CBGCv2.r05.chrysalis.I20TRGSWCNP/zppy/post/lnd/180x360_aave/ts/monthly]$ ls
48yr

there is no directory for 144yr. when I checked the error message, it shows the following error message (not in 48yr output, though). Does anyone know what causes this issue for 144yr but not for 48yr process?

ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0250 --gaa remap_version=5.1.3    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/home/ac.sfeng/zppy/CBGCv2production/map_r05_cmip6_180x360_aave.20221218.nc" "trash/NPP_187001_201312.nc" "output/NPP_187001_201312.nc"
/lcrc/soft/climate/e3sm-unified/spack/spack_for_mache_1.9.0/var/spack/environments/e3sm_unified_1_8_0_chrysalis_gnu_openmpi/.spack-env/view/bin/ncremap: line 3421: 2040631 Killed                  ncks -O -t 1 --no_tmp_fl --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0250 --gaa remap_version=5.1.3 --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/home/ac.sfeng/zppy/CBGCv2production/map_r05_cmip6_180x360_aave.20221218.nc" "trash/SNOW_DEPTH_187001_201312.nc" "output/SNOW_DEPTH_187001_201312.nc"
ncremap: ERROR Failed to horizontally regrid. cmd_rgr[0] failed. Debug this:
 ncks -O -t 1 --no_tmp_fl --hdr_pad=10000 --no_tmp_fl --hdr_pad=10000 --gaa remap_script=ncremap --gaa remap_hostname=chr-0250 --gaa remap_version=5.1.3    --rgr lat_nm_out=lat --rgr lon_nm_out=lon --rgr sgs_frc_nm=landfrac#sgs_msk_nm=landmask#sgs_nrm=1.0 --map_fl="/home/ac.sfeng/zppy/CBGCv2production/map_r05_cmip6_180x360_aave.20221218.nc" "trash/SNOW_DEPTH_187001_201312.nc" "output/SNOW_DEPTH_187001_201312.nc"
Tue Aug 22 12:58:05 CDT 2023: Regridded output/TLAI_187001_201312.nc
ncclimo: ERROR Failed to regrid. cmd_rgr[1] failed. Debug this:
 ncremap -u .pid2036211.split.1.tmp --nco_opt="--no_tmp_fl --hdr_pad=10000" -t 1  --no_stagger -P elm  --map=/home/ac.sfeng/zppy/CBGCv2production/map_r05_cmip6_180x360_aave.20221218.nc trash/EFLX_LH_TOT_187001_201312.nc output/EFLX_LH_TOT_187001_201312.nc
/lcrc/soft/climate/e3sm-unified/spack/spack_for_mache_1.9.0/var/spack/environments/e3sm_unified_1_8_0_chrysalis_gnu_openmpi/.spack-env/view/bin/ncclimo: line 2922: kill: (2036983) - No such process
/lcrc/soft/climate/e3sm-unified/spack/spack_for_mache_1.9.0/var/spack/environments/e3sm_unified_1_8_0_chrysalis_gnu_openmpi/.spack-env/view/bin/ncclimo: line 2922: kill: (2036986) - No such process
/lcrc/soft/climate/e3sm-unified

My .cfg file is /home/ac.sfeng/zppy/CBGCv2production/post.CBGCv2.LR.I20TRGSWCNP.cfg . on Chrysalis. Thanks.

[chengzhuzhang]

@BunnyVon Thanks for the post. Not directly related to this issue, but to add what I think for zppy to support the land variables diagnostics, for now, we need:

• ts and climatology for land variables
• e3sm_diags model vs model task for lat_lon_land [need to add additional variables]
• global time series for land variables
• over view plot for land variables
• ilamb [only a default setting is supported right now]
  I think these capabilities are built, but there was not a configuration file that including these tasks with a land forcus. A campaign configuration file for land group will be ideal.

In addition, e3sm_diags needs to support additional variables requested after its first implementation of lat_lon_land.

[chengzhuzhang]

@forsyth2 is out of office. I'm wondering if @czender could help to look at the ncclimo error and help debug?

[czender]

ncclimo calls ncremap to regrid timeseries. The error occurs in ncremap when regridding, e.g., this file: trash/NPP_187001_201312.nc. Given that it works for shorter timeseries and breaks for longer, the problem could be an out-of-memory (OOM) error? It's hard to check without having access to this file, or to the complete command with which ncclimo was invoked so the location of the input files is more clear. You could rule in/out the OOM hypothesis by copying the file to a node with more RAM (e.g., Perlmutter has 512 GB/node) and seeing if it works there.

[BunnyVon]

I can give it a try and keep you all posted. Thank you @chengzhuzhang and @czender

[BunnyVon]

Yep, it's a memory issue. zppy ran through 144 years on Perlmutter @czender @chengzhuzhang

[czender]

A way to possibly workaround this issue, without moving to Perlmutter, might be to exclude all the 3D (i.e., with a depth dimension) ELM variables from the processing (if you do not need them)

[forsyth2]

Does [ts] in the zppy .cfg file control the preprocessing for E3SM_diags?

It depends on the task. The "enso_diags"/"qbo"/"area_mean_time_series/streamflow" tasks require [ts] as a dependency:

https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/default.ini

# `reference_data_path` but for "enso_diags"/"qbo"/"area_mean_time_series" runs
# Required for run_type="model_vs_model" "enso_diags"/"qbo"/"area_mean_time_series" runs
reference_data_path_ts = string(default="")
# `reference_data_path` but for "streamflow" runs
# Required for run_type="model_vs_model" "streamflow" runs
reference_data_path_ts_rof = string(default="")

[forsyth2]

https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg

A [ts] subtask:

  [[ land_monthly ]]
  extra_vars = "landfrac"
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  vars = "FSH,LAISHA,LAISUN,RH2M"
  ts_fmt = "cmip"

This is currently used as a dependency for ILAMB:

$ git grep -n land_subsection
ilamb.py:62:                        c["ts_land_subsection"],
templates/default.ini:298:ts_land_subsection = string(default="land_monthly")

So, I don't think E3SM Diags, at the moment, is using it. @chengzhuzhang, do I have that right?

[chengzhuzhang]

e3sm_diags can support model vs model for land variables, and we'd like to have a zppy configuration for land diagnostics. I think what we need to figure out is if zppy can support this with a new configuration. And other requested features are documented here.

[acme-y9s]

@chengzhuzhang Yes, we can use e3sm_diags to make plots for model vs model for some selected land variables. My concerns are whether there is a convenient way to add additional land variables to e3sm_diags, and has the capability of the time series for land variable been included to e3sm_diags? Any help needed from our land side? I can provide the land variable list and some scripts we use for land diagnostics. Looking forward to having a new zippy configuration for land component diagnostic.

[acme-y9s]

@forsyth2 It seems that is very convenient to add more land variables for iLAMB land diagnostic through the subtask you pointed out:[[ land_monthly ]]
extra_vars = "landfrac"
frequency = "monthly"
input_files = "elm.h0"
input_subdir = "archive/lnd/hist"
vars = "FSH,LAISHA,LAISUN,RH2M"
ts_fmt = "cmip"

Is there one similar sub task for land variables that we can do with e3sm_diags? Thanks!

[chengzhuzhang]

Hi @acme-y9s thank you for chiming in! It is great to know that ELM developers are interested in adopting zppy workflow. We just had a work meeting on hashing out steps to support a land component configuration. @forsyth2 will take the lead to generate a configuration for land. Adding more variables to e3sm_diags is pretty straightforward. I did a first round of variables based on Dan's script. The PR is can be find herehttps://github.com/E3SM-Project/e3sm_diags/pull/591/files. It would be great if someone from the land team can follow this and to expand the variable list. I also chatted a little with @BunnyVon regarding land support. We'll update when we have a prototype configuration and also ask for future input. And at some point it would be useful to have a meeting to work on refine the workflow and communicate about the requirement.