Skip to content

Commit

Permalink
[develop] Streamline SRW App's interface to MET/METplus (ufs-communit…
Browse files Browse the repository at this point in the history
…y#1005)

This PR streamlines the SRW App's interface to the MET/METplus verification tool and implements some bug fixes. Details:
* Replace the field-specific METplus configuration jinja2 templates associated with each METplus tool (these templates are hard-coded for each field) with a single template that contains jinja2 code to any valid field to be verified.
* Add yaml configuration files for verification that specify the fields to verify (including field levels and thresholds). This is in order to consolidate the field/level/threshold information in one place instead of having it spread out and repeated in several hard-coded configuration files.
* Add a python script (decouple_fcst_obs_vx_config.py) to parse these two vx configuration files and create a dictionary of the field/level/threshold information that can then be passed to the unified workflow templating tool.
* Modify the ex-scripts for the verification tasks (exregional_run_met_....sh) to allow the use of the new jinja2 METplus config templates. This includes adding code to call the new script decouple_fcst_obs_vx_config.py and then passing its output to the unified workflow templating tool to generate METplus configuration files from the (new) jinja2 templates.
* Add new environment variables to the rocoto workflow configuration files (verify_[pre|det|ens].yaml) that are needed for using the new jinja2 METplus config templates.
* Bug fixes
  • Loading branch information
gsketefian authored May 1, 2024
1 parent eea4c29 commit 2d94ed4
Show file tree
Hide file tree
Showing 60 changed files with 6,040 additions and 9,779 deletions.
15 changes: 8 additions & 7 deletions jobs/JREGIONAL_CHECK_POST_OUTPUT
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,11 @@ print_info_msg "
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
This is the J-job script for the task that checks that all the post-
processed output files in fact exist and are at least a certain age.
These files may have been generated by UPP as part of the current SRW
App workflow, or they may be user-staged.
This is the J-job for the task that checks that no more than
NUM_MISSING_FCST_FILES_MAX of each forecast's (ensemble member's) post-
processed output files are missing. Note that such files may have been
generated by UPP as part of the current SRW App workflow, or they may be
user-staged.
========================================================================"
#
#-----------------------------------------------------------------------
Expand All @@ -70,9 +71,9 @@ Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
#-----------------------------------------------------------------------
#
# Create a flag file to make rocoto aware that the make_grid task has
# successfully completed (so that other tasks that depend on it can be
# launched).
# Create a flag file to make rocoto aware that the check_post_output task
# has successfully completed (so that other tasks that depend on it can
# be launched).
#
#-----------------------------------------------------------------------
#
Expand Down
97 changes: 97 additions & 0 deletions jobs/JREGIONAL_PARSE_VX_CONFIG
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
#
#
#-----------------------------------------------------------------------
#

#
#-----------------------------------------------------------------------
#
# Source the variable definitions file and the bash utility functions.
#
#-----------------------------------------------------------------------
#
. $USHdir/source_util_funcs.sh
source_config_for_task "task_parse_vx_config" ${GLOBAL_VAR_DEFNS_FP}
. $USHdir/job_preamble.sh
#
#-----------------------------------------------------------------------
#
# Save current shell options (in a global array). Then set new options
# for this script/function.
#
#-----------------------------------------------------------------------
#
{ save_shell_opts; . $USHdir/preamble.sh; } > /dev/null 2>&1
#
#-----------------------------------------------------------------------
#
# Get the full path to the file in which this script/function is located
# (scrfunc_fp), the name of that file (scrfunc_fn), and the directory in
# which the file is located (scrfunc_dir).
#
#-----------------------------------------------------------------------
#
scrfunc_fp=$( $READLINK -f "${BASH_SOURCE[0]}" )
scrfunc_fn=$( basename "${scrfunc_fp}" )
scrfunc_dir=$( dirname "${scrfunc_fp}" )
#
#-----------------------------------------------------------------------
#
# Print message indicating entry into script.
#
#-----------------------------------------------------------------------
#
print_info_msg "
========================================================================
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
This is the J-job script for the task that reads in the \"coupled\" yaml
verification (vx) configuration file (python dictionary) and generates
from it two \"decoupled\" vx configuration dictionaries, one for forecasts
and another for observations. The task then writes these two decoupled
dictionaries to a new configuration file in the experiment directory
that can be read by downstream vx tasks.
Note:
The \"coupled\" vx configuration file contains items (dictionary keys and
values representing field names, levels, and thresholds) that consist of
both the forecast and the observation value for that item separated by a
delimiter string. Thus, they first need to be separated (decoupled) into
a value for forecasts and one for the observations before they can be
further processed.
========================================================================"
#
#-----------------------------------------------------------------------
#
# Call the ex-script for this J-job and pass to it the necessary varia-
# bles.
#
#-----------------------------------------------------------------------
#
$SCRIPTSdir/exregional_parse_vx_config.sh || \
print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
#-----------------------------------------------------------------------
#
# Run job postamble.
#
#-----------------------------------------------------------------------
#
job_postamble
#
#-----------------------------------------------------------------------
#
# Restore the shell options saved at the beginning of this script/func-
# tion.
#
#-----------------------------------------------------------------------
#
{ restore_shell_opts; } > /dev/null 2>&1

4 changes: 2 additions & 2 deletions jobs/JREGIONAL_RUN_MET_PB2NC_OBS
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ print_info_msg "
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
This is the J-job script for the task that runs METplus for point-stat
by initialization time for all forecast hours.
This is the J-job script for the task that runs METplus for pb2nc on
NDAS observations.
========================================================================"
#
#-----------------------------------------------------------------------
Expand Down
Loading

0 comments on commit 2d94ed4

Please sign in to comment.