Skip to content

Commit

Permalink
Feature #2547 Read WRF output files natively (#2790)
Browse files Browse the repository at this point in the history
* Per #2547, add the file_type = NETCDF_WRF configuration option.

* Per #2547, rename vx_data2d_nc_interp as vx_data2d_nc_wrf and vx_data2d_nccf as vx_data2d_nc_cf.

* Per #2547, rename nccf files as nc_cf for consistency throughout

* Per #2547, rename Pinterp classes as Wrf

* only set -L and -I arguments for atlas and eckit if the appropriate environment variable is set -- this matches how it is handled for other external library dependencies

* add rpath for atlas and eckit lib dirs so dynamic libraries can be found when running on seneca

* ignore directory automatically created by CLion

* rename pinterp variables and classes to Wrf

* more rename and fix typos

* per #2547, add support for reading WRF files by adding support for different X/Y/Z dimensions, read X/Y/Z dimension names when reading a field instead of storing them per file, track if data is staggered in X/Y/Z dimension (still need to add DataPlane function to destagger data), handle pressure field that also includes time dimension

* per #2547, add DataPlane function to handle staggering of grid

* per #2547, change file type returned from pinterp to wrf

* per #2547, call function to stagger DataPlane if necessary

* per #2547, keep store if variable is on pressure levels and only read pressure field if it is

* ignore cmake directory

* per #2547, support U_PL, U_ZL, U, and U<x> where <x> is an integer in is_u_wind and the same for is_v_wind

* change regex to match zero or more instances of a number after U or V instead of 1 or more to include the fields named U and V

* error and exit if var with unsupported subgrid dimension is processed

* clean up logging

* only destagger Z dimension for bottom_top_stag because other _stag Z dimensions don't need to be destaggered, clean up logic to read X/Y/Z dimension info

* Per #2547, instead of checking dimensions of all vars in file, check dimensions for the var that is being read. Only read pressure var if the z dimension matches the var to read. Set class variable DimNames instead of local gDimNames so dump function will work as expected

* per #2547, support reading variable that corresponds to dimension with different name as dimension and includes a time dimension, e.g. P_PL(Time, num_press_levels_stag)

* per #2547, added unit tests to process WRF out data with plot_data_plane and grid_stat

* merged develop and resolved conflicts

* rename lib from pinterp to wrf

* rename lib

* fixed incorrect merge changes

* fixed broken unit test format

* Per #2547, restore the mention of NAK.poly to Appendix B.

* Change model to WRF

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>

* change obtype to WRF

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>

* plot higher vertical level to get non-zero output

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>

* indentation

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>

---------

Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
  • Loading branch information
georgemccabe and JohnHalleyGotway authored Jan 19, 2024
1 parent 51227b9 commit 464f9a7
Show file tree
Hide file tree
Showing 136 changed files with 1,355 additions and 680 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,5 @@ share
Makefile
make.log
make_install.log
.idea
cmake-build-debug
8 changes: 4 additions & 4 deletions Make-include
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ MET_CPPFLAGS = -I${top_builddir}/src/basic/vx_cal \
-I${top_builddir}/src/libcode/vx_data2d_factory \
-I${top_builddir}/src/libcode/vx_data2d_grib \
-I${top_builddir}/src/libcode/vx_data2d_grib2 \
-I${top_builddir}/src/libcode/vx_data2d_nccf \
-I${top_builddir}/src/libcode/vx_data2d_nc_met \
-I${top_builddir}/src/libcode/vx_data2d_nc_pinterp \
-I${top_builddir}/src/libcode/vx_data2d_nc_wrf \
-I${top_builddir}/src/libcode/vx_data2d_nc_cf \
-I${top_builddir}/src/libcode/vx_data2d_python \
-I${top_builddir}/src/libcode/vx_data2d_ugrid \
-I${top_builddir}/src/libcode/vx_python3_utils \
Expand Down Expand Up @@ -58,9 +58,9 @@ MET_LDFLAGS = -L${top_builddir}/src/basic/vx_cal \
-L${top_builddir}/src/libcode/vx_data2d_factory \
-L${top_builddir}/src/libcode/vx_data2d_grib \
-L${top_builddir}/src/libcode/vx_data2d_grib2 \
-L${top_builddir}/src/libcode/vx_data2d_nccf \
-L${top_builddir}/src/libcode/vx_data2d_nc_met \
-L${top_builddir}/src/libcode/vx_data2d_nc_pinterp \
-L${top_builddir}/src/libcode/vx_data2d_nc_wrf \
-L${top_builddir}/src/libcode/vx_data2d_nc_cf \
-L${top_builddir}/src/libcode/vx_data2d_python \
-L${top_builddir}/src/libcode/vx_data2d_ugrid \
-L${top_builddir}/src/libcode/vx_python3_utils \
Expand Down
11 changes: 6 additions & 5 deletions configure
Original file line number Diff line number Diff line change
Expand Up @@ -10133,11 +10133,12 @@ fi

# Create configured files

ac_config_files="$ac_config_files Makefile scripts/Rscripts/Makefile scripts/Rscripts/include/Makefile scripts/python/Makefile scripts/python/examples/Makefile scripts/python/met/Makefile scripts/python/pyembed/Makefile scripts/python/utility/Makefile scripts/python/tc_diag/Makefile scripts/python/tc_diag/atcf_tools/Makefile scripts/python/tc_diag/config/Makefile scripts/python/tc_diag/diag_lib/Makefile scripts/python/tc_diag/tc_diag_driver/Makefile data/Makefile data/climo/Makefile data/climo/seeps/Makefile data/colortables/Makefile data/colortables/NCL_colortables/Makefile data/config/Makefile data/map/Makefile data/map/admin_by_country/Makefile data/poly/Makefile data/poly/HMT_masks/Makefile data/poly/NCEP_masks/Makefile data/ps/Makefile data/table_files/Makefile data/tc_data/Makefile src/Makefile src/basic/Makefile src/basic/enum_to_string/Makefile src/basic/vx_cal/Makefile src/basic/vx_config/Makefile src/basic/vx_log/Makefile src/basic/vx_math/Makefile src/basic/vx_util/Makefile src/basic/vx_util_math/Makefile src/libcode/Makefile src/libcode/vx_afm/Makefile src/libcode/vx_analysis_util/Makefile src/libcode/vx_color/Makefile src/libcode/vx_data2d/Makefile src/libcode/vx_data2d_factory/Makefile src/libcode/vx_data2d_grib/Makefile src/libcode/vx_data2d_grib2/Makefile src/libcode/vx_data2d_nc_met/Makefile src/libcode/vx_data2d_nc_pinterp/Makefile src/libcode/vx_data2d_nccf/Makefile src/libcode/vx_data2d_ugrid/Makefile src/libcode/vx_geodesy/Makefile src/libcode/vx_gis/Makefile src/libcode/vx_gnomon/Makefile src/libcode/vx_grid/Makefile src/libcode/vx_gsl_prob/Makefile src/libcode/vx_nav/Makefile src/libcode/vx_solar/Makefile src/libcode/vx_nc_obs/Makefile src/libcode/vx_nc_util/Makefile src/libcode/vx_pb_util/Makefile src/libcode/vx_plot_util/Makefile src/libcode/vx_ps/Makefile src/libcode/vx_pxm/Makefile src/libcode/vx_render/Makefile src/libcode/vx_shapedata/Makefile src/libcode/vx_stat_out/Makefile src/libcode/vx_statistics/Makefile src/libcode/vx_time_series/Makefile src/libcode/vx_physics/Makefile src/libcode/vx_series_data/Makefile src/libcode/vx_regrid/Makefile src/libcode/vx_tc_util/Makefile src/libcode/vx_summary/Makefile src/libcode/vx_python3_utils/Makefile src/libcode/vx_data2d_python/Makefile src/libcode/vx_bool_calc/Makefile src/libcode/vx_pointdata_python/Makefile src/libcode/vx_seeps/Makefile src/tools/Makefile src/tools/core/Makefile src/tools/core/ensemble_stat/Makefile src/tools/core/grid_stat/Makefile src/tools/core/mode/Makefile src/tools/core/mode_analysis/Makefile src/tools/core/pcp_combine/Makefile src/tools/core/point_stat/Makefile src/tools/core/series_analysis/Makefile src/tools/core/stat_analysis/Makefile src/tools/core/wavelet_stat/Makefile src/tools/other/Makefile src/tools/other/ascii2nc/Makefile src/tools/other/lidar2nc/Makefile src/tools/other/gen_ens_prod/Makefile src/tools/other/gen_vx_mask/Makefile src/tools/other/gis_utils/Makefile src/tools/other/ioda2nc/Makefile src/tools/other/madis2nc/Makefile src/tools/other/mode_graphics/Makefile src/tools/other/modis_regrid/Makefile src/tools/other/pb2nc/Makefile src/tools/other/plot_data_plane/Makefile src/tools/other/plot_point_obs/Makefile src/tools/other/wwmca_tool/Makefile src/tools/other/gsi_tools/Makefile src/tools/other/regrid_data_plane/Makefile src/tools/other/point2grid/Makefile src/tools/other/shift_data_plane/Makefile src/tools/other/mode_time_domain/Makefile src/tools/other/grid_diag/Makefile src/tools/tc_utils/Makefile src/tools/tc_utils/tc_dland/Makefile src/tools/tc_utils/tc_pairs/Makefile src/tools/tc_utils/tc_stat/Makefile src/tools/tc_utils/tc_gen/Makefile src/tools/tc_utils/rmw_analysis/Makefile src/tools/tc_utils/tc_rmw/Makefile src/tools/tc_utils/tc_diag/Makefile"
ac_config_files="$ac_config_files Makefile scripts/Rscripts/Makefile scripts/Rscripts/include/Makefile scripts/python/Makefile scripts/python/examples/Makefile scripts/python/met/Makefile scripts/python/pyembed/Makefile scripts/python/utility/Makefile scripts/python/tc_diag/Makefile scripts/python/tc_diag/atcf_tools/Makefile scripts/python/tc_diag/config/Makefile scripts/python/tc_diag/diag_lib/Makefile scripts/python/tc_diag/tc_diag_driver/Makefile data/Makefile data/climo/Makefile data/climo/seeps/Makefile data/colortables/Makefile data/colortables/NCL_colortables/Makefile data/config/Makefile data/map/Makefile data/map/admin_by_country/Makefile data/poly/Makefile data/poly/HMT_masks/Makefile data/poly/NCEP_masks/Makefile data/ps/Makefile data/table_files/Makefile data/tc_data/Makefile src/Makefile src/basic/Makefile src/basic/enum_to_string/Makefile src/basic/vx_cal/Makefile src/basic/vx_config/Makefile src/basic/vx_log/Makefile src/basic/vx_math/Makefile src/basic/vx_util/Makefile src/basic/vx_util_math/Makefile src/libcode/Makefile src/libcode/vx_afm/Makefile src/libcode/vx_analysis_util/Makefile src/libcode/vx_color/Makefile src/libcode/vx_data2d/Makefile src/libcode/vx_data2d_factory/Makefile src/libcode/vx_data2d_grib/Makefile src/libcode/vx_data2d_grib2/Makefile src/libcode/vx_data2d_nc_met/Makefile src/libcode/vx_data2d_nc_wrf/Makefile src/libcode/vx_data2d_nc_cf/Makefile src/libcode/vx_data2d_ugrid/Makefile src/libcode/vx_geodesy/Makefile src/libcode/vx_gis/Makefile src/libcode/vx_gnomon/Makefile src/libcode/vx_grid/Makefile src/libcode/vx_gsl_prob/Makefile src/libcode/vx_nav/Makefile src/libcode/vx_solar/Makefile src/libcode/vx_nc_obs/Makefile src/libcode/vx_nc_util/Makefile src/libcode/vx_pb_util/Makefile src/libcode/vx_plot_util/Makefile src/libcode/vx_ps/Makefile src/libcode/vx_pxm/Makefile src/libcode/vx_render/Makefile src/libcode/vx_shapedata/Makefile src/libcode/vx_stat_out/Makefile src/libcode/vx_statistics/Makefile src/libcode/vx_time_series/Makefile src/libcode/vx_physics/Makefile src/libcode/vx_series_data/Makefile src/libcode/vx_regrid/Makefile src/libcode/vx_tc_util/Makefile src/libcode/vx_summary/Makefile src/libcode/vx_python3_utils/Makefile src/libcode/vx_data2d_python/Makefile src/libcode/vx_bool_calc/Makefile src/libcode/vx_pointdata_python/Makefile src/libcode/vx_seeps/Makefile src/tools/Makefile src/tools/core/Makefile src/tools/core/ensemble_stat/Makefile src/tools/core/grid_stat/Makefile src/tools/core/mode/Makefile src/tools/core/mode_analysis/Makefile src/tools/core/pcp_combine/Makefile src/tools/core/point_stat/Makefile src/tools/core/series_analysis/Makefile src/tools/core/stat_analysis/Makefile src/tools/core/wavelet_stat/Makefile src/tools/other/Makefile src/tools/other/ascii2nc/Makefile src/tools/other/lidar2nc/Makefile src/tools/other/gen_ens_prod/Makefile src/tools/other/gen_vx_mask/Makefile src/tools/other/gis_utils/Makefile src/tools/other/ioda2nc/Makefile src/tools/other/madis2nc/Makefile src/tools/other/mode_graphics/Makefile src/tools/other/modis_regrid/Makefile src/tools/other/pb2nc/Makefile src/tools/other/plot_data_plane/Makefile src/tools/other/plot_point_obs/Makefile src/tools/other/wwmca_tool/Makefile src/tools/other/gsi_tools/Makefile src/tools/other/regrid_data_plane/Makefile src/tools/other/point2grid/Makefile src/tools/other/shift_data_plane/Makefile src/tools/other/mode_time_domain/Makefile src/tools/other/grid_diag/Makefile src/tools/tc_utils/Makefile src/tools/tc_utils/tc_dland/Makefile src/tools/tc_utils/tc_pairs/Makefile src/tools/tc_utils/tc_stat/Makefile src/tools/tc_utils/tc_gen/Makefile src/tools/tc_utils/rmw_analysis/Makefile src/tools/tc_utils/tc_rmw/Makefile src/tools/tc_utils/tc_diag/Makefile"


if test -n "$MET_DEVELOPMENT"; then
ac_config_files="$ac_config_files src/tools/dev_utils/Makefile src/tools/dev_utils/shapefiles/Makefile internal/test_util/Makefile internal/test_util/basic/Makefile internal/test_util/basic/vx_config/Makefile internal/test_util/basic/vx_log/Makefile internal/test_util/basic/vx_util/Makefile internal/test_util/libcode/Makefile internal/test_util/libcode/vx_data2d/Makefile internal/test_util/libcode/vx_data2d_factory/Makefile internal/test_util/libcode/vx_data2d_grib/Makefile internal/test_util/libcode/vx_data2d_nc_met/Makefile internal/test_util/libcode/vx_data2d_nccf/Makefile internal/test_util/libcode/vx_geodesy/Makefile internal/test_util/libcode/vx_grid/Makefile internal/test_util/libcode/vx_plot_util/Makefile internal/test_util/libcode/vx_ps/Makefile internal/test_util/libcode/vx_tc_util/Makefile internal/test_util/libcode/vx_nc_util/Makefile internal/test_util/libcode/vx_python3_utils/Makefile internal/test_util/libcode/vx_physics/Makefile internal/test_util/libcode/vx_series_data/Makefile internal/test_util/libcode/vx_solar/Makefile internal/test_util/tools/Makefile internal/test_util/tools/other/Makefile internal/test_util/tools/other/mode_time_domain/Makefile"
ac_config_files="$ac_config_files src/tools/dev_utils/Makefile src/tools/dev_utils/shapefiles/Makefile internal/test_util/Makefile internal/test_util/basic/Makefile internal/test_util/basic/vx_config/Makefile internal/test_util/basic/vx_log/Makefile internal/test_util/basic/vx_util/Makefile internal/test_util/libcode/Makefile internal/test_util/libcode/vx_data2d/Makefile internal/test_util/libcode/vx_data2d_factory/Makefile internal/test_util/libcode/vx_data2d_grib/Makefile internal/test_util/libcode/vx_data2d_nc_met/Makefile internal/test_util/libcode/vx_data2d_nc_cf/Makefile internal/test_util/libcode/vx_geodesy/Makefile internal/test_util/libcode/vx_grid/Makefile internal/test_util/libcode/vx_plot_util/Makefile internal/test_util/libcode/vx_ps/Makefile internal/test_util/libcode/vx_tc_util/Makefile internal/test_util/libcode/vx_nc_util/Makefile internal/test_util/libcode/vx_python3_utils/Makefile internal/test_util/libcode/vx_physics/Makefile internal/test_util/libcode/vx_series_data/Makefile internal/test_util/libcode/vx_solar/Makefile internal/test_util/tools/Makefile internal/test_util/tools/other/Makefile internal/test_util/tools/other/mode_time_domain/Makefile"


fi

Expand Down Expand Up @@ -11065,8 +11066,8 @@ do
"src/libcode/vx_data2d_grib/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_grib/Makefile" ;;
"src/libcode/vx_data2d_grib2/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_grib2/Makefile" ;;
"src/libcode/vx_data2d_nc_met/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_nc_met/Makefile" ;;
"src/libcode/vx_data2d_nc_pinterp/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_nc_pinterp/Makefile" ;;
"src/libcode/vx_data2d_nccf/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_nccf/Makefile" ;;
"src/libcode/vx_data2d_nc_wrf/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_nc_wrf/Makefile" ;;
"src/libcode/vx_data2d_nc_cf/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_nc_cf/Makefile" ;;
"src/libcode/vx_data2d_ugrid/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_data2d_ugrid/Makefile" ;;
"src/libcode/vx_geodesy/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_geodesy/Makefile" ;;
"src/libcode/vx_gis/Makefile") CONFIG_FILES="$CONFIG_FILES src/libcode/vx_gis/Makefile" ;;
Expand Down Expand Up @@ -11147,7 +11148,7 @@ do
"internal/test_util/libcode/vx_data2d_factory/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_data2d_factory/Makefile" ;;
"internal/test_util/libcode/vx_data2d_grib/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_data2d_grib/Makefile" ;;
"internal/test_util/libcode/vx_data2d_nc_met/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_data2d_nc_met/Makefile" ;;
"internal/test_util/libcode/vx_data2d_nccf/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_data2d_nccf/Makefile" ;;
"internal/test_util/libcode/vx_data2d_nc_cf/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_data2d_nc_cf/Makefile" ;;
"internal/test_util/libcode/vx_geodesy/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_geodesy/Makefile" ;;
"internal/test_util/libcode/vx_grid/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_grid/Makefile" ;;
"internal/test_util/libcode/vx_plot_util/Makefile") CONFIG_FILES="$CONFIG_FILES internal/test_util/libcode/vx_plot_util/Makefile" ;;
Expand Down
6 changes: 3 additions & 3 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -1321,8 +1321,8 @@ AC_CONFIG_FILES([Makefile
src/libcode/vx_data2d_grib/Makefile
src/libcode/vx_data2d_grib2/Makefile
src/libcode/vx_data2d_nc_met/Makefile
src/libcode/vx_data2d_nc_pinterp/Makefile
src/libcode/vx_data2d_nccf/Makefile
src/libcode/vx_data2d_nc_wrf/Makefile
src/libcode/vx_data2d_nc_cf/Makefile
src/libcode/vx_data2d_ugrid/Makefile
src/libcode/vx_geodesy/Makefile
src/libcode/vx_gis/Makefile
Expand Down Expand Up @@ -1405,7 +1405,7 @@ if test -n "$MET_DEVELOPMENT"; then
internal/test_util/libcode/vx_data2d_factory/Makefile
internal/test_util/libcode/vx_data2d_grib/Makefile
internal/test_util/libcode/vx_data2d_nc_met/Makefile
internal/test_util/libcode/vx_data2d_nccf/Makefile
internal/test_util/libcode/vx_data2d_nc_cf/Makefile
internal/test_util/libcode/vx_geodesy/Makefile
internal/test_util/libcode/vx_grid/Makefile
internal/test_util/libcode/vx_plot_util/Makefile
Expand Down
11 changes: 6 additions & 5 deletions data/config/ConfigConstants
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,12 @@ MB_to_PA(p) = HPA_to_PA(p);
GRIB1 = 1;
GRIB2 = 2;
NETCDF_MET = 3;
NETCDF_PINT = 4;
NETCDF_NCCF = 5;
PYTHON_NUMPY = 6;
PYTHON_XARRAY = 7;
NETCDF_UGRID = 8;
NETCDF_WRF = 4;
NETCDF_PINT = 5;
NETCDF_NCCF = 6;
PYTHON_NUMPY = 7;
PYTHON_XARRAY = 8;
NETCDF_UGRID = 9;

// Set Logic
NONE = 1;
Expand Down
2 changes: 1 addition & 1 deletion docs/Users_Guide/appendixA.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1684,7 +1684,7 @@ Q. Why is the grid upside down?
but the data is packed upside down.

Try using the "file_type" entry. The "file_type" entry specifies the
input file type (e.g. GRIB1, GRIB2, NETCDF_MET, NETCDF_PINT, NETCDF_NCCF)
input file type (e.g. GRIB1, GRIB2, NETCDF_MET, NETCDF_WRF, NETCDF_PINT, NETCDF_NCCF)
rather than letting the code determine it itself. For valid file_type
values, see "File types" in the *data/config/ConfigConstants* file. This
entry should be defined within the "fcst" or "obs" dictionaries.
Expand Down
1 change: 1 addition & 0 deletions docs/Users_Guide/config_options.rst
Original file line number Diff line number Diff line change
Expand Up @@ -950,6 +950,7 @@ to be verified. This dictionary may include the following entries:
file_type = GRIB1; GRIB version 1
file_type = GRIB2; GRIB version 2
file_type = NETCDF_MET; NetCDF created by another MET tool
file_type = NETCDF_WRF; NetCDF WRF output.
file_type = NETCDF_PINT; NetCDF created by running the p_interp
or wrf_interp utility on WRF output.
May be used to read unstaggered raw WRF
Expand Down
Loading

0 comments on commit 464f9a7

Please sign in to comment.