Embed Size (px)
Citation preview
Mechanics of running HWRF
1
Ligia Bernardet
NOAA ESRL Global Systems Division, Boulder CO University of Colorado CIRES , Boulder CO
HWRF v3.5b Tutorial Taipei, Taiwan, May 22, 2014
Overview of this lecture
2
� Datasets and fix files needed for running HWRF � Location of source code and compiled executables � Location and overview of scripts � Tasks involved in a HWRF run � Scripts, functions and wrappers � Definition of global variables to be used by scripts
� Paths to input and output, initialization time, storm etc. � Namelists and tables � Location of output files � Submission of jobs in TTFRI machine
Location: /lfs/tutorial/HWRF/datasets
Datasets and fix files
3
Dataset Subdirectory File name
TC Vitals Tcvitals syndat_tcvitals.$YYYY
A decks abdecks a$basin$SID$YYYY.dat
B decks abdecks b$basin$SID$YYYY.dat
GFS anal & forec
GFS/gridded/$YYYY$MM$DD$HH gfs.$YYYY$MM$DD$HH.pgrbf$fff
GFS track GFS/gridded/$YYYY$MM$DD$HH avn.$YYYY$MM$DD$HH.cyclone.trackatcfunix
UPP fix fix/upp *
UPP coefficients
fix/upp/HWRFSpcCoeff fix/upp/HWRFTauCoeff
*
WPS fix wps_geog landuse_2m, topo_2m etc
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc
WPSV3 UPP hwrf-
utilities gfdl-
vortextracker
main bin trk_exec exec
real_nmm.exe wrf.exe
geogrid.exe ungrib.exe metgrid.exe
copygb.exe ndate.exe unipost.exe
diffwrf_3dvar.exe grbindex.exe hwrf_anl_4x_step2.exe hwrf_anl_bogus_10m.exe hwrf_anl_cs_10m.exe hwrf_bin_io.exe hwrf_create_nest_1x_10m.exe hwrf_create_trak_fnl.exe hwrf_create_trak_guess.exe hwrf_data_flag.exe hwrf_inter_2to1.exe hwrf_inter_2to2.exe hwrf_inter_2to6.exe hwrf_inter_4to2.exe hwrf_inter_4to6.exe hwrf_merge_nest_4x_step12_3n.exe hwrf_pert_ct1.exe hwrf_prep.exe hwrf_readtdrstmid.exe hwrf_readtdrtime.exe hwrf_split1.exe hwrf_swcorner_dynamic.exe hwrf_wrfout_newtime.exe wgrib.exe
hwrf_gettrk.exe hwrf_tave.exe hwrf_vint.exe
WPSV3
Structure of Compiled source
4
WRFV3_Idealized
main
ideal.exe wrf.exe
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc/hwrf-utilities
wrappers_scripts
Scripts, Wrappers, and Functions
5
hwrf-utilities
scripts
funcs
The scripts (ksh93) to run HWRF are located in hwrf-utilities/scripts
The scripts need to receive global variables with information about initialization date, which storm to run etc. Those will be passed by the wrappers
The scripts use a library of functions
A generic automation system could be use to drive these script (automation not covered in this tutorial)
Example of a wrapper: hwrf_wrapper
6
#!/usr/bin/env ksh93!!#----------------------------------------------------# Global definitions of environment variables!#----------------------------------------------------. PATH_TO_GLOBAL/global_vars.ksh!#----------------------------------------------------# Local definitions of environment variables!#----------------------------------------------------!export WRF_MODE=main!export WRF_CORES=${HWRF_COUPLED_CORES}!!# export DEBUG=1!${HWRF_SCRIPTS}/wrf.ksh!
Every wrapper is a ksh93 script that • Sets global variables (e.g., storm, initialization time, dir for input/output) • Sets local variables, which are specific to the script that will be called • Calls a script
# Main!function main {! typeset errno! # Init the function library! . ${HWRF_SCRIPTS}/funcs/init!set -e! # Check to see if all the variables are set! check_vars DOMAIN_DATA START_TIME!! # Create a working directory (and cd into it)! create_work_dir ${DOMAIN_DATA}/messages!! # Create the tcvital and tcvitals.as files! lat_lon=( $( tcvitals ) )!! # Create the storm centre file! storm_centre ${lat_lon[@]}!
Example of a script: hwrfdomain.ksh
7
Functions used only once • are located in script that uses them • storm_centre
Functions used more than once • are located in a library of functions • check_vars and create_work_dir
Tasks, wrappers, and scripts I
8
Task Wrapper Script
Determine domain location based on TC Vitals hwrfdomain_wrapper hwrfdomain.ksh
Run WPS geogrid geogrid_wrapper geogrid.ksh
Run WPS ungrib ungrib_wrapper ungrib.ksh
Run WPS metgrid metgrid_wrapper metgrid.ksh
Run real real_wrapper real.ksh
Run WRF Analysis (90s) wrfanalysis_wrapper wrf.ksh
Run WRF Ghost (90s) wrfghost_wrapper wrf.ksh
Run postprocessor and tracker on WRF Analysis output to find location of vortex in GFS
track_analysis_wrapper track_analysis.ksh
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc/hwrf-utilities
Tasks, wrappers, and scripts II
9
Task Wrapper Script
Run the vortex initialization
relocate1_wrapper relocate_stage1_3d.ksh
relocate2_wrapper relocate_stage2_3d.ksh
relocate3_wrapper relocate_stage3_3d.ksh
Merge information on various domains to create IC for WRF
merge_wrapper merge.ksh
Run main forecast hwrf_wrapper wrf.ksh
Run postprocessor unipost_wrapper unipost.ksh
Run external vortex tracker tracker_wrappe tracker.ksh
Create plots rungrads_wrapper run_grads
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc/hwrf-utilities
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc/hwrf-utilities/wrapper_scripts/global_vars.ksh
Defining global variables for scripts I
10
Soulik 07W 2013
START_TIME 2013070912
Initialization time
START_TIME_MINUS6 2012102806 Initialization time minus 6 hr
FCST_LENGTH 12 Length of forecast (in hrs)
FCST_INTERVAL 6 Time between consecutive HWRF cycles
STORM_NAME SOULIK Storm name
SID 07W Storm ID L=Atl; E=E Pac; W=W Pac
BASIN WP Basin AL, EP, WP, IO
Wrappers source global_vars.ksh to pass global variables to scripts
Defining global variables for scripts II
11
Soulik 07W 2013
RUN_PREP_HYB F T= Use global model in binary format; F=Use global model in GRIB format
UPP_PROD_SAT T T= UPP outputs synthetic satellite fields
BKG_MODE GFS Select GFS or GDAS for initial conditions
RUN_GSI F
Disable data assimilation RUN_GSI_WRFINPUT F
RUN_GSI_WRFGHOST F
INNER_CORE_DA 0
T-Run GSI, F-No GSI*
Full description of global_vars.ksh can be found in Appendix of HWRF v3.5a Users’ Guide http://www.dtcenter.org/HurrWRF/users/docs/users_guide/HWRF_v3.5a_Users_Guide.pdf
Defining global variables for scripts III
12
Soulik 07W 2013
HWRF_SRC_DIR /lfs/tutorial/$USER/HWRF_v3.5b/sorc Path to HWRF source code
HWRF_DATA_DIR /lfs/tutorial/HWRF/datasets Path to HWRF input datasets
CYCLE_DATA ${HWRF_OUTPUT_DIR}/${SID}/${START_TIME_MINUS6}
Path to previous cycle output
DOMAIN_DATA ${HWRF_OUTPUT_DIR}/${SID}/${START_TIME}
Path to current cycle output
GFS_DIR ${HWRF_DATA_DIR}/GFS Path to toplevel GFS input
GFS_GRIDDED_DIR ${GFS_DIR}/gridded
Path to GFS gridded data
TCVITALS ${HWRF_DATA_DIR}/Tcvitals Path to TC Vitals files
GEOG_DATA_PATH ${HWRF_DATA_DIR}/wps_geog Path to geographical fixdata
GRADS_BIN /lfs/tutorial/HWRF/bin Path to GrADS binaries
GADDIR /lfs/tutorial/HWRF/grads/data Path to grads tables
Defining global variables for scripts IV
13
Soulik 07W 2013
MPIRUN mpiexec Command to run parallel code
WRF_ANAL_CORES 12 Number of cores to run WRF Analysis
WRF_GHOST_CORES 12 Number of cores to run WRF Ghost
HWRF_FCST_CORES 202 Number of cores to run HWRF forecast
GEOGRID_CORES 12 Number of cores to run geogrid
METGRID_CORES 12 Number of cores to run metgrid
REAL_CORES 1 Number of cores to run real
UNI_CORES 12 Number of cores to run UPP
ATCFNAME HCOM ATCF identifier for HWRF forecast
Namelists and tables
14
hwrf_cntrl.* Namelists for UPP
hwrf_namelist.input Template namelist for WRF runs (ghost, analysis, main)
hwrf_namelist.wps Template namelist for WPS (geogrid, ungrib, metgrid)
hwrf_co2_trans hwrf_eta_micro_lookup.dat hwrf_ETAMPNEW_DATA hwrf_ETAMPNEW_DATA_DBL hwrf_GENPARM.TBL hwrf_LANDUSE.TBL hwrf_RRTM_DATA hwrf_RRTM_DATA_DBL hwrf_SOILPARM.TBL hwrf_tr49t67 hwrf_tr49t85 hwrf_tr67t85 hwrf_VEGPARM.TBL
Tables used for WRF runs
hwrf_grib_reduce.parms List of variables used to thin the input GFS file toexpedite processing
hwrf_storm_* Tables used for vortex relocation
Location: /lfs/tutorial/$USER/HWRF_v3.5b/sorc/hwrf-utilities/parm
WRF namelist template
15
start_year = @[start:0:4], @[start:0:4], @[start:0:4],!
start_month = @[start:4:2], @[start:4:2], @[start:4:2],!
start_day = @[start:6:2], @[start:6:2], @[start:6:2],!
!
num_soil_layers = 4,!
mp_physics = 85, 85, 85,!
ra_lw_physics = 98, 98, 98,!
ra_sw_physics = 98, 98, 98,!
� Some variables are directly defined in template namelist
� Other variables (with @ in front) are populated when scripts run
Scripts � Read hwrf-utilities/parm/hwrf_namelist.input � Modify namelist according to global variables defined � Write namelist in working directory
Output directory structure
16
Subdirectories Creator script
messages hwrfdomain.ksh
geoprd geogrid.ksh
ungribprd/$YYYYMMDDHH ungrib.ksh
metgridprd/$YYYYMMDDHH metgrid.ksh
realprd_GFS real.ksh
wrfghostprd/$YYYYMMDDHH wrf.ksh run in Ghost mode
wrfanalysisprd/$YYYYMMDDHH wrf.ksh run in Analysis mode
trkanalysisprd/$YYYYMMDDHH track_analysis.ksh
relocateprd/$YYYYMMDDHH relocate*.ksh
mergeprd merge.ksh
wrfprd wrf.ksh in main mode
postprd run_unipost
gvtprd tracker.ksh
Location: /lfs/tutorial/${USER}/HWRF_v3.5b/results/$SID/$YYYYMMDDHH
Some subdirectories have a date string on them. For this tutorial, it is the same as initialization time.
This is done because for some data assimilation configurations it is necessary to process data for other valid time as well
What you learned in this lecture
17
• The source code has many components • WPS, WRF, UPP, hwrf-utilities, GFDL vortex tracker
• Several tasks are needed to do an end-to-end HWRF run • Hwrf-utilities scripts, functions, wrappers are used to run HWRF • Wrappers • Source global_vars.ksh (paths, initialization time, storm etc.) • Define local variables • Call scripts
• File structure for input and output files • Next: how to submit a job in TTFRI machine
How to log in to the TTFRI computer
18
� Open 2 login windows on your laptop: editing and running � Linux: select open terminal from the right mouse button menu. � OS X: start the Terminal and XQuartz applications from the /Applications/Utilities
folder. � Windows: start the application XSession from the Start menu and then Putty
� Open an X-enabled window to a login node � Log in to the supercomputer : ssh -X [email protected]!� At the prompt, enter your temporary password
Practical session instructions http://www.dtcenter.org/HurrWRF/users/tutorial/2014_Taiwan_tutorial/practical/index.htm
How to submit wrapper scripts
19
� Define your wrapper directory setenv WRAPPER_DIR /lfs/tutorial/$USER//HWRF_v3.5b/hwrf-utilities/wrapper_scripts!or !!
export WRAPPER_DIR =/lfs/tutorial/$USER/HWRF_v3.5b/hwrf-utilities/wrapper_scripts!
� Set the options for the job bsub_options='-l procs=16,walltime=6:00:00,vmem=30G -N hwrf -o hwrf.out -e hwrf.err -S /bin/bash -k oe -z’!
� Submit the job bsub $bsub_options ${WRAPPER_DIR}/geogrid_wrapper!
Descriptions of batch system options � -l resources needed to run script � -N job name � -o standard out file name � -e standard err name � -S use bash � -k keep standard error and output � -Z suppresses the job ID from being written to standard output
Thank you for your interest!
20
For more information , you can… � Ask questions during the tutorial
� Contact me later: [email protected]
� Reach our user helpdesk: [email protected]
� Visit our website: http://www.dtcenter.org/HurrWRF/users
� HWRF v3.5a Users’ Guide (http://www.dtcenter.org/HurrWRF/users/docs/users_guide/HWRF_v3.5a_Users_Guide.pdf)
� HWRF v3.5a Scientific Documentation (http://www.dtcenter.org/HurrWRF/users/docs/
scientific_documents/HWRFv3.5a_ScientificDoc.pdf)
� WRF-NMM V3 Users Guide (http://www.dtcenter.org/HurrWRF/users/docs/users_guide/
users_guide_nmm_chap1-7upp.pdf)
