Execute pre-calibration scripts

Before to perform the actual calibration, you need to run the following scripts, in the order as they are presented.

Also, ensure that all paths to folders and files defined in settings are correct and existing.

1. CAL_1_CAL_VAL_PERIODS.py

python CAL_1_CAL_VAL_PERIODS.py settings.txt

The 1rst script reads the settings (settings.txt in this case) and the Qgis and Qtss specifed in settings.txt. Computes calibration and validation periods based on the available streamflow data. Catchments with streamflow records shorter than the number of years specifed by MinQlength in settings.txt will not be processed. If the record length is twice MinQlength, the available streamflow record is split into equally long validation and calibration parts. If the record length is less than twice MinQlength, MinQlength is used for calibration and the remaining part for validation. In all cases, the first part of the record is used for validation and the second part for calibration. If necessary, the calibration and validation periods can be changed manually by editing Qgis2.csv. However, be sure to use the correct date format (DD/MM/YYYY).

2. CAL_2_PREP_MISC_REMOVE_SMALL_CATCH.py

python CAL_2_PREP_MISC_REMOVE_SMALL_CATCH.py settings.txt

This script reads the settings file (settings.txt in this case) and subsequently the Qgis and Qtss files specified in settings.txt. Stations from the Qgis list are first associated to the corresponding model river network, to identify the upstream/downstream relation among stations lying in the same river basin. Then, if the parameter MaxPercArea is set to a number larger than 0, the script eliminates from the calibration list all stations lying within MaxPercArea times the upstream area of each river station. If MaxPercArea=0.1 (default value), all river stations with upstream area up to 10% larger than that of the upstream station will be removed. For example, if 5 stations are available along the same river, with upstream area respectively of A1=100, A2=104, A3=107, A4=111, A5=118 km^2, the script will remove the 2nd, 3rd and the 5th station, leaving the 1st and the 4th in the calibration list. The idea behind this is to avoid calibrating clusters of stations, where the downstream ones bring little benefit and often assumes anomaluos calibrated parameter values, as they are heavily affected by the simulated inflow of the upstream station.

3. CAL_3_PREP_MISC.py

python CAL_3_PREP_MISC.py settings.txt

The second script reads the settings file (settings.txt in this case) and subsequently the Qgis and Qtss files specified in settings.txt. Computes maps with station locations (outlet.map), interstation regions (interstation regions.map),sampling frequency (sampling frequency.map), and in ow locations (inlets.map). In addition, generates a CSV file with direct station linkages (direct links.csv), and a CSV file listing the catchment area (in number of pixels) and the numeric portion of the catchment identifier (Qgis2.csv).

4. CAL_4_SPREAD_WORKLOAD.py

python CAL_4_SPREAD_WORKLOAD.py settings.txt

This script subdivides the modeling domain to allow running the calibration in parallel. Considered the time needed for running the calibration, we opted for spreading the total amount of catchments on multiple lists that can be used to spread the run of the calibration over several medium (PCs , nodes on grid engine, single PC). Each list is independent from the others, this let you decide where to run each list.

5. CAL_5_PREP_FORCING.py

python CAL_5_PREP_FORCING.py settings.txt CatchmentsToProcess_XX.txt

This script should be run on each PC, so the entire directory should be copied to each PC. The script reads the settings file (settings.txt in this case) and the Qgis2.csv file that was generated by the previous script. Loops through the stations in CatchmentsToProcess X.txt, makes directories for each station, and produces mask and station maps subsetted to the interstation regions of the respective catchments.

6. CAL_6_CUTMAPS_LAUNCH.py

python CAL_6_CUTMAPS_LAUNCH.py settings.txt CatchmentsToProcess_XX.txt

7. CAL_7_PERFORM_CAL.py

python CAL_7_PERFORM_CAL.py settings.txt CatchmentsToProcess_XX.txt

This script should be run on each PC. It reads the settings file (settings.txt in this case), the direct links.csv and Qgis2.csv files, and the Qtss file specified in settings.txt. Loops through the stations listed in CatchmentsToProcess X.txt (in ascending order of catchment area), creates an inflow.tss file if there are directly linked upstream stations, and runs the script cal single objfun.py settings.txt x” (use if there is a single objective function) or cal multiple objfun.py settings.txt x” (use if there are multiple objective functions), where x denotes the numeric portion of the station identifier as listed in Qgis2.csv. The cal_single_objfun.py script performs the calibration for each station using the Non-dominated Sorting Genetic Algorithm-II (NSGA-II) multi-objective genetic algorithm (Deb et al., 2002). Any objective function can be used by modifying the scripts. The LISFLOOD runs are distributed to the available cores if use multiprocessing is set to one in the settings.txt file. Outputs for each catchment are CSV files with objective function scores of each LISFLOOD run (runs log.csv), objective function statistics for the Pareto optimal solutions of each generation (front history.csv), final Pareto optimal solutions and corresponding model parameter values (pareto front.csv), and streamflow time series of the \best” run (streamflow simulated best.csv and streamflow simulated best.tss).

CAL_8_CLEAN_SUBCATCH_DIRS.py

python CAL_8_CLEAN_SUBCATCH_DIRS.py settings.txt CatchmentsToProcess_XX.txt

This script should be run on each PC. Reads the settings file (settings.txt in this case), loops through the catchments, and deletes unnecessary files created with each run (e.g., prerun and run XML files and lzavin maps). After this script the files from each PC should be copied to a single PC on which you run the following scripts.

Calibration tool