OpenWQ

Model Calibration

A comprehensive guide to calibrating OpenWQ models
— sensitivity analysis, optimization algorithms, and best practices

Calibration Framework Overview

OpenWQ includes a comprehensive calibration framework with:

106+ calibratable parameters
DDS optimization algorithm
Morris screening for sensitivity
Sobol analysis for detailed SA
Docker/Apptainer deployment
Checkpoint/restart capability

┌─────────────────────────────────────┐
│     Calibration Workflow            │
├─────────────────────────────────────┤
│                                     │
│  ┌─────────────────────────────┐    │
│  │ 1. Morris Screening (~200)  │    │
│  │    → Identify sensitive     │    │
│  └──────────────┬──────────────┘    │
│                 ▼                   │
│  ┌─────────────────────────────┐    │
│  │ 2. Sobol Analysis (~1000)   │    │
│  │    → Rank top parameters    │    │
│  └──────────────┬──────────────┘    │
│                 ▼                   │
│  ┌─────────────────────────────┐    │
│  │ 3. DDS Optimization (~300)  │    │
│  │    → Optimize top 10-15     │    │
│  └──────────────┬──────────────┘    │
│                 ▼                   │
│  ┌─────────────────────────────┐    │
│  │ 4. Validation               │    │
│  │    → Independent period     │    │
│  └─────────────────────────────┘    │
│                                     │
└─────────────────────────────────────┘

Parameter Categories (106+ total)

Category	Count	Examples	Typical Range
BGC (NATIVE_BGC_FLEX)	26	k_nitrification, k_denitrification, theta	0.001 - 1.0 /day
PHREEQC	22	Initial concentrations, pCO2, SI	Varies
Sorption	16	Kfr, qmax, KL, bulk_density	0.01 - 100 L/kg
Sediment Transport	14	erosion_index, erodibility, cohesion	0.1 - 10
Transport	5	dispersion_x/y/z, characteristic_length	0.1 - 100 m²/s
Lateral Exchange	4	k_exchange (river-soil, soil-GW)	0.0001 - 0.1 /s
Source/Sink	21	Load scaling, export coefficients	0.1 - 5.0×

                    Key Insight: Source/sink parameters are typically THE MOST SENSITIVE — start calibration there!
                

01

Sensitivity Analysis

Morris Screening (Elementary Effects)

Efficient method to identify influential parameters with minimal model runs.

Cost: O(r × (k+1))

r = 10-20 trajectories, k = parameters

Example: 30 params × 15 trajectories = ~465 runs

Metrics

μ* (mu-star)	Mean absolute effect → Overall influence
σ (sigma)	Standard deviation → Non-linearity

Interpretation

μ*	σ	Meaning
High	Low	Linear influence → Calibrate
High	High	Non-linear → Calibrate carefully
Low	Low	Non-influential → Fix at default
Low	High	Interactive → Investigate

Typical result: Reduces 100+ params to 30-40 for detailed analysis

Sobol Sensitivity Analysis

Variance-based method for detailed sensitivity ranking.

Cost: O(N × (2k+2))

N = 1000-5000 samples, k = parameters

Example: 30 params × 2000 samples = ~124,000 runs

Indices

S1	First-order: Direct parameter effect
ST	Total-order: Including interactions

Interpretation

S1 ≈ ST: Parameter acts independently
ST >> S1: Strong interactions
Sum(S1) < 1: Significant interactions exist

                            Use after Morris: Apply Sobol only to the ~30 parameters identified as potentially influential
                        

02

DDS Optimization

Dynamically Dimensioned Search (DDS)

Derivative-free optimization designed for expensive models.

Key Features

Reduces search dimensionality over time
Early: Explore many parameters
Late: Focus on few promising ones
No gradient computation needed

Typical budget: 200-500 evaluations for 10-15 parameters

   DDS Behavior Over Iterations

   Early (exploration)
   ┌──────────────────────────────┐
   │ ████████████████████████████ │ Many params
   │ ██████████████████           │ perturbed
   │ ████████████                 │
   └──────────────────────────────┘

   Late (exploitation)
   ┌──────────────────────────────┐
   │ ███                          │ Few params
   │ ██                           │ refined
   │ █                            │
   └──────────────────────────────┘

   Probability of perturbation:
   P(i) = 1 - ln(i)/ln(max_iter)

Objective Functions

RMSE

√(Σ(obs-sim)²/n)

Root Mean Square Error

0 → ∞ (lower better)

NSE

1-Σ(o-s)²/Σ(o-ō)²

Nash-Sutcliffe Efficiency

-∞ → 1 (1 perfect)

KGE

1-√((r-1)²+(α-1)²+(β-1)²)

Kling-Gupta Efficiency

Recommended!

PBIAS

100×Σ(s-o)/Σ(o)

Percent Bias

-∞ → +∞ (0 perfect)

                    KGE advantages: Balances correlation (r), variability ratio (α), and bias ratio (β) — better for multi-species calibration
                

Temporal Resolution

Aggregate observations and model outputs to a common temporal scale before computing objective functions.

Available Resolutions

Option	Use Case
native	Original timestamps (no aggregation)
daily	Daily patterns matter
weekly	Smooth day-to-day noise
monthly	Seasonal patterns, sparse obs
yearly	Long-term trends, budgets

Aggregation Methods

mean (default) | sum | median | min | max

Use mean for concentrations, sum for loads/fluxes

# Configuration in calibration file
temporal_resolution = "monthly"
aggregation_method = "mean"

# How it works:
# 1. Group obs by reach, species, month
# 2. Extract model outputs for same periods
# 3. Aggregate both to monthly means
# 4. Compute KGE/NSE/RMSE on aggregated data

                            Tip: Use monthly for sparse grab samples — ensures fair comparison with continuous model output
                        

                    Output: Performance plots generated at specified resolution (time series, scatter, residuals)
                

03

Calibration Workflow

Step-by-Step Workflow

1

Configure the model

Set up & validate your model with the 1_Model_Config report (paths, modules, species, observation data).

2

Open the interactive calibration setup report

Run calibration_config_template.py → a browser report opens. Use the tabs to set Targets (species, reaches/HRUs, one compartment), Parameters, the workflow mode (influential parameters · both · calibration) and the calibration/validation period (slider shows per-species observation coverage).

3

Pick how to run & save the script

In the Execution tab choose Docker (local) or Apptainer (HPC) + parallel runs, then click “Save the script”.

4

Run it — locally or on HPC

Locally with Docker, or on HPC with the auto-filled SLURM job + one-paste copy & run commands. The chosen mode runs Morris/Sobol screening and/or DDS calibration.

5

Results & validation

The interactive results report opens automatically — review the fit, apply the best parameters, and validate on the held-out window.

Observation Data Sources

Two options for preparing observation data:

📄 Option 1: Manual CSV

observation_data_source = "csv"

Prepare data manually
Full control over format
Use any data source (USGS, EPA, local monitoring, etc.)

Best for: Custom datasets, local monitoring networks

🌍 Option 2: GRQA Database

observation_data_source = "grqa"

43 water quality parameters
~100 million observations worldwide
Auto station-to-reach matching
Local data or Zenodo download

Best for: Large-scale studies, data-rich regions

                    Local Data Support: GRQA supports pointing to local data folders if already downloaded from Zenodo
                

GRQA Database Integration

Global River Water Quality Archive

43 water quality parameters
~100 million observations worldwide
Automatic Zenodo download
Spatial station-reach matching

grqa_config = {
    # Local or download from Zenodo
    "local_data_path": "/data/GRQA",
    "river_network_shapefile": "rivers.shp",
    "max_station_distance_m": 500,
    "species_mapping": {
        "NO3": "NO3-N",
        "NH4": "NH4-N"
    }
}

Species Mapping (GRQA → Model)

GRQA	Model Species
NO3	NO3-N
NH4	NH4-N
TN	TN
PO4	PO4-P
TP	TP
TSS	TSS
DOC	DOC

                            Automatic: GRQA stations are extracted, matched to
                            reaches/HRUs and clipped on the first run, then reused — the
                            >1 GB raw database is never re-downloaded or re-processed.
                        

Observation Data CSV Format

# observations.csv (all sources produce this format)
datetime,reach_id,species,value,units,source,uncertainty,quality_flag
2018-01-15 10:00:00,1200014181,NO3-N,2.50,mg/l,USGS_station_A,0.25,GOOD
2018-01-15 10:00:00,1200014181,NH4-N,0.15,mg/l,USGS_station_A,0.02,GOOD
2018-02-01 10:00:00,1200014181,NO3-N,3.10,mg/l,USGS_station_A,0.31,GOOD
2018-02-01 10:00:00,1200014181,TP,0.08,mg/l,USGS_station_A,0.01,GOOD
...

Required Columns

datetime	YYYY-MM-DD HH:MM:SS
reach_id	Matching model output
species	Case-sensitive name
value	Measured concentration
units	mg/l, ug/l, etc.

Optional Columns

source	Data provider ID
uncertainty	Measurement error
quality_flag	GOOD, SUSPECT, BAD

Priority-Based Calibration

When resources are limited, focus on the most influential parameters:

TIER 1 Must Calibrate

Source/sink scaling factors
k_nitrification
k_denitrification
Kfr_PO4 or qmax_PO4
dispersion_x

5-8 parameters

TIER 2 Important

Temperature coefficients (θ)
k_mineralization
k_P_adsorption
erosion_index
Secondary sorption params

Next 5-8 parameters

TIER 3 Refinement

Half-saturation constants
Lateral exchange rates
Volatilization rates
Langmuir exponents

Remaining parameters

Running Calibration

Configure in the setup report, click “Save the script”, then run the generated <template>_run.py. The workflow mode (influential parameters · both · calibration) is set by the slider in the report and baked into the script — there is no mode flag.

# Run locally with Docker:
cd .../containers && docker compose up -d  # start container
cd .../3_Calibration                       # where calibration_lib lives
python <template>_run.py            # run (mode set in the report)
python <template>_run.py --resume   # resume from last checkpoint
python <template>_run.py --clean    # delete stale eval_* first
python <template>_run.py --dry-run  # validate config, no run
python <template>_run.py --report   # (re)build the results report

                            Keep it awake: long runs are auto-prefixed with
                            caffeinate (macOS) / systemd-inhibit (Linux)
                            so the machine doesn't sleep mid-run. Progress is reported by
                            simulated date through the calibration window.
                        

Run modes (set in the report)

Influential params	Morris / Sobol screening only
Both	Screening → DDS calibration
Calibration	DDS optimisation only

CLI flags: --resume, --clean, --keep, --dry-run, --report.

On HPC (Apptainer)

Set Container runtime = apptainer in the Execution tab. The report builds a ready-to-run SLURM .sbatch and one-paste copy → re-point → submit commands (HPC details prefilled from hpc_settings.json) — nothing to edit.

Calibration Output

Output Files

`best_parameters.json`	Optimal values
`calibration_history.json`	All evaluations
`parameter_definitions.json`	Parameter metadata & bounds
`matched_data.csv`	Obs-model matched pairs
`<template>_config_report.html`	Interactive setup / config report
`<template>_results_report.html`	Interactive results report (maps, convergence, performance)
`sensitivity_results.json`	SA results (if run)

                            Report naming: reports are named after the calibration
                            template — e.g. model_config_template_mizuRoute_results_report.html
                            — so each host-model variant keeps its own clearly-labelled outputs.
                        

Runtime Estimates

Model Runtime	300 Evals
5 min	~25 hours
15 min	~75 hours
30 min	~150 hours

                            HPC: the setup report builds a ready-to-run SLURM
                            .sbatch + one-paste copy & submit commands (Apptainer);
                            set parallel runs in the Execution tab. Restrict each eval to the
                            calibration window (period slider) to keep runtime & memory low.
                        

Best Practices

✅ Do

Start with Morris screening to reduce parameters
Use log transform for rate constants
Split data: calibration + validation periods
Use KGE for multi-species calibration
Document parameter choices
Check physical plausibility of results

❌ Don't

Calibrate all 100+ parameters at once
Ignore parameter correlations
Use entire dataset for calibration
Accept physically unrealistic values
Skip sensitivity analysis
Overfit to noisy observations

Golden Rule: Fewer well-chosen parameters beats many poorly constrained ones

05

Post-Calibration Reporting

Interactive HTML Reports

Two self-contained, interactive HTML reports drive the workflow — a setup report (configure & launch) and a results report (diagnose the fit).

Setup report (configure & run)

Tabs: Overview · Settings · Targets · Parameters · Execution
Workflow-mode slider (influential params · both · calibration)
Calibration/validation slider with per-species observation coverage lanes
Live-updating Python script + “Save the script”
Flat step-by-step run locally / on HPC guide
HPC details prefilled from hpc_settings.json

                            Self-contained: single HTML file with embedded data — share via email, no server needed
                        

Results report (diagnose)

Plotly charts	Convergence, parameter evolution, correlation, Morris/Sobol, obs-vs-sim + scatter
Observation map	Stations coloured by performance
Per-figure export	Lock-zoom + 📄 PDF (white, branded)
Per-species metrics	KGE / NSE / RMSE / PBIAS
Next Steps	Apply best params · validate · refine
Dark/Light	Theme toggle · sortable tables

Auto-Generated

Both are named after the calibration template (<template>_config_report.html / <template>_results_report.html); the results report opens automatically and can be re-built mid-run with --report.

                    Host-model observation semantics:
                    for mizuRoute every matched station is primary (scored);
                    for SUMMA only the pouring-point (HRU-outlet) station feeds the
                    metric by default (use_primary_only=True) — untick
                    “use only primary” to score the secondary stations too.
                

Interactive Basin Maps

Calibration reports include interactive Leaflet.js maps showing the basin spatial context.

Map Layers

HRU polygons — colored by area quintiles
River network — styled by Strahler order
Observation stations — markers colored by the objective metric; mizuRoute matches to the nearest reach, SUMMA to the HRU pouring point

Basemaps

3 selectable basemaps: CARTO Light, OpenTopoMap, Esri Satellite

Data source: GeoPackage files (*_basinHru.gpkg, *_riverNetwork.gpkg)

Basin Info Grid

Total HRUs	Count from GeoPackage
River reaches	Network segments
Total area	km² from HRU polygons
Network length	km of river network
Max Strahler order	Stream hierarchy
Obs stations	Matched monitoring sites

Interactive Controls

Layer toggle, zoom, scale bar, legend, click-for-info popups

Per-Basin Multi-Variant Reports

When running multiple variants (A/B/C/D), a consolidated basin report is auto-generated comparing all variants.

Basin Report Contents

Overview KPIs — best KGE per variant
Basin map — shared spatial context
Variant comparison table — side-by-side metrics
Detail cards — per-variant parameter tables
Links — to detailed per-variant reports

                            Auto-discovery: Sibling workspaces (workspace_{basin}_{variant}) are detected automatically
                        

Post-Calibration Pipeline

Calibration completes
        │
        ▼
parameter_definitions.json  ← saved
matched_data.csv            ← saved
        │
        ▼
<template>_results_report.html  ← per-variant
        │
        ▼
Detect sibling workspaces
        │
        ▼
basin_report.html             ← multi-variant

Summary

🔍 Screen

Morris screening to identify influential parameters (106 → 30)

📊 Rank

Sobol analysis for detailed sensitivity ranking (30 → 10-15)

🎯 Optimize

DDS calibration of top parameters (200-500 evaluations)

📋 Report

Auto-generated interactive HTML reports with maps & charts

                    Key Message: Hierarchical approach (Screen → Rank → Optimize → Report) with

                    automated post-calibration diagnostics for every basin and variant

Thank You

Questions?

Calibration scripts: supporting_scripts/Calibration/

Observation data: CSV | GRQA Database | Reports: HTML with interactive maps & charts