Abstract

Evaluating joint predictive performance for multivariate hydroclimatic models requires metrics that simultaneously assess marginal accuracy and cross-variable dependence recovery. Existing metricsthe Energy Score, Variogram Score, and their derivativesdo not adapt to the structural complexity of the residual correlation matrix, treating a single correlated pair identically to a fully dense dependence structure. We propose two novel metric families: Metric E (E-CVWMD: Enhanced Coefficient-of-Variation Weighted Marginal-Dependence) and Metric E2 (E-CVWMD-Pairwise), which are designed for mixed-type multivariate responses combining continuous and binary outcomes within a cross-validation framework. We position Metrics E and E2 as diagnostic ranking tools for comparing competing models rather than as strictly proper scoring rules, and we provide a strictly proper Log-Loss variant (E-LL/E2-LL) for applications that require the full properness guarantee. Metric E assigns variable-level weights proportional to the coefficient of variation (CV) of each outcome on the training partition and adaptively calibrates the marginal-dependence trade-off parameter α* via a global distance-correlation test. Metric E2 refines this by replacing the global test with a pairwise Spearman screening index π^, the proportion of variable pairs with significant residual correlationwhich maps linearly to α*(π^)=1−π^/2∈[0.5,1]. Applied to the validation of a Generalized Multivariate Functional Additive Mixed Model (GMFAMM) on 62 Valle del Cauca meteorological stations (Ntest≈ 31,663), the naive significance-based index saturates (π^=1.0) at this large sample sizeevery pair, including correlations as small as |ρ^s|≈0.01, is flagged “significant”which is precisely the sample-size sensitivity we address. Under the effect-size screening (|ρ^s|≥0.05), three negligibly correlated pairs are excluded, yielding π^=0.70 and αE2*=0.65, a better-calibrated weight than Metric E’s αE*≈0.797 under the same data. A large-scale simulation study with 37,440 model evaluations confirms that Metric E inverts the correct ranking at correlation levels ρ≥0.40 (CDR = 0%), while E2 maintains correct discrimination in 14 of 15 simulation conditions (M1 vs. M3). We also delimit the metrics’ scope: E2 degrades under near-saturated uniform dependencea regime in which the strictly proper Energy Score remains preferableand the pairwise index is sensitive to sample size, for which we provide an effect-size-based variant. An R package (mvmetrics v0.2.0) implementing both metrics, the Log-Loss variant, alternative weighting schemes, and the effect-size screening is publicly available.

IPC Classification

Keywords