How this statistics score is computed (sub-scores)
How this dynamical consistency score is computed
Setup. An ensemble Kalman smoother runs the model's own step: 500 members, 3 month lag, observation error 5 percent of each variable's scale, 1980 to 2014 with the first 24 months dropped. Case A observes \(T_C, T_E\) and reconstructs \(h_W, u, \tau\); case B observes \(h_W, u, \tau\) and reconstructs \(T_C, T_E\).
Per variable. \(\hat x_v\) is the reconstruction (ensemble mean) of an unobserved variable, \(x_v\) its observed truth:
\[ \mathrm{NSE}_v = \max\!\Big(0,\ 1 - \frac{\overline{(\hat x_v - x_v)^2}}{\operatorname{var}(x_v)}\Big) \]This equals \(r^2\) when the amplitude is right and falls below it when the reconstruction is too weak or too strong.
Total. The mean over the five reconstructed variables:
\[ S_{\mathrm{dyn}} = \tfrac15\Big(\mathrm{NSE}_{h_W} + \mathrm{NSE}_{u} + \mathrm{NSE}_{\tau} + \mathrm{NSE}_{T_C} + \mathrm{NSE}_{T_E}\Big) \]How this predictivity score is computed
Setup. Every month of 2015 to 2024 is a forecast origin (108 in all). The model starts from the observed state in the right calendar month, runs a 50 member ensemble for leads \(L = 1\) to \(12\) months, and its ensemble mean \(F\) is verified against the observed \(T_C\) and \(T_E\). The baseline is persistence \(P\): the anomaly at the origin held fixed.
Per variable. With \(x\) the observed value at the verifying month, averaged over origins:
\[ \mathrm{NRMSE}_v(L) = \frac{\sqrt{\overline{(F_{v,L} - x_{v,L})^2}}}{\operatorname{std}(x_v)}, \qquad \mathrm{skill}_v = \max\!\Big(0,\ 1 - \frac{\overline{\mathrm{NRMSE}_v(L)}^{\,L}}{\overline{\mathrm{NRMSE}^{P}_v(L)}^{\,L}}\Big) \] \[ \mathrm{skill}^{\mathrm{ext}}_v = \max\!\Big(0,\ 1 - \frac{\mathrm{RMSE}(F_v, x_v)}{\mathrm{RMSE}(P_v, x_v)}\Big) \quad \text{over the (origin, lead) pairs with } |x_v| > \operatorname{std}(x_v) \]The first is skill against persistence on all targets; the second on the extreme targets only, where a curve fit that follows the mean state breaks down.
Total.
\[ S_{\mathrm{pred}} = 0.7\cdot\tfrac12\big(\mathrm{skill}_{T_C} + \mathrm{skill}_{T_E}\big) + 0.3\cdot\tfrac12\big(\mathrm{skill}^{\mathrm{ext}}_{T_C} + \mathrm{skill}^{\mathrm{ext}}_{T_E}\big) \]