Is τ = 0.24 really a cost ratio? — checking calibration of an XGBoost credit model

The previous note, “A threshold is a decision”, read the operating threshold of the Credit Default model as a cost ratio — \(\tau = 0.24\) means a missed defaulter costs about \(3.2\times\) a false alarm — and ended with a caveat: that reading holds only if the scores are calibrated probabilities. This note runs the check.

§1What “calibrated” means

A score \(\hat p\) is calibrated when, among all clients who receive the score \(p\), a fraction \(p\) actually default:

\[ \Pr\big(Y = 1 \mid \hat p(X) = p\big) = p \quad \text{for all } p. \](1)

Ranking quality (ROC AUC) says nothing about this: any increasing transformation of \(\hat p\) has the same AUC. Three measures, all computed on the untouched test set:

\[ \text{Brier} = \frac1N\sum_{i}(\hat p_i - y_i)^2, \qquad \text{ECE} = \sum_{b=1}^{10}\frac{n_b}{N}\,\big|\bar p_b - \bar y_b\big|, \](2)

plus the log loss. ECE (expected calibration error) compares, in each of 10 probability bins, the mean predicted probability \(\bar p_b\) with the observed default rate \(\bar y_b\).

§2Protocol

Same split as the project. 80/20 stratified train/test, then 80/20 train/validation, both with random_state = 42.
Saved model. The production pipeline was refit on all 24,000 training rows, so it is evaluated only on the 6,000 test rows.
Recalibration needs data the model never saw. The same hyperparameters were refit on the 19,200-row split; Platt scaling (logistic regression on the logit) and isotonic regression were fitted on the 4,800 validation rows; everything was scored on test.
Sanity check. The refit reproduces the notebook exactly: validation \(\tau = 0.24\), F1 = 0.5405, ROC AUC = 0.7833. The saved model gives test ROC AUC = 0.7770, as reported.

§3Result: the scores are already calibrated

saved model · test set · 10 binsn = 6,000

y: observed default rate · x: mean predicted probability

Each point is a bin of predicted probability (area ∝ n); the strip below shows how many clients fall in each bin. The 0.9–1.0 bin (n = 2) is not plotted. Source: research/calibration/calibration.py.

The points sit on the diagonal. Across the whole test set the mean predicted probability is 0.2213 against an observed default rate of 0.2212, and ECE = 0.0076 (bootstrap 95% interval 0.007–0.020; ECE is biased upward in finite samples, so the interval leans right). The largest gaps are a slight overconfidence in the 0.6–0.8 bins (predicted 0.66 vs observed 0.62, and 0.75 vs 0.72) — with about 240 clients per bin, that is about 1–1.2 standard errors, within noise.

model	Brier	log loss	ECE	ROC AUC	mean p̂	g(0.24)	c_FN/c_FP
saved (24k)	0.1355	0.4310	0.0076	0.7770	0.2213	0.240	3.17×
refit, raw	0.1352	0.4301	0.0124	0.7794	0.2208	0.240	3.17×
refit + Platt	0.1352	0.4300	0.0098	0.7794	0.2232	0.244	3.11×
refit + isotonic	0.1356	0.4394	0.0102	0.7781	0.2238	0.272	2.68×

Test set (n = 6,000, observed default rate = 0.2212). Calibrators fitted on validation (n = 4,800). g(0.24) = calibrated probability corresponding to the raw threshold 0.24.

Recalibration confirms it from the other side. The fitted Platt map is \(\operatorname{logit} g(p) = 0.978\,\operatorname{logit} p - 0.006\) — almost the identity. Isotonic regression slightly worsens the log loss (0.4394 vs 0.4301): with 4,800 validation rows, its step function fits noise.

§4So, is τ = 0.24 a cost ratio?

Yes, approximately. Recalibration is monotone, so it does not change any decision — it only changes what the threshold means. Mapping the raw cut-off through each calibrator gives

\[ \frac{c_{FN}}{c_{FP}} = \frac{1 - g(0.24)}{g(0.24)} \in [\,2.7,\; 3.2\,]. \](3)

The business reading survives: the model treats a missed defaulter as roughly three times as costly as a false alarm. That is now a defensible statement rather than an assumption.

§5A correction: the threshold itself is noisy

The previous note compared the empirical optimum 0.24 with the theoretical F1-optimal threshold for calibrated scores, \(F1^{*}/2 \approx 0.27\), and attributed the gap to imperfect calibration. Calibration turns out to be fine, so that explanation was wrong. Resampling the validation set 500 times and recomputing the F1-optimal threshold gives a 95% bootstrap interval of [0.22, 0.33] (median 0.24). The validation F1 curve is flat — between 0.532 and 0.541 for every \(\tau\) from 0.22 to 0.30 — so 0.27 is well inside the noise. The gap is sampling variability, not miscalibration.

§6Takeaways

Check calibration before reading scores as probabilities — and before “fixing” it. Here, recalibrating would have added a component without improving anything.
Recalibration has a cost: isotonic regression overfits small calibration sets. Platt scaling is the safer default when data is limited.
An optimal threshold is an estimate. Report its uncertainty; on a flat objective, a 0.24-vs-0.27 difference means nothing.
With calibration verified, the API's probability output can be read literally: a client scored 0.30 belongs to a group in which about 30% default.

Code and results: research/calibration/ (scikit-learn 1.9.0, xgboost 3.3.0) · Model: github.com/Igor-Millis/credit-default-api · Platt, J. (1999), Probabilistic outputs for support vector machines; Niculescu-Mizil & Caruana (2005), Predicting good probabilities with supervised learning, ICML.