A threshold is a decision — choosing τ for a credit-default model

A classifier like XGBoost does not output a decision. It outputs a score \(\hat p(x) \in [0,1]\), and a decision appears only when we choose a rule:

\[ \hat y(x) = \mathbb{1}\{\hat p(x) \ge \tau\} \](1)

Libraries default to \(\tau = 0.5\). That default looks neutral, but it isn't: it encodes a specific belief about the cost of errors. In the Credit Default project, moving \(\tau\) from 0.50 to 0.24 raised recall on defaulters from 35.6% to 60.1% — a bigger change than any hyperparameter search delivered.

§1The cost view

Let \(c_{FP}\) be the cost of flagging a client who would have paid (false positive) and \(c_{FN}\) the cost of missing a defaulter (false negative). For a client with true default probability \(p\), the expected cost of each action is

\[ \mathbb{E}[\text{cost} \mid \hat y = 1] = (1-p)\,c_{FP}, \qquad \mathbb{E}[\text{cost} \mid \hat y = 0] = p\,c_{FN}. \](2)

Predicting “default” is the better action whenever \(p\,c_{FN} \ge (1-p)\,c_{FP}\), which gives the Bayes-optimal threshold

\[ \tau^{*} = \frac{c_{FP}}{c_{FP} + c_{FN}}. \](3)

So \(\tau = 0.5\) is optimal only if both errors cost the same. Inverting (3), any threshold implies a cost ratio:

\[ \frac{c_{FN}}{c_{FP}} = \frac{1-\tau}{\tau}, \qquad \tau = 0.24 \;\Rightarrow\; \frac{c_{FN}}{c_{FP}} \approx 3.17. \](4)

Read literally: choosing 0.24 says that missing a defaulter costs about three times as much as a false alarm. That is a statement a credit team can agree or disagree with — which is exactly the point. A threshold turns a business assumption into a number.

Caveat — calibration. Equation (3) applies to probabilities. If \(\hat p\) is not calibrated (if, among clients scored 0.3, the default rate is not about 30%), the implied ratio in (4) is only indicative. Calibration was not evaluated in the project — the follow-up note does it: the scores are calibrated, and the implied ratio holds at about 2.7–3.2×.

§2What the project actually did

No explicit costs were available, so the threshold was chosen by a proxy objective: maximize F1 on the validation split (4,800 clients, never used for the final evaluation). The grid search produced \(\tau = 0.24\) with validation F1 = 0.5405. On the untouched test set (6,000 clients), the same threshold gave F1 = 0.5396 and recall = 0.6006 — almost no degradation, a good sign that the choice was not overfitted.

The explorer below uses the test-set curve from the model-selection notebook (0.05 grid) to show the whole trade-off and the cost ratio each \(\tau\) implies:

xgboost (grid search) · test set · 0.05 gridn = 6,000

threshold τ 0.50

precisionrecallF1

—

precision

—

recall

—

implied c_FN/c_FP

Curve: notebooks/09_model_selection.ipynb (test set, n = 6,000). The operating threshold 0.24 was selected by maximizing F1 on the validation split (F1 = 0.5405) in notebooks/010_model_validation.ipynb. Implied cost ratio = (1 − τ)/τ, valid only if probabilities are calibrated.

§3A check from the literature

For a calibrated classifier, the F1-optimal threshold equals half the maximum attainable F1 (Lipton, Elkan & Narayanaswamy, 2014):

\[ \tau_{F1}^{*} = \frac{F1^{*}}{2}. \](5)

With validation \(F1^{*} \approx 0.54\), (5) predicts \(\tau \approx 0.27\). The empirical optimum, 0.24, is close. Update: the follow-up note checks calibration — the scores are well calibrated, and the gap is sampling noise: the bootstrap 95% interval of the F1-optimal threshold is [0.22, 0.33]. It also explains why F1-optimal thresholds are usually well below 0.5: F1 is at most 1, so for a calibrated model the optimal threshold can never exceed 0.5.

§4Takeaways

A threshold is part of the model. Report it, justify it, and version it — the API returns it with every prediction for this reason.
Choose \(\tau\) on validation data, never on the test set. The test set is for one final, honest number.
If costs are known, use (3). If not, a proxy like F1 works — but translate the result back into an implied cost ratio with (4) and ask whether it makes sense.
Check calibration before reading scores as probabilities.

Reference: Z. C. Lipton, C. Elkan, B. Narayanaswamy. Thresholding Classifiers to Maximize F1 Score. ECML PKDD, 2014. · Data: github.com/Igor-Millis/credit-default-api, notebooks 09 and 010.