A predictive score is not an IT deliverable. It is probabilistic, it can be wrong, and it erodes over time. Yet most organisations specify it once, test it once, and announce a saving — six full-time equivalents, say — without ever pricing the cost of an error. The first time the equation is written out, the saving usually shrinks: in one claims-triage case, six FTE became about two, and the business accepted the figure because it could finally see how it was built.
The equation has four parts. A confusion matrix with asymmetric unit costs set by the business, not by the data team. A control leg — a share of cases handled the old way — so that performance is measured against reality rather than assumed. The cost of labelling, of running and of supervising the model. And the sensitivity of the winning threshold to the cost of an error, which tells you how robust the operating point is.
The principle is simple: data illuminates, the business decides. Thresholds, kill switch, production sign-off and retraining budget belong to a named business owner, with an operational surveillance committee that meets and signs the continuation of production under an agreed loss limit.
Monitoring is a doctrine, not a dashboard: six levels, three time horizons, seven corrective gestures, an escalation chain. And honesty about what is not measured — recall in production, for instance — is part of the deliverable. A method that cannot say “we do not know this yet” will eventually say something false.
Was this useful? One click, no account.
Want to take it further? Contact us about this · Share on LinkedIn