Rank-order Statistics for Validating Discriminative Power of Credit Risk Models
This paper provides practical insights into common statistical measures used to validate a model's discriminatory power for the probability of default (PD), loss liven default (LGD) and exposure at default (EAD). The review of available rank-order statistics is not based on analysing empirical data. Thus, the study has more of an informative value without delivering empirical evidence. When there is an alternative model available for comparison, this paper proposes to use the cumulative accuracy curve and the accuracy ratio to assess the rank-order ability for PD models given their popularity in practice. When there is no model available for comparison, due to the limited techniques in this area, this paper proposes to compare the confidence intervals in order to prove that a rating system has any discriminative power. For the LGD/EAD/slotting models, this paper recommends using a graph to check the rank-order ability. No statistical test is recommended. Focusing on enhancing practical implications for the financial industry, this paper advises banks on the existing CRR self-attestation requirements. (original abstract)
- BCBS (2001), The New Basel Capital Accord, consultative document BCBS, 31 May, Banking Committee on Banking Supervision, Bank for International Settlements, http://www.bis.org/publ/bcbsca03. pdf, accessed on 11 September 2015.
- Blochwitz S., Hamerle A., Hohl S., Rauhmeier R., Rosch D. (2004), Myth and reality of discriminatory power for rating systems, 27 July, http://ssrn.com/abstract=2350369, accessed on 11 September 2015.
- Calabrese R. (2009), The validation of credit rating and scoring models, Swiss Statistics Meeting Paper, 29 October.
- DeLong E.R., DeLong D.M., Clarke-Pearson D.L. (1988), Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach, Biometrics, 44(3), 837-845.
- Engelmann B., Hayden E., Tasche D. (2003), Measuring the discriminative power of rating systems, Deutsche Bundesbank Discussion Paper, Series 2: Banking and Financial Supervision, 01/2003.
- Financial Conduct Authority (2015), Prudential sourcebook for investment firms, IFPRU Chapter 4. Credit risk, https://www.handbook.fca.org.uk/handbook/IFPRU/4.pdf, accessed on 11 September 2015.
- Fritz C.O., Morris P.E., Richler J.J. (2012), Effect size estimates: current use, calculations, and interpretation, Journal of Experimental Psychology: General, 141, 2-18.
- Goodman L.A., Kruskal W.H. (1954), Measures of association for cross classifications, Journal of the American Statistical Association, 49(268), 732-764.
- Hamerle A., Rauhmeier R., Rösch D. (2003), Uses and misuses of measures for credit rating accuracy, 28 April, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2354877, accessed on 11 September 2015.
- Hayden E., Tasche D. (2003), Testing rating accuracy, Risk Magazine, 03 January.
- Hong C.S. (2009), Optimal threshold from ROC and CAP curves, Communications in Statistics, 38(10), 2060-2072.
- Kendall M.G., Gibbons J.D. (1990), Rank correlation methods, Oxford University Press.
- Kraft H., Kroisandt G., Müller M. (2002), Assessing the discriminatory power of credit scores, Humboldt Universitat zu Berlin Working Paper, 15 August, http://edoc.hu-berlin.de/series/sfb-373- papers/2002-67/PDF/67.pdf, accessed on 11 September 2015.
- Kraft H., Kroisandt G., Müller M. (2014), Redesigning ratings: assessing the discriminatory power of credit scores under censoring, Journal of Credit Risk, 10(4), 71-94.
- Linker D., Van Baal G. (2008), Backtest report LGD model 2007 VW Bank, Capgemini Annual LGD Model Review Report, Volkswagen Pon Financial Services.
- Luo X., Shevchenko P.V. (2013), Markov chain Monte Carlo estimation of default and recovery: dependent via the latent systematic factor, CSIRO Mathematics Working Paper, 10 April.
- Myers J.L., Well A.D. (2003), Research design and statistical analysis, Lawrence Erlbaum.
- Newson R. (2002), Parameters behind "nonparametric" statistics: Kendall's tau, Somers' D and median differences, The Stata Journal, 2(1), 45-64.
- Newson R. (2006), Efficient calculation of Jackknife confidence intervals for rank statistics, Journal of Statistical Software, 15(1), 1-10.
- PRA (2013), Internal ratings based approaches, Prudential Regulation Authority, Bank of England Supervisory Statement, SS11/13.
- Rezac M., Rezac F. (2011), How to measure the quality of credit scoring models, Czech Journal of Economics and Finance, 61(5), 486-507.
- Satchell S., Xia W. (2006), Analytic models of the ROC curve: applications to credit rating model validation, Quantitative Finance Research Centre Paper, 181, University of Technology Sydney.
- Siegel S. (1957), Nonparametric statistics, The American Statistician, 11, 13-19.
- Sobehart J., Keenan S. (2004), The score for credit, Risk Magazine, 2 March.
- Somers R.H. (1962), A new asymmetric measure of association for ordinal variables, American Sociological Review, 27(6), 799-811.
- Tasche D. (2009), Estimating discriminatory power and PD curves when the number of defaults is small, http://arxiv.org/pdf/0905.3928v2.pdf, accessed on 11 September 2015.
- Van der Burgt M. (2008), Calibrating low-default portfolios, using the cumulative accuracy profile, Journal of Risk Model Validation, 1(4), 17-33.
- Zenga M. (2007), Inequality curve and inequality index based on the ratio between lower and upper arithmetic means, Universita delgi Studi di Milano-Bicocca Working Paper, http://dipeco.economia. unimib.it/web/pdf/iniziative/Zenga.pdf, accessed on 11 September 2015.