Backtestability and the ridge backtest

Carlo Acerbi; Balázs Székely

doi:10.3934/fmf.2023019

Article Contents

2023, Volume 2, Issue 4: 497-521. Doi: 10.3934/fmf.2023019 open access

This issue Previous Article Distributed energy resources flexibility as volumetric options on electricity Next Article A stochastic gradient descent algorithm to maximize power utility of large credit portfolios under Marshall–Olkin dependence

Backtestability and the ridge backtest

Carlo Acerbi^1,2, , and
Balázs Székely^3,†,

1.
Larix Risk Consulting, Genève, Switzerland
2.
École Polytechnique Fédérale de Lausanne, Switzerland
3.
Morgan Stanley, Budapest, Hungary

^*Corresponding author: Carlo Acerbi
^*Corresponding author: Carlo Acerbi

^†This paper is the private opinion of the author and does not necessarily reflect the policy and views of Morgan Stanley.

Received: September 2023

Early access: October 2023

Published: December 2023

Abstract / Introduction Full Text(HTML) Figure(3) / Table(3) Related Papers Cited by

Abstract

We propose a formal definition of backtestability for a statistical functional of a distribution: a functional is backtestable if there exists a backtest function depending only on the forecast of the functional and the related random variable, which is strictly monotonic in the former and has zero expected value for an exact forecast. We discuss the relationship with elicitability and identifiability which turn out being necessary conditions for backtestability. The variance and the expected shortfall are not backtestable for this reason. We compare (absolute) model validation in the context of hypothesis tests via backtest functions, versus (relative) model selection between competing forecasting models, via scoring functions. We define a backtest to be sharp when it is strictly monotonic with respect to the real value of the functional and not only to its forecast. This decides whether the expected value of the backtest determines also the prediction discrepancy and not only its significance. We show that the quantile backtest is not sharp and in fact it provides no information whatsoever on its true value. The expectile is also not sharp; we provide bounds for its true value, which are looser for outer confidence levels. We then introduce the notion of ridge backtests, applicable to particular non–backtestable functionals, such as the variance and the expected shortfall, which coincide with the attained minimum of the scoring function of another elicitable auxiliary functional (the mean and the value at risk, respectively). This permits approximated sharp backtests up to a small and one–sided sensitivity to the prediction of the auxiliary variable. The ridge mechanism explains why the variance has always been de–facto backtestable and allows for similar efficient ways to backtest the expected shortfall. We discuss the relevance of this result in the current debate of financial regulation (banking and insurance), where value at risk and expected shortfall are adopted as regulatory risk measures.

Keywords:

Mathematics Subject Classification: 91G70, 91G60, 91B05, 62G10, 60G25.

Citation:

FullText(HTML)

Figure 1. Testing risk predictions is difficult, because the true risk is not observable a posteriori. What is revealed is just one random draw

Download: Full-size image PowerPoint slide

Figure 2. Dependence on $ v $ of tests $ Z_{ \mathrm{\bf{ES}}} $ and $ Z_2 $ in the case of correct predictions for $ \mathrm{\bf{ES}} $. Dashed lines represent critical values at 5% for the two tests. Notice the linear sensitivity of the latter and the muted, quadratic sensitivity of the former. We can see that $ Z_2 $ can easily generate a type Ⅰ error. Source: [3]

Download: Full-size image PowerPoint slide

Figure 3. Similar example in the case of an underestimation $ e = 0.8 \mathrm{\bf{ES}} $. $ Z_2 $ can generate a type Ⅱ error, while $ Z_{ \mathrm{\bf{ES}}} $ can not. Source: [3]

Download: Full-size image PowerPoint slide

Table 1. Common examples of canonical scoring functions

$ \mathrm{\bf{y}} $	$ S_ \mathrm{\bf{y}}(y,x) $	$ \mathcal{F}_S $
$ {\mathit{\boldsymbol{\mu}}} $	$ (y-x)^2 $	maximal
$ \mathrm{\bf{q}}_{1/2} $	$ \|y-x\| $	maximal
$ \mathrm{\bf{q}}_\alpha $	$ \alpha (x-y)_+ + (1-\alpha)(x-y)_- $	maximal
$ \mathrm{\bf{e}}_\alpha $	$ \alpha (x-y)_+^2 + (1-\alpha)(x-y)_-^2 $	maximal

| Show Table

DownLoad: CSV

Table 2. Common examples of canonical identification functions. $ c\in [-\alpha, 1-\alpha] $, see remark 2.4

$ \mathrm{\bf{y}} $	$ I_ \mathrm{\bf{y}}(y,x) $	$ \mathcal{F}_I $
$ {\mathit{\boldsymbol{\mu}}} $	$ y-x $	maximal
$ \mathrm{\bf{q}}_{1/2} $	$ {\bf{1}}_{\{y>x\}} - {\bf{1}}_{\{y<x\}} + 2c{\bf{1}}_{\{y=x\}} $	$ F(x) $ cont. in $ \mathrm{\bf{q}}_{1/2} $
$ \mathrm{\bf{q}}_\alpha $	$ (1-\alpha) {\bf{1}}_{\{y>x\}}- \alpha {\bf{1}}_{\{y<x\}} + c{\bf{1}}_{\{y=x\}} $	$ F(x) $ cont. in $ \mathrm{\bf{q}}_{\alpha} $
$ \mathrm{\bf{e}}_\alpha $	$ (1-\alpha) (x-y)_- - \alpha (x-y)_+ $	maximal

| Show Table

DownLoad: CSV

Table 3. Common examples of backtest functions. These are unique up to a positive constant (see corollary 3.9)

$ \mathrm{\bf{y}} $	$ Z_ \mathrm{\bf{y}}(y,x) $	$ \mathcal{F}_Z $
$ {\mathit{\boldsymbol{\mu}}} $	$ y-x $	maximal
$ \mathrm{\bf{q}}_{1/2} $	$ {\bf{1}}_{\{y>x\}} - {\bf{1}}_{\{y<x\}} + 2c{\bf{1}}_{\{y=x\}} $	$ F(x) $ str. incr., cont. in $ \mathrm{\bf{q}}_{1/2} $
$ \mathrm{\bf{q}}_\alpha $	$ (1-\alpha) {\bf{1}}_{\{y>x\}}- \alpha {\bf{1}}_{\{y<x\}} + c{\bf{1}}_{\{y=x\}} $	$ F(x) $ str. incr., cont. in $ \mathrm{\bf{q}}_{\alpha} $
$ \mathrm{\bf{e}}_\alpha $	$ (1-\alpha) (x-y)_- - \alpha (x-y)_+ $	maximal

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	C. Acerbi and B. Székely, Backtesting expected shortfall, RISK, 2014.
[2]	C. Acerbi and B. Székely, General properties of backtestable statistics, SSRN, preprint, 2017.
[3]	C. Acerbi and B. Székely, The minimally biased backtest for expected shortfall, RISK, 2019.
[4]	C. Acerbi and D. Tasche, On the coherence of expected shortfall, Journal of Banking and Finance, 26 (2002), 1487-1503.
[5]	P. Artzner, F. Delbaen, J.-M. Eber and D. Heath, Coherent measures of risk, Mathematical Finance, 9 (1999), 203-228. doi: 10.1111/1467-9965.00068.
[6]	Amendment to the Capital Accord to Incorporate Market Risks, Basel Committee on Banking Supervision, 1996. Available from: http://www.bis.org/publ/bcbs24.pdf.
[7]	, Fundamental Review of the Trading Book: A Revised Market Risk Framework, Basel Committee on Banking Supervision, 2013. Consultative paper.
[8]	Minimum Capital Requirements for Market Risk, Basel Committee on Banking Supervision, 2019. Available from: http://www.bis.org/bcbs/publ/d457.pdf.
[9]	S. Bayer and T. Dimitriadis, Regression based expected shortfall backtesting, Journal of Financial Econometrics, 20 (2022), 437-471. doi: 10.1093/jjfinec/nbaa013.
[10]	F. Bellini, B. Klar, A. Müller and E. Rosazza Gianin, Generalized quantiles as risk measures, Insurance: Mathematics and Economics, 54 (2014), 41-48. doi: 10.1016/j.insmatheco.2013.10.015.
[11]	N. Costanzino and M. Curran, Backtesting general spectral risk measures with application to expected shortfall, SSRN, preprint, 2015.
[12]	N. Costanzino and M. Curran, A simple traffic light approach to backtesting expected shortfall, Risks, 6 (2018), 1-7. doi: 10.3390/risks6010002.
[13]	Z. Du and J. C. Escanciano, Backtesting expected shortfall: Accounting for tail risk, Management Science, 63 (2017), 940-958. doi: 10.1287/mnsc.2015.2342.
[14]	S. Emmer, M. Kratz and D. Tasche, What is the best risk measure in practice? A comparison of standard measures, The Journal of Risk, 18 (2015), 31-60. doi: 10.21314/JOR.2015.318.
[15]	T. Fissler, T. Gneiting and J. F. Ziegel, Expected shortfall is jointly elicitable with value–at–risk: Implications for backtesting, RISK, 2015.
[16]	T. Fissler and J. F. Ziegel, Higher order elicitability and Osband's principle, Annals of Statistics, 44 (2016), 1680-1707. doi: 10.1214/16-AOS1439.
[17]	T. Gneiting, Making and evaluating point forecasts, Journal of the American Statistical Association, 106 (2011), 746-762. doi: 10.1198/jasa.2011.r10138.
[18]	Risk-based Global Insurance Capital Standard, International Association of Insurance Supervisors, 2014. Consultative paper. Available from: https://www.iaisweb.org/uploads/2022/01/Risk-based_Global_Insurance_Capital_Standard_Consultation_Document.pdf.pdf.
[19]	Risk-Based Global Insurance Capital Standard, International Association of Insurance Supervisors, 2016. Consultative paper, version 1.0. Available from: https://www.iaisweb.org/uploads/2022/01/160719-2016-Risk-based-Global-Insurance-Capital-Standard-ICS-Consultation-Document.pdf.
[20]	J. Kerkhof and B. Melenberg, Backtesting for Risk-Based Regulatory Capital, Journal of Banking and Finance, 28 (2004), 1845-1865. doi: 10.1016/j.jbankfin.2003.06.007.
[21]	M. Kratz, Y. H. Lok and A. J. McNeil, Multinomial VaR backtests: A simple implicit approach to backtesting expected shortfall, Journal of Banking and Finance, 88 (2018), 393-407. doi: 10.1016/j.jbankfin.2018.01.002.
[22]	N. Lambert, D. M. Pennock and Y. Shoham, Eliciting properties of probability distributions, Proceedings of the 9$^th$ ACM Conference on Electronic Commerce, EC 08, 2008,129-138. doi: 10.1145/1386790.1386813.
[23]	R. Loser, D. Wied and D. Ziggel, New backtests for unconditional coverage of expected shortfall, Journal of Risk, 21 (2019), 39-59. doi: 10.21314/JOR.2019.406.
[24]	A. J. McNeil and R. Frey, Estimation of tail-related risk measures for heteroscedastic financial time series: An extreme value approach, Journal of Empirical Finance, 7 (2000), 271-300. doi: 10.1016/S0927-5398(00)00012-8.
[25]	F. Moldenhauer and M. Pitera, Backtesting Expected Shortfall: A simple recipe, Journal of Risk, 22 (2019), 17-42. doi: 10.21314/JOR.2019.418.
[26]	W. K. Newey and J. L. Powell, Asymmetric least squares estimation and testing, Econometrica, 55 (1987), 819-847. doi: 10.2307/1911031.
[27]	N. Nolde and J. F. Ziegel, Elicitability and backtesting: Perspectives for banking regulation, Annals of Applied Statistics, 11 (2017), 1833-1874. doi: 10.1214/17-AOAS1041.
[28]	K. H. Osband, Providing Incentives for Better Cost Forecasting, Ph.D. thesis, University of California, Berkeley, 1985.
[29]	R. T. Rockafellar and S. Uryasev, Conditional Value–at–Risk for general loss distributions, Journal of Banking and Finance, 26 (2002), 1443-1471.
[30]	J. F. Ziegel, Coherence and elicitability, Mathematical Finance, 26 (2016), 901-918. doi: 10.1111/mafi.12080.