Backtesting is a term used in modeling to refer to testing a predictive model on historical data. Backtesting is a type of retrodiction, and a special type of cross-validation applied to previous time period(s).
In the economic and financial field, backtesting seeks to estimate the performance of a strategy or model if it had been employed during a past period. This requires simulating past conditions with sufficient detail, making one limitation of backtesting the need for detailed historical data. A second limitation is the inability to model strategies that would affect historic prices. Finally, backtesting, like other modeling, is limited by potential overfitting. That is, it is often possible to find a strategy that would have worked well in the past, but will not work well in the future.[1] Despite these limitations, backtesting provides information not available when models and strategies are tested on synthetic data.
Backtesting has historically only been performed by large institutions and professional money managers due to the expense of obtaining and using detailed datasets. However, backtesting is increasingly used on a wider basis, and independent web-based backtesting platforms have emerged. Although the technique is widely used, it is prone to weaknesses.[2]Basel financial regulations require large financial institutions to backtest certain risk models.
For a Value at Risk 1-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table:[3]
backtesting exceptions 1Dx250
1-day VaR at 99% backtested 250 days
Zone
Number exceptions
Probability
Cumul
Green
0
8.11%
8.11%
1
20.47%
28.58%
2
25.74%
54.32%
3
21.49%
75.81%
4
13.41%
89.22%
Orange
5
6.66%
95.88%
6
2.75%
98.63%
7
0.97%
99.60%
8
0.30%
99.89%
9
0.08%
99.97%
Red
10
0.02%
99.99%
11
0.00%
100.00%
...
...
...
For a Value at Risk 10-day at 99% backtested 250 days in a row, the test is considered green (0-95%), orange (95-99.99%) or red (99.99-100%) depending on the following table:
In oceanography[5] and meteorology,[6]backtesting is also known as hindcasting: a hindcast is a way of testing a mathematical model; researchers enter known or closely estimated inputs for past events into the model to see how well the output matches the known results.
Hindcasting usually refers to a numerical-model integration of a historical period where no observations have been assimilated. This distinguishes a hindcast run from a reanalysis. Oceanographic observations of salinity and temperature as well as observations of surface-wave parameters such as the significant wave height are much scarcer than meteorological observations, making hindcasting more common in oceanography than in meteorology. Also, since surface waves represent a forced system where the wind is the only generating force, wave hindcasting is often considered adequate for generating a reasonable representation of the wave climate with little need for a full reanalysis. Hydrologists use hindcasting for model stream flows.[7]
An example of hindcasting would be entering climate forcings (events that force change) into a climate model. If the hindcast showed reasonably-accurate climate response, the model would be considered successful.
The ECMWF re-analysis is an example of a combined atmospheric reanalysis coupled with a wave-model integration where no wave parameters were assimilated, making the wave part a hindcast run.
^Bailey, Borwein, Lopez de Prado, Zhu (2014). "Pseudo-mathematics and financial charlatanism. Notices of the American Mathematical Society, Volume 61, Number 5, pp. 458-471" (PDF).{{cite web}}: CS1 maint: multiple names: authors list (link)
^FinancialTrading (2013-04-27). "Issues related to back testing".
^"Supervisory framework for the use of "backtesting" in conjunction with the internal models approach to market risk capital requirements" (PDF). Basle Committee on Banking Supervision. January 1996. p. 14.
^Taken from p.145 of Yeates, L.B., Thought Experimentation: A Cognitive Approach, Graduate Diploma in Arts (By Research) dissertation, University of New South Wales, 2004.
^"Hindcast approach". OceanWeather Inc. Retrieved 22 January 2013.
^Huijnen, V.; J. Flemming; J. W. Kaiser; A. Inness; J. Leitão; A. Heil; H. J. Eskes; M. G. Schultz; A. Benedetti; J. Hadji-Lazaro; G. Dufour; M. Eremenko (2012). "Hindcast experiments of tropospheric composition during the summer 2010 fires over western Russia". Atmos. Chem. Phys. 12 (9): 4341–4364. Bibcode:2012ACP....12.4341H. doi:10.5194/acp-12-4341-2012. Retrieved 22 January 2013.
^"Guidance on Conducting Streamflow Hindcasting in CHPS" (PDF). NOAA. Retrieved 22 January 2013.
November 24, 2023
backtesting, term, used, modeling, refer, testing, predictive, model, historical, data, type, retrodiction, special, type, cross, validation, applied, previous, time, period, contents, financial, analysis, hindcast, also, further, reading, referencesfinancial,. Backtesting is a term used in modeling to refer to testing a predictive model on historical data Backtesting is a type of retrodiction and a special type of cross validation applied to previous time period s Contents 1 Financial analysis 2 Hindcast 3 See also 4 Further reading 5 ReferencesFinancial analysis editIn the economic and financial field backtesting seeks to estimate the performance of a strategy or model if it had been employed during a past period This requires simulating past conditions with sufficient detail making one limitation of backtesting the need for detailed historical data A second limitation is the inability to model strategies that would affect historic prices Finally backtesting like other modeling is limited by potential overfitting That is it is often possible to find a strategy that would have worked well in the past but will not work well in the future 1 Despite these limitations backtesting provides information not available when models and strategies are tested on synthetic data Backtesting has historically only been performed by large institutions and professional money managers due to the expense of obtaining and using detailed datasets However backtesting is increasingly used on a wider basis and independent web based backtesting platforms have emerged Although the technique is widely used it is prone to weaknesses 2 Basel financial regulations require large financial institutions to backtest certain risk models For a Value at Risk 1 day at 99 backtested 250 days in a row the test is considered green 0 95 orange 95 99 99 or red 99 99 100 depending on the following table 3 nbsp backtesting exceptions 1Dx2501 day VaR at 99 backtested 250 days Zone Number exceptions Probability CumulGreen 0 8 11 8 11 1 20 47 28 58 2 25 74 54 32 3 21 49 75 81 4 13 41 89 22 Orange 5 6 66 95 88 6 2 75 98 63 7 0 97 99 60 8 0 30 99 89 9 0 08 99 97 Red 10 0 02 99 99 11 0 00 100 00 For a Value at Risk 10 day at 99 backtested 250 days in a row the test is considered green 0 95 orange 95 99 99 or red 99 99 100 depending on the following table nbsp backtesting exceptions 10Dx25010 day VaR at 99 backtested 250 days Zone Number exceptions Probability CumulGreen 0 36 02 36 02 1 15 99 52 01 2 11 58 63 59 3 8 90 72 49 4 6 96 79 44 5 5 33 84 78 6 4 07 88 85 7 3 05 79 44 8 2 28 94 17 Orange 9 1 74 95 91 24 0 01 99 99 Red 25 0 00 99 99 Hindcast edit nbsp Temporal representation of hindcasting 4 In oceanography 5 and meteorology 6 backtesting is also known as hindcasting a hindcast is a way of testing a mathematical model researchers enter known or closely estimated inputs for past events into the model to see how well the output matches the known results Hindcasting usually refers to a numerical model integration of a historical period where no observations have been assimilated This distinguishes a hindcast run from a reanalysis Oceanographic observations of salinity and temperature as well as observations of surface wave parameters such as the significant wave height are much scarcer than meteorological observations making hindcasting more common in oceanography than in meteorology Also since surface waves represent a forced system where the wind is the only generating force wave hindcasting is often considered adequate for generating a reasonable representation of the wave climate with little need for a full reanalysis Hydrologists use hindcasting for model stream flows 7 An example of hindcasting would be entering climate forcings events that force change into a climate model If the hindcast showed reasonably accurate climate response the model would be considered successful The ECMWF re analysis is an example of a combined atmospheric reanalysis coupled with a wave model integration where no wave parameters were assimilated making the wave part a hindcast run See also editApplied research customer foresight Black box model Climate ECMWF re analysis Forecasting NCEP re analysis Economic forecast Retrodiction Statistical arbitrage Thought Experiment Value at riskFurther reading editGerman Algo SystemReferences edit Bailey Borwein Lopez de Prado Zhu 2014 Pseudo mathematics and financial charlatanism Notices of the American Mathematical Society Volume 61 Number 5 pp 458 471 PDF a href Template Cite web html title Template Cite web cite web a CS1 maint multiple names authors list link FinancialTrading 2013 04 27 Issues related to back testing Supervisory framework for the use of backtesting in conjunction with the internal models approach to market risk capital requirements PDF Basle Committee on Banking Supervision January 1996 p 14 Taken from p 145 of Yeates L B Thought Experimentation A Cognitive Approach Graduate Diploma in Arts By Research dissertation University of New South Wales 2004 Hindcast approach OceanWeather Inc Retrieved 22 January 2013 Huijnen V J Flemming J W Kaiser A Inness J Leitao A Heil H J Eskes M G Schultz A Benedetti J Hadji Lazaro G Dufour M Eremenko 2012 Hindcast experiments of tropospheric composition during the summer 2010 fires over western Russia Atmos Chem Phys 12 9 4341 4364 Bibcode 2012ACP 12 4341H doi 10 5194 acp 12 4341 2012 Retrieved 22 January 2013 Guidance on Conducting Streamflow Hindcasting in CHPS PDF NOAA Retrieved 22 January 2013 Retrieved from https en wikipedia org w index php title Backtesting amp oldid 1158091278, wikipedia, wiki, book, books, library,
