The real world is messy. For this reason, it rarely fits into the models that scientists and analysts use to try to understand it. There are always edge cases, events that have not occurred in the past and therefore are not included in the possibilities, and then there is pseudoscience that does not even try very hard to match real world scenarios but none the less produces confident results that are unrealistic. The financial world is no different.
Many of the tools that are used to understand the financial world come with their own limitations and flaws. Usually, these limitations are contained in the assumptions of the methodology or model that is used. Therefore, it is important to not just understand the output of a financial model but also to understand the methodology that the model was built on so that the assumptions of that model, including the flaws in those assumptions, are understood.
Perhaps the most prevalent flaw in financial assumptions is the use of the Bassian curve, sometimes called a normal distribution. This assumes that outcomes will fall into a “normal” distribution which makes the math easier but is rarely the case in the real world. Most outcomes do not fall into a normal distribution with a specific average and defined deviances from that average. This includes financial market data. Yet this normal distribution is often the underlying assumption of many financial models.
In some cases, practitioners recognize that a given outcome is unlikely to fall into a normal distribution, so they apply other statistical techniques such as introducing skewedness among other adjustments. This is an attempt to change the assumptions to better fit the data which can improve the accuracy of model outputs but still leaves plenty of room for error as the real-world experience is still unlikely to perfectly fit the model assumptions.
For example, in the runup to the financial crisis Wall Street banks came to rely on a risk metric called Value at Risk (VAR). This statistical tool was designed to narrow a banks’ risk down to one number that represented the amount that bank stood to lose on mortgage loans should multiple negative events occur. Of course, if this number had been more accurate many risky loans may not have been made and the financial crisis may have been avoided. But the number was not accurate because the statistical assumptions that were at its core did not accurately reflect the real-life situations it was trying to model. Yet, absent a better methodology many Wall Street leaders relied on this number when making important decisions even though they knew at its heart the model was flawed. They just hoped those flaws would not cause the model to fail during their tenure.
Another, but similar, metric that is used in investing is the maximum drawdown statistic. This number may be based on historical returns. However, as everyone who reads investment disclosures should know, past performance is not necessarily indicative of future returns. Therefore, relying purely on the past without considering how the world has changed, how the financial markets have evolved, or how investor psychology may differ from the past is inherently using assumptions that may be flawed.
Other models use statistical methodologies, similar to the Value at Risk (VAR) models, to estimate the maximum drawdown of an investment portfolio. However, these models fall prey to the same shortfalls that helped to create the financial crisis. That does not mean that these statistics are useless, but they should only be used by practitioners who understand the flaws in the assumptions and therefore know to take the results with a grain of salt. Unfortunately, these numbers are too often communicated to investors as if they are facts.
The future is uncertain. Financial models attempt to model this uncertainty and are the best tools available. But it is important to understand the weakness in the assumptions before acting.