This article appeared in The Malaysian Accountant journal,

Jan-Feb 2011 issue

Auditing Derivatives: Things that can go wrong – Value at Risk

By Jasvin Josen

Introduction
Value at Risk or just “VaR” has been around for many years and is a major risk management
task of any investment bank. VaR has not been receiving good press lately, especially when
it was accused for not foreseeing the 2008 crisis. In this article, we will analyse what can go
wrong with value at risk and discuss the auditor’s approach.

What is Value at Risk

VaR is a technique for predicting the maximum losses from a portfolio of assests with a
known level of confidence. For example, VaR might predict with a 95% confidence level that
the maximum losses from a portfolio of risky assets would be RM 100,000. The effectiveness
of this tool is in its ease of comprehension, making it a commonly used figure in
boardrooms, with regulators and in annual reports.

VaR was developed as a risk assessment tool in the 1990s, driven by the failure in the risk
tracking systems used in the early 1990s to detect dangerous risk taken by traders. It
became so widely accepted that regulators like the Basle Committee allowed banks to
calculate their capital requirements for market risk with their own VaR models, using certain
parameters provided by the committee.

Trades in any portfolio may consist of stocks, bonds, commodities, currency trades or
derivatives. Their mark-to-market values will change depending on how the market factors
(equity prices, interet rates, currency rates...) change. VaR attempts to develop techniques
to change these market factors in a robust manner, to observe the possible changes to the
portfolio value.

Three main basic methods are used – historical simulation, variance-covariance and Monte
Carlo simulation. Briefly,

• the historical simulation approach looks back at previous market factors behaviour
and applies that same behaviour into the current portfolio to observe the changes in
portfolio value.
 • in the Monte Carlo approach the market factors are simulated according to a defined
statistical distribution. Then the simulated market factors are applied to the current
portfolio to observe the changes in value.
• the variance-covariance approach works differently. It derives historical risk
measures like standard deviation and correlation for each market factor. Then it
applies these risk measures to the current portfolio, (after breaking down the
portfolio into standard instruments) to derive the value at risk.

What can go wrong in using VaR?

VaR cannot sense a catastrophic event as it pays attention to ‘normal’ loss and not
‘abnormal’ loss

It is said that VaR is a measure of ‘normal’ market risk. But what is normal? If one says that
there is a 5% probability that a portfolio will not lose more than $100,000 in the following
week, how sure are we that this is a normal loss? The choice of the confidence level is open
to interpretation.

Joe Nocera of New York Times (Jan 4, 2009) suggests that VaR was useful to risk experts but
nevertheless exacerbated the crisis by giving false security to bank executives and
regulators.
The 2008 crisis dealt with new products like the Credit Default Swaps. These products
generate small gains and very rarely have losses. But when they do have losses, they are
huge. As such, it was outside the 99 percent probability and did not show up in the VaR
number as “it was lost in the tails”.

Some critics claim that VaR looks at only a small slice of the risk and a great deal of valuable
information in the distribution is ignored. Manageable risk near the centre of the
distribution is focussed on and the tails are ignored.

What does the Auditor do?

Depending on the level of importance that is placed on VaR in a firm, the auditor should
observe if risk managers have a clear comprehension of all the important confidence levels
and are able to place enough importance to them.

Ignoring important risks -- contagion and liquidity

Nassem Taleeb (the writer of “The Black Swan”) is concerned with what one calls
measureable risks. “Measurable risk is when you have a handle on the randomness. If I
throw a pair of dice, for example, I can pretty much measure my risk because I know that I
have one-sixth probability of having a three pop up. Non-measurable uncertainty is when
I'm throwing the dice without knowing what's on them. In the real world, most social events
are non-measurable because nobody hard-coded the rules of the game”

VaR attempts to measure only market risk – the risks that directly change the value of
financial instruments in the markets. But there are other risks which are just as important –
contagion and liquidity.

Although VaR models take into account the increased risk brought on by leverage, it fails to
distinguish between leverage and liquidity risk – borrowed money market instruments can
be called in at any time and cause a major liquidity crisis. Liquidity crises can do serious
damage to dynamic hedging to rebalance the portfolios – a major panic that dries up
liquidity will cause unreasonable bid-ask spreads, making hedging extremely difficult.

The contagion effect which happens in no certain pattern in periods of market stress will
break down correlations assumptions in VaR models, particularly the variance-covariance
model.

What does the Auditor do?

The auditor must watch for the extent of leverage in the firm and market to deduce the
appropriateness of VaR to interpret risk. He could discuss with management to assess the
importance placed on contagion and liquidity risk which are not measureable but
detrimental to the market and economy.

Relying on the wrong past

This is the problem of the historical simulation where the prior N days from which the
historical sample was drawn upon is not representative of the present. One cannot be
confident that errors of this sort will “average out”.

Traders will know whether the actual prices changes over the last 100 days were typical, and
therefore will know for which position the VaR is underestimated or overestimated. If VaR is
used to set risk or position limits, the traders can exploit their knowledge of the biases in the
VaR system and expose the company to more risk that the risk management committee
intended, creating an incentive to take “excessive but remote risks”.

What does the Auditor do?

Alertness to industry and market knowledge will prompt the auditor to have considerable
intuition of the appropriateness of the past data to represent the present. In times of
disagreement, he should take it up with management.

Wrong distribution assumption

The variance-covariance approach is the popular choice for investment banks to compute
value at risk as it is easier to implement. However, a major assumption in the model is that
returns of all assets are normally distributed. If the actual returns are not normally
distributed, the computed value at risk will surely understate the true value at risk.

The Monte Carlo approach also has a tendency to choose the Normal Distribution to derive
the random market factors. But this method also allows the designer to choose another
statistical distribution. However, this flexibility could lead him to making a bad choice. The
chosen distribution might not adequately approximate the actual distribution of the market
factors.

The normal distribution assumption is far more detrimental for credit spreads. Credit
spreads exhibit huge jumps and is proven to be nowhere near the Normal Distribution. For
this, Credit VaR models are designed alongside the mainstream VaR models.

What does the Auditor do?

Backtesting procedures discussed later could give the auditor a good insight to the gaps
between the actual and assumed distribution of asset returns.

Input errors and measurement errors

In the variance covariance approach, even if the returns distribution assumption holds up,
the value at risk can still be wrong if the variance and covariance estimates are incorrect.
There is because of a problem called the standard error. To quote Taleeb’s example: “When
I use a thermometer, I may be aware that there is one or two degrees of error in my
measurement of the temperature. But here, I don't know much about the instrument,
particularly when it comes to rare events”

A related problem occurs when the variance and covariance across assets change over time.
Fundamentals that drive these numbers do change over time. For example, the correlation
between the USD and MYR may change if oil prices increase by 20%. This can lead to a
breakdown in the computed VaR number.

What does the Auditor do?

While recognising the inherent problem of the standard error, the auditor could review for
scientific efforts to minimise this error. As for non-stationary of risk measures, the auditor
could review for efforts to determine its impact. For example, one could break down the
historical data into shorter periods (or by major events) to observe for any significant
changes in the risk measures.

Some VaR models are not designed for options

The limitation of the variance-covariance approach is that it incorporates options by
replacing them with “delta-equivalent” spot positions. This is to “linearise” the options
positions. However, options are not linear instruments. The price of the option does not
behave in the same manner as the underlying price. This method can only work with
moderate option positions, in a very short holding period.

What does the Auditor do?

The auditor needs to watch for the level of options and option-like products (e.g.
convertible bonds) included in portfolios.

Other considerations for the Auditor

Other than the above concerns, the following considerations could prove useful for the
auditor to determine the effectiveness of VaR.

Improvements and Alternatives to VaR

Over the years there has been a lot of effort to make value at risk a more reliable measure.
Some deal with the non-normality of asset returns, some with the non-linearity problem for
options, and others try to counter the non-stationary problem in volatility with time series
models.

There is also the inverse risk logic that does not attempt to calculate probabilities but
adopts a stress testing approach, in a more realistic way to explain where the pressures are.
[reference: Asian Link, January 2010: The Inverse Risk Logic Strategic Paradigm by Dr. David
Bobker]

Bearing in mind that VaR was built under the assumption of asset normality, probability of
losses and value at risk are the simplest to compute with the normal distribution. It gets
progressively more difficult with asymmetric and fat tailed distributions.

In the course of the auditor’s work, especially in the present environment where
instruments are getting more complex and plentiful in the Asian investment banks, the
auditor should be attentive to these improvements and the extent it could complement the
downside of VaR.

Back test, Back test, Back test!

Concerns about the reliability of any method can be addressed by comparing actual changes
in value to the model numbers; and this applies to VaR as well. It is performed by comparing
a sample of VaR numbers and the actual mark-to-market portfolio profit and losses, and
answering two questions;
 (i) does the distribution of actual mark-to market profits and losses appear similar
to the distribution used to determine the VaR amount?
(ii) do the actual losses exceed the VaR amount with the expected frequency?

However, chances are that the distribution of the actual portfolio returns will almost always
differ from the expected distributions. Because of this, very large samples may be required.
Nevertheless, it provides the auditor with a good feel of the VaR numbers and how much
reliance to place on it.

Stress Testing

Value at risk tells us that the loss will not exceed a certain amount – but how much can the
losses be? Stress testing usually acts as a complimentary tool. It is a set of scenario analyses
to investigate the effects of extreme market conditions. However, parameters are stressed
purely on the judgement and experience of the risk manager, for example 5 or 10 standard
deviations away from the mean or by picking actual extreme events.

The auditor must review stress testing results hand in hand to compliment the VaR and
watch for the adequacy of scenarios.

The human element

Focussing on quantitative techniques will cause one to miss major problems that occurs
systemically like over-leveraged firms, rogue trading and fraud. The auditor must always be
mindful of human factors that are usually the main cause of downfalls.

Another impact of the human element is in the VaR itself. The effectiveness of VaR depends
of the people who understand the technology, interpret the results of VAR analysis, balance
it with other means of testing and articulate the VaR results to management, shareholders
and analysts.

Conclusion

Taleeb postulates that value at risk should be nothing but a small footnote in the way we
view the risks, not the dominating tool. Damodaran (New York University) is of the opinion
that it is more prudent to use all of the information in the probability distribution rather
than a small slice of it. Mr Pasquier, head of investment risk at Axa Investment Managers
says: “It is necessary to have models but also to understand their limitations – common
sense and discussion are still very important.”[source: Financial Times, Oct 17, 2010]
The risk management objective function is survival, not profits and losses. To a risk
manager, VaR is the level of losses at which you stop trying to guess what will happen next,
and start preparing for anything that may happen. It is far more important to worry about
what happens when losses exceed VaR. It is perhaps best for the auditor to think the same
way too.