Guidance Notes - Further risk quantification refinement by risk experts

The information from the two-phase risk workshop process provides a useful starting point for the risk quantification now carried out by the risk experts. Each expert arrives at a considered quantification of the risks after a short break to absorb the outcomes of the phase one and two workshops. There may be further structured interviews between the risk analyst and each expert.

As noted, probabilities and uncertainties in cost prediction vary from stage to stage in a project. Ideally the contingency and tolerance levels applied to risk at the business case stage of a project should give a reasonable indication of project risk. Then the cost tolerances should reduce as the project is developed and risks are better understood, reduced and removed.

Simple valuation technique

Risk experts should realistically assess how likely final costs are to be above,l or below the amount included in the Raw PSC. The number of point estimates used in valuing risk (each having a different expected consequence) should reflect the materiality of the risk and the information available. Where empirical evidence is unavailable or incomplete, commonsense approximations may be used.

Value of Risk = consequence x probability of occurrence

(The risk assumptions tables attached in the appendices for both the simple and advanced valuation techniques illustrate this formula.) The value of each risk is the sum of these probability weighted consequences (assuming that they are all independent), plus any contingency amount in the financial model which is to be attached.

The following probabilities and consequences have been estimated:

Assumption

Probability
%

Consequence
$'000s

Value of risk
$'000s

Below base amount

20

(10 000)

(2 000)

No deviation from base amount

10

0

0

Overrun: likely

40

15 000

6 000

Overrun: moderate

20

20 000

4 000

Overrun: extreme

10

25 000

2 500

100

10 500

Note: Base amount refers to the cost of the raw plant and equipment estimated in the Raw PSC of $50 million.

Timing of risk: Operating period from Year 3 to Year 12

Allocation of risk: Transferred to the private party

(For an example of how to model this risk in the PSC financial model, please refer to Appendix C: Public Sector Comparator financial model - Simple risk evaluation method. This risk is recorded in the Risk assumptions worksheet. The NPC modelling of this risk is in the Risk - Simple worksheet.)

Advanced valuation technique

By this stage the risk experts should be relatively comfortable with the task ahead. However, people often consider it more difficult to provide a probability distribution than they do a single point estimate. There are two components of uncertainty included in the distribution - the inherent uncertainty in the variable itself, and the uncertainty arising from the expert's lack of knowledge of the variable. In a risk analysis model these two are not differentiated. The combined uncertainty is entered into the model. Experts may be reluctant to include lack of knowledge in the analysis, but there is no alternative. (There is no perfect expert). Some suggestions for putting the expert at ease are:

1.  Explain that providing a distribution for a variable does not require a greater knowledge
of the variable than a single point estimate - quite the reverse.  It gives the expert a
means to express their lack of exact knowledge.

2.  Reassure them that the estimation of a probability distribution does not require any
great knowledge of probability theory.

3.  Reassure them that the only expectation is that they are 90 per cent confident that the
risk outcome will lie somewhere within their estimation of the risk.

4.  Remind them that there will be an opportunity to revise the estimates at a later stage, particularly if they are found to be significant drivers of the overall risk.

Considerable reluctance can also be overcome by careful phrasing of the question. For example, if trying to elicit the rates of failure of an average contractor against a service requirement, it makes much more sense for a group of people to be asked 'Over the last 10 year period, how many failures have you had with your contractors?' and 'How good do you think your contractor is compared with the average contractor?' rather than 'What is the rate of failure of an average contractor?'.

Defining distributions

A probability distribution describes a probability that a variable will have a given value or occur within a given range. The fact that the area under the graph of a probability distribution is equal to one means that the cost will fall within the range of costs shown on the graph. There are many standard distributions available within Monte Carlo analysis software.

1.  The most commonly used distribution for modelling project risk is the triangular distribution, based on a three point estimate of cost outcome.  It is a popular distribution to use as it is very simple and clear and can be used when there is little, or no statistical information on a variable's distribution. Note that it overestimates the tails of outcome at the expense of values close to the mean.  Related to the triangular distribution is the truncated triangular distribution which can be used to place confidence levels (e.g.  5 per cent and 95 per cent) on the best case and worst case estimates. Where this distribution is used under simulation, values that fall outside of these estimates may be selected by the Monte Carlo analysis.

Triangular distribution is frequently used in situations where the actual distribution is not known.

2.  Normal (Gaussian) distribution is another frequently used distribution, in part because of the central limit theorem which states that the mean of a set of values drawn independently from the same distribution will be normally described.  Many natural variables fall into a normal distribution, such as human heights, horse weights etc. A normal distribution is suitable where a distribution is not known, but it is understood to be symmetrical about a mean value, and more likely to be near the centre than at the extremes.

3.  Uniform distribution is used where the variable is bounded by a known maximum and minimum value and all values in between occur with equal likelihood. In common with triangular distribution, this has the advantage of being intuitively obvious and highlights the risk where there is little, or no statistical information about its distribution.

4.  Other distributions that are infrequently used to describe project risk, but may be relevant for a particular risk include:

•  binomial (based on a number of trial events and the known probability for each trial - the simple valuation technique of point estimates is based on this distribution)

•  Poisson (describes the number of events that will occur in a given unit of time, given that the rate is known)

•  exponential (describes the amount of time between occurrences)

•  log normal (useful for representing quantities that vary over several orders of magnitude).