Independent review of appraisal results - a 'must-have' feature

[petagny]

The UK House of Commons Transport Select Committee commissioned an independent review of the economic appraisal of the proposed Y-shaped high-speed rail line (HS2) linking London with Birmingham and beyond. The review was intended to guide the Committee's questioning of expert witnesses at the parliamentary enquiry into the £32 billion scheme (in the UK railway investment projects must be subject to parliamentary approval).

The reviewers were asked to answer a series of questions posed by the Committee. Both the questions and the answers make interesting reading (see attachment). The conclusions are that:

'The monetised estimates are surrounded by a degree of uncertainty—indeed, the sensitivities published by HS2 Ltd for HS2 show a range of 0.7–2.7 for the BCR [benefit-cost ratio] excluding WEIs [wider economic impacts].'

Note that a BCR <1.0 means that the scheme would not represent good value for money.

The wider economic impacts, i.e., effects outside those on the direct users of the new services, are positive and enhance the case for the scheme. Generally, the review seems to find that the estimates of WEIs are reasonably well balanced.

The review draws attention to experience with other major rail schemes in Europe and highlights the issue of 'optimism bias':

'...in almost all schemes considered, unforeseen circumstances have arisen and contributed to higher costs or lower revenues than expected, although some schemes have still delivered good VfM.'

The optimism bias risk has to a certain extent been mitigated by adjustment factors applied to the appraisal of HS2. This is standard practice in the UK (see the UK finance ministries 'Green Book'). All the same, perhaps one should not rule out the possibility of gaming, i.e., deliberate under-under-estimation of costs in the knowledge that adjustment factors will be applied!

[Napodano]

For the uninitiated persons

 A benefit-cost ratio (BCR) is an indicator, used in the formal discipline of cost-benefit analysis, that attempts to summarize the overall value for money of a project or proposal. A BCR is the ratio of the benefits of a project or proposal, expressed in monetary terms, relative to its costs, also expressed in monetary terms. All benefits and costs should be expressed in discounted present values.

Rationale
In the absence of funding constraints, the best value for money projects are those with the highest net present value. Where there is a budget constraint, the ratio of NPV to the expenditure falling within the constraint should be used. In practice, the ratio of PV of future net benefits to expenditure is expressed as a BCR. (NPV-to-investment is net BCR.) BCRs have been used most extensively in the field of transport cost-benefit appraisals. The NPV should be evaluated over the service life of the project.

Problems
Long-term BCRs, such as those involved in climate change, are very sensitive to the discount rate used in the calculation of net present value, and there is often no consensus on the appropriate rate to use.

The handling of non-monetary impacts also present problems. They are usually incorporated by estimating them in monetary terms, using measures such as WTP (willingness to pay), though these are often difficult to assess. Alternative approaches include the UK's New Approach to Appraisal framework.

A further complication with BCRs concerns the precise definitions of benefits and costs. These can vary depending on the funding agency.

Source: Wikipedia
http://en.wikipedia.org/wiki/Benefit-cost_ratio

[petagny]

Thanks Napodano. It's easy to forget that not everyone is initiated into the 'dark arts'!

Do you have the 'New Approach to Appraisal Framework'? It was mentioned in the Oxera review with respect to estimation methods for Wider Economic Impacts. Since this is an area where optimism bias can spiral into the stratosphere it would be good to understand what the UK approach is.

[Napodano]

Do you have the 'New Approach to Appraisal Framework'? It was mentioned in the Oxera review with respect to estimation methods for Wider Economic Impacts. Since this is an area where optimism bias can spiral into the stratosphere it would be good to understand what the UK approach is.

No, I do not. Anybody to the rescue?

[petagny]

This is a useful link:


http://www.dft.gov.uk/webtag/documents/overview/unit1.1.php

[John Short]

Continuing the theme - but on a new tram system in Old Reekie -
http://thescotsman.scotsman.com/edinburghs-new-tram-network/Tram-cost-to-reach-700m.6789719.jp

I will try and see if there was any appraisal document.  The business plan can be found in:

http://www.edinburghtrams.com/index.php/story_so_far/phase/preparation_and_procurement/

[petagny]

In South Korea, it is mandatory to re-appraise a project when capital costs rise by more than 20% in real terms. Similarly, if there is a major shift in demand. This is done on a sunk-cost basis.

In the case of the Edinburgh tram, it looks like such a review would have been useful some time ago.

[petagny]

Here's a not so independent review of the case for the second high-speed rail link (HS2). It's prepared by a think-tank that is generally opposed to government spending in any form. They are nevertheless probably right to point out the risks of higher than expected costs and lower than expected benefits -  'optimism bias'.

[petagny]

Here's a summary of the Transport Select Committee's findings concerning the proposed extension of the UK's high speed rail network (HS2):

http://www.parliament.uk/business/committees/committees-a-z/commons-select/transport-committee/news/hsr---substantive/

The Committee says that 'There is a good case for a high speed rail network, linking London and the major cities of the Midlands, the North and Scotland.'

On the other hand, the Committee also says that ''Investment in HS2 must not lead to reduced investment in the 'classic' rail network.''

My interpretation of this statement is that, although HS2 might be a good investment, investment in the rest of the network would be a better use of resources if the budget for railway investment is fixed. This is consistent with an article that appeared in the FT back in 2008 when Network Rail began looking into expanding the high speed rail network:

'Problems are mounting because of severe overcrowding on lines around London, Birmingham and Leeds. Rather than considering sweeping new lines, these congested pinch-points must be the first target for any new investment in intercity routes.

The other target for investment is the commuter train system, and it should be the priority for fresh funding. London has seen a 32 per cent increase in commuter numbers in the past decade, and there have been even larger increases in demand for commuter trains in Leeds, Bristol, Manchester and Birmingham. An obsessive focus on intercity travel will distract from the vital priority of ensuring that these cities continue to lead the econ¬omic renaissance of Britain’s regions. The TGV in France is a technological triumph, but the price of building the super-fast train was that France was left with a wretched commuter network. Britain cannot afford a similar mistake.'

Of course, the government could choose to expand capital spending on railways, but this would then involve making trade-offs with other sectors.