UK's questionable approach to optimism bias

[petagny]

Last week the UK Government announced that the estimated cost of the planned high-speed rail link (HS2) from London to the Midlands and North had increased by nearly a quarter (from £34.5 billion to £42.6 billion). The Treasury's Green Book recommends that cost estimates should include an allowance for optimism bias (based on analysis of previous experience of systematic cost over-runs)and indeed this was done when preparing the first, lower, estimate: the attached risk assessment of costs completed in 2012 includes an uplift of one third based on an item-by-item assessment of the likelihood of optimism bias (as well as substantial allowances for physical contingencies). Clearly this was not enough and there seems to have been optimism bias in assessing the optimism bias!

Is the general approach, i.e., using coefficients to allow for optimism bias in cost estimates, flawed because it doesn't allow for gaming by planners?

http://www.ft.com/cms/s/0/7dda688a-de83-11e2-b990-00144feab7de.html#axzz2X3gH5L62

[Napodano]

My understanding of optimism bias approach is that the revenue flow of a project should be subject to sensitivity analysis, to 'reduce the initial optimistic revenue forecast'. A reduced revenue flow should force the project proponents to consider alternative options to the project which should be less expensive to guarantee an acceptable IRR.

Optimism bias approach applied to increasing costs is just a 'polite' way to allowing for gaming. The only plus is that gaming comes early on in the process and the increased costs can be planned ahead (not a small feat, I admit, because it reduces the chances for unfininshed projects or long-delayed completion).

 

[petagny]

As defined in the UK Green Book, 'optimism bias' can apply to any dimension of a project, but results essentially in a systematic tendency to over-state benefits or under-state costs.

This is how the Green Book puts it and the proposed solution:

'5.61 There is a demonstrated, systematic, tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors. Many project parameters are affected by optimism – appraisers tend to overstate benefits, and understate timings and costs, both capital and operational.

5.62 To redress this tendency, appraisers should make explicit adjustments for this bias. These will take the form of increasing estimates of the costs and decreasing, and delaying the receipt of, estimated benefits. Sensitivity analysis should be used to test assumptions about operating costs and expected benefits.

5.63 Adjustments should be empirically based, (e.g. using data from past projects or similar projects elsewhere), and adjusted for the unique characteristics of the project in hand. Cross-departmental guidance for generic project categories is available, and should be used in the absence of more specific evidence. But if departments or agencies have a more robust evidence base for cost overruns and other instances of bias, this evidence should be used in preference. When such information is not available, departments are encouraged to collect data to inform their estimates of optimism, and in the meantime use the available data that best fits the case in hand.

5.64 Adjusting for optimism should provide a better estimate, earlier on, of key project parameters. Enforcing these adjustments for optimism bias is designed to complement and encourage, rather than replace, existing good practice, in terms of calculating project specific risk adjustments. They are also designed to encourage more accurate costing. Accordingly, adjustments for optimism may be reduced as more reliable estimates of relevant costs are built up, and project specific risk work is undertaken. Both cost estimates and adjustments for optimism should be independently reviewed before decisions are taken. Annex 4 provides further detail on how to deal with optimism bias.'

Benefit streams have also been questioned with respect to the HS2 project. Here's a recent article in the FT (together with accompanying picture of a white elephant!):

http://www.ft.com/cms/s/0/79412d4e-e276-11e2-87ec-00144feabdc0.html#axzz2XpRxJdo4

The assumption used in the CBA was that time spent travelling on a train was unproductive and therefore any time savings could be put to productive uses. The reality is that business travellers with their laptops and smartphones can nowadays use time on a train productively. (Personally, I can't stop myself falling asleep, but apparently others are much more disciplined!). According to the article, the transport ministry admits that the values of time used are based on survey data that is 10 years old and a lot has changed since 2003!

Also according to the article, the head of the UK's supreme audit institution (the National Audit Office) suggests that the estimates of the value of business travellers' time are an example of optimism bias. Where doe the boundary between weak analysis and optimism bias lie, I wonder?

Here's another article from a member of the previous government, which first approved the HS2 project. He now believes that this was a mistake and that the objectives of the project could now be met through other more cost-effective means.

http://www.ft.com/cms/s/0/5db4c212-e301-11e2-bd87-00144feabdc0.html#axzz2XpRxJdo4


The article indicates the flimsiness of the case upon which the original decision was made. It also describes the political context and illustrates nicely how a big infrastructure project can gather almost unstoppable political momentum. In this context, it becomes very difficult for planners to be less than optimistic! Here are some selected quotes from Peter Mandelson's article

‘…I now fear HS2 could be an expensive mistake.

…In 2010, when the then Labour government decided to back HS2, we did so based on the best estimates of what it would involve. But these were almost entirely speculative. The decision was also partly politically driven. In addition to the projected cost, we gave insufficient attention to the massive disruption to many people’s lives construction would bring…

‘Remember the context… We were on the eve of a general election and keen to paint an upbeat view of the future. Such publicly built infrastructure projects seemed to provide so much of the answer to our short- and longer-term economic and employment needs…

‘But in truth, this was about the limit of our collective cabinet consideration. We were focusing on the coming electoral battle, not on the detailed facts and figures… The vision was exciting…and the cabinet adopted HS2 as a “national cause”, competing with the then Conservative leadership whose enthusiasm for the project had predated our own.’

[Napodano]

Attached is the UK Green Book which I have found.

Petagny, do you have a newer version?

[petagny]

This is the version I have. The Treasury (UK finance ministry) is currently undertaking consultations for a forthcoming revision.

[John Short]

Letter to Times on 4th July from Adam Mills who was "one of the few people in the UK who have had direct responsibility for running a high-speed railway":
"I inherited forecasts based on this rationale (time equals money).  It rapidly became apparent that such forecasts were away with the fairies.  I spent two years trying to defend the indefensible..........................."

[petagny]

In a report titled, 'High-Speed Gravy Train', the Institute of Economic Affairs (a UK think-tank) now suggests that the cost of HS2 could reach as high as £80 billion.

This is the summary:

*The decision to build High Speed 2 is not justified by an analysis of the costs and benefits of the scheme. Even the government’s own figures suggest that HS2 represents poor value for money compared with alternative investments in transport infrastructure.
*
Ministers appear to have disregarded the economic evidence and have chosen to proceed with the project for political reasons. An analysis of the incentives facing transport policymakers provides plausible explanations for their tendency to favour a low-return, high-risk project over high-return, low-risk alternatives.

*A group of powerful special interests appears to have had a disproportionate influence on the government’s decision to build HS2. The high-speed-rail lobby includes engineering firms likely to receive contracts to build the infrastructure and trains for HS2, as well as senior officials of the local authorities and transport bureaucracies that expect to benefit from the new line.

*An effective lobbying campaign in favour of HS2 was initiated and funded by concentrated interests expecting to make economic gains from the project. This effort appears to have been effective at marshalling support for the scheme among policymakers.

*‘Vote buying’ incentives were also important in building political support for a high-speed line. The policy was initially adopted partly as a response to local opposition to Heathrow expansion.

*The main losers from HS2 - the taxpayers in every part of the UK who will be forced to fund it - are highly dispersed, and therefore have weak incentives to actively oppose it. By contrast, members of communities along the route, where losses are concentrated, have had very strong incentives to campaign. This pattern of activity has enabled the debate to be misleadingly framed in the media in terms of local objections versus national economic benefits.

*Policymakers have strong incentives to ‘buy off’ opposition along the route at the expense of taxpayers, for example by increasing the amount of tunnelling or diverting the line. The large scale of HS2, its high political salience and its potential electoral importance, increase the risk that budgets will be expanded.

*Local authorities, transport bureaucracies and business groups are already lobbying central government to fund new infrastructure along the route, with several schemes already identified. HS2 will trigger billions of pounds of additional expenditure on commercially loss-making, taxpayer-funded projects.

*Along with design changes to ‘buy off’ opposition and subsidised regeneration projects, these proposals threaten to push total spending far beyond the basic budget. £80 billion plus is a plausible estimate of the overall cost, if these extras and the trains are included.

*In addition to the direct costs, there will be even larger opportunity costs from the misallocation of transport investment. Institutional reform is needed to reduce the malign influence of rent-seeking special interests on transport policy. New infrastructure could then be provided on a more economically rational basis.

[petagny]

An article in today's FT indicates that Treasury officials are 'privately' worried that the cost of HS2 will rise as high as £73 billion.

http://www.ft.com/cms/s/0/1f331cde-0988-11e3-8b32-00144feabdc0.html#axzz2cQ2K5Qul

But this a 'financial' estimate - including VAT and inflation  - and is not therefore strictly comparable with the recently revised official figure of £42.6 billion, which is expressed in 2011 prices (and probably not VAT). The cost of new rolling stock is on top of this figure.

By circulating a headline catching higher figure, are officials trying to encourage a re-think?

[petagny]

Apparently, the body representing the UK's business leaders -  the Institute of Directors -  has now called HS2 a 'grand folly'.

If you remember, a large part of the ongoing discussion centres around whether time spent travelling in work time is lost (and therefore any savings can be put to productive use). The assumption in the current cost-benefit analysis is that 100% of time spent travelling is lost to productive use. The IoD, carried out a survey of its membership on this issue and according to the Guardian:

'The survey of IoD members found only 27% felt HS2 represented good value for money and 70% thought it would have no impact on the productivity of their business. The IoD said in a statement that the government assumption that time spent on a train was unproductive for business was "wildly inaccurate" as only 6% of directors said they never worked on a train. By contrast, 48% of members claimed they spent at least half the journey working, 26% worked for between a quarter and half the time, and 21% up to a quarter of the journey.'

To be fair, the CBA is currently being revised to reflect the impact of new technology on working practices. It will be interesting to see what the results are!

http://www.theguardian.com/uk-news/2013/aug/27/hs2-high-speed-rail-project-grand-folly

[petagny]

Earlier this year KPMG published a report claiming £15 billion in regional economic benefits for HS2. The scale of the suggested benefits has been questioned by a number of academics specialising in spatial economics and this issue has now been discussed by the Treasury select committee as reported in the Guardian:

http://www.theguardian.com/uk-news/2013/nov/05/hs2-report-overstated-benefits-expert

The Guardian may, in turn, have exaggerated the scale of the exaggeration (the headline is 'HS2 report overstated benefits by six to eight times, experts say'). Here's how Professor Overman questioned the benefit estimates on the LSE's Spatial Economics blog:


http://spatial-economics.blogspot.co.uk/2013/09/hs2-regional-economic-impact-garbage-in.html

'...modelling this kind of regional economic impact is very difficult. That said, assuming I have understood it correctly, this report does things that are technically wrong (and these things are crucial for the analysis)...

So, on my reading, technically wrong and possibly out by orders of magnitude. I can imagine why the government has rushed this report out, but it would appear to add very little, if anything, to the debate.'

There is, however, a potential impact from better connectivity that deserves to be measured properly. As Overman says in another blog, 'better connected places have higher productivity'. It's just that KPMG have used the wrong methods and bumped up the figure by an order of magnitude. Interestingly, Overman's research suggests that the 20 minute time saving from Birmingham to Manchester/Leeds (the second phase of HS2) could have a bigger impact than the 40 minutes saving from London to Birmingham.

http://spatial-economics.blogspot.co.uk/2013/09/the-regional-economic-impacts-of-hs2.html

There is also the question of additivity: these are estimates of regional benefits that cannot necessarily be added to national benefits, since one region's gain might be another region's loss. They are nevertheless interesting for policy makers interested in greater regional balance.  Raised productivity (reflected in increased wages) also does not come free: private investment will be required to raise productivity, beyond the public sector's investment in HS2. This also needs to be taken into consideration.

[pfmkaro]

Napodano, petagny, john short and others

I am looking for one or possibly two experienced and thoughtful PIM practitioners to serve as resource persons at a workshop on PIM in developing countries sometime in March/April location tbd.  If interested, could you contact me at aramkaro@yahoo.com ?   Since this topic has been little visited I may repeat this message in a more recent posting. 

pfmkaro

[Napodano]

Hi, Pfmkaro;

I suggest you make another post in the section 'company promoting' at http://pfmboard.com/index.php?board=20.0 .

In my view Petagny and Scandizzo http://pfmboard.com/index.php?action=profile;u=8 are the right candidates.

[pfmkaro]

Napodano

Thanks for the quick response and the suggestion of who would fit the profile best.  Much appreciated.

Pfmkaro