Conversation with Marc Robinson on performance budgeting

[Napodano]

Dear PFM Boarders,

Our past fire-side online conversations have been one of the most successful features on the Board (recording around ten thousand views).

The main reason is no doubt the international caliber of our interviewees. This is also the case for  our fifth fire- side conversation which gives the opportunity to members to talk to Marc Robinson. He is a former IMF Fiscal Affairs Department economist, now a successful PFM consultant. He is also an influential blogger on PFM through his blog http://blog.pfmresults.com/wordpress/  and a regular contributor to the IMF PFM blog at http://blog-pfm.imf.org/ .

One of his recent works included a collaboration with the CLEAR Initiative (http://www.theclearinitiative.org/ ) for which he developed the Manual on Performance-Based Budgeting (attached below - you need to register to download it).

If you are a registered member of the Board, you have the possibility to pose a question to Marc online. The focus of the conversation with him is 'performance budgeting – its role in the overall PFM reform and its practice in transition and development countries ’.

The time allotted for questions is  Monday 10th to Wednesday 19th September 2012. After this period the interview will be closed and remain in the archive for future reference.

[Napodano]

Hi, Marc;

let me start our conversation with two questions:

Question #1
What does it take to have a performance in budget? I have seen many levels of indicators besides the output/outcome ones, are those necessary elements of a performance budget? Can an activity-based budget be considered a performance budget?  When is appropriate to talk about unit costs?

Question #2
Recently there are have been many voices against performance budgeting,  due to its perceived complexity and unwielding results in countries where there have been applied (see for instance http://blog-pfm.imf.org/pfmblog/2012/09/performance-budgeting-facing-up-to-the-hard-questions.html ). What do you reply to these critics? Would you recommend developing countries to launch budget reforms that include the introduction of a performance framework?

[Glen Wright]

I think this will be a very interesting discussion.   I did look through the PB manual and find it quite useful in defining the PB approach.  Just to follow on with my main concerns about this approach which has been confusing to me.

Question # 3
First is the question of how interlinked does Program Budgeting and Performance Budgeting have to be.  It seems to me in the literature these two approaches have been treated as two separate, and maybe equal approaches.  Sometimes I am not sure whether an author is describing program budgeting or performance budgeting. This has gotten more complicated with the introduction of medium term budgeting and how program budgeting and performance budgeting fits into this approach.  I notice that in your manual you begin with description of performance budgeting and it is not until chapter 8 that you start to describe program budgeting.  This makes me think that performance budgeting is a level above program budgeting.  To my mind you need first the program budget structure before you can get to performance budgeting.  Can you clarify this relationship of performance budgeting to program budgeting and how they should be developed.

Question # 4
Second,  just based on your experience what examples of well developed performance budgeting can you identify, either on country basis or a sector level basis that would be useful to examine.

[Marc Robinson]

Hi colleagues,

Thanks very much to Mauro and PFM Board for the opportunity to discuss these important topics with so many other PFM professionals.

I believe that performance budgeting has a great deal to offer, but at the same time it mustn't be seen as a magic solution to either budgeting or performance problems. Rather, it is part of a broader set of reforms, and will only work with them.

A key first point is that performance budgeting is not just about the use of performance indicators in the budget. Performance indicators, no matter how good they are, do not tell you conclusively whether programs are effective or not, nor do they conclusively demonstrate the degree of efficiency of a program or process. Performance indicators simply provide the raw data for performance analysis – starting point for program evaluation, policy analysis, benchmarking et cetera. It is entirely to be expected that those who believe that simply injecting a whole lot of performance indicators into the budget would change things radically were going to be disappointed. Without the development of better evaluation and, more generally expenditure analysis – making use of good performance indicators – it is inevitably difficult to link budgeting and performance systematically.

In short, performance budgeting is about the use of performance information in general – including evaluation and other forms of expenditure analysis  – in the budget, and not just about performance indicators.

What has made the disappointment about the budgetary use of performance indicators even greater has been the tendency in quite a few countries to include as "program" indicators a whole lot of indicators which are not very useful for budget decision-makers because they are more about internal ministry management. For budgetary purposes, it is primarily the top-level indicators – outcome and output indicators – which are really useful, not a mass of input and process indicators. It is, for example, of little use of the budget preparation to attach to programs indicators such as the number of meetings held, the number policy documents developed, the number of positions filled (or vacant) etc. Yet this is precisely what many countries have done in developing so-called performance budgeting systems.

The other key thing is that even when you've developed good performance information which really tells you something about the effectiveness and efficiency of programs, you can't assume that this information will automatically be properly used on budget preparation. You have to create the channels and routines by which this information is fed into the budget preparation process and actually used. In my view, one of the most important "missing links" here is spending review -- that is, a routine process for examining baseline expenditure as part of the budget process. It is in a spending review process that the performance information generated by performance budgeting can be most effectively used. It is good that at the present time so many countries are focusing new attention on developing spending review processes. Because in the past, budgeting has all too often just focused on the consideration of new spending proposals – under which circumstances it is hardly surprising that performance budgeting systems have not appear to have made too much impact.

[Marc Robinson]

Let me also please comment on a couple of the other interesting points which have been made.

Firstly, Glenn is right in saying that there is much confusion about the relationship between program budgeting and performance budgeting. I think the best way of looking at this is to set performance budgeting as the general concept, and program budgeting as one form of performance budgeting. Performance budgeting is, generally defined, the systematic use of performance information in the budget to improve the effectiveness and efficiency. Program budgeting uses one particular approach – a budget classification based on outcomes and outputs – to make use of performance information in the budget. But there are, as I've made clear in my writings, a number of other approaches to performance budgeting.

Secondly, the question of unit costs in performance budgeting: The idea that budgeting can be based on unit costs is one of the other forms of performance budgeting I've just referred to. The idea, roughly speaking, is that you can budget by firstly identifying all the types of outputs – that is, the specific services provided to the public – which ministries deliver, and then multiplying the unit cost of each of these types of outputs by the quantity which you plan to deliver during the budget year. For example, if you want to vaccinate 140,000 people this year, and vaccinations cost $20 each, then you give the Health Ministry $2.8 million for this output.  The problem with this approach is that it only works for a (quite narrow) sub-set of the services (outputs) which government delivers to the community, and that there are many types of services which can't be budgeted for in this way because they have no stable unit cost. I like to give here the example of criminal investigations by the police – the cost of one murder investigation can differ radically from the cost of another investigation because of case complexity, and there is no stable predictable average or unit cost for murder investigations. So while unit costs budgeting is a good performance budgeting model in some specific sectors – such as public hospital funding under the so-called "diagnostic related group" funding system – it does not provide a model for performance budgeting across the whole of government. I think it is unfortunate that consultants with a poor idea of how the public sector really works have all too often peddled the idea of unit costs as the best and universally-applicable model of performance budgeting.

[iancdavies]

Dear Marc,

A number of multilateral organisations such as ILO, FAO, Council of Europe, etc. have developed "results based budgeting and/or management" approaches for their own governance and management. They typically use logic modeling techniques, e.g. program logics, logical frameworks, results hierarchies, etc., to identify all manner of results, e.g. outputs, outcomes, impacts, etc. and corresponding indicators. In doing so they accept, most often without realising it fully, a causal performance construct as a basis for valuing their work.

Problems arise when these organisations and/or their activities are primarily normative and political, e.g. the ILO is about working towards towards social justice, promoting social dialogue, establishing norms and standards, building capacity, etc. The dominant performance budgeting and/or management paradigm is causal in nature and appears poorly suited to capturing, valuing and communicating meaningfully the worth or value of the entreprise. The risks are twofold:

- first that, if you only do what you can measure with the typical armamentarium of causal techniques you risk mission displacement, i.e. you end up doing something else than what you were mandated for; and

- second, because you are not doing a convincing job of "reporting on results" you risk seeing your funding cut. The latter has happened now to a couple of institutions through the MOPAN initiative and the UK government's (aka DFID) decision to cut or reduce funding significantly as "results are not being shown".

Finally, if there's one thing I learned during my years as a civil servant actually working with budgets, e.g. I was advisor to the President of the Treasury Board of Canada, is that a budget particularly in government is first and foremost a political statement, and it is negotiated (some would say horse traded) at the margin.

Question # 5
Any ideas or suggestions as to how government departments, multilaterals and the like, can value what they do in ways that are more consistent with the nature of their business and their raison d'être? Are there some basic principles and/or guidelines to help use wisely and appropriately use measurement and valuing techniques, to provide sound and evidence informed advice to management and governance functions? Many thanks for your insights.

[Gord Evans]

Marc.  I just skimmed through the attached manual on budgeting, so congratulations on preparing a highly readable and thorough presentation of the PFM Board's favourite topic (strengthening public expenditure management through sound budgetary practices).  Like many of the best efforts to explain how government's determine where to spend their money, it presents the budget as a clear process with distinct steps and time frames orchestrated by the Ministry of Finance. 

Although it certainly acknowledges that deviations from the norm are common, I think there is one gap that is not fully addressed: the constant interaction between the budget and the government's ongoing decision-making system.  From my experience in government (initially in Canada but since in many countries), de facto budget decisions are made every week during meetings with the Prime Minister and in Cabinet Committee and Cabinet meetings, but they are not acknowledged as such - instead, they come forward as draft laws (40-300 per year depending on the country), draft regulations (200-2000), and policy papers (0-150 per year), many if not most of which have fiscal implications. 

Although there is generally a "fiscal implications" requirement included in such documents, these are of notoriously poor quality and under-vetted by the Ministry of Finance except in a few "best practice" countries.  By the time the budget "process" rolls around again, it proceeds as a mad scramble by MoF to figure out how to fund not just the Prime Minister's new pet projects, but the whole slew of now-approved legal commitments which ministries parade forth as "givens" for funding. 

Question # 6
I don't really have any brilliant suggestions at to how one would present the chaotic world of political deal-making and decision-making in a budget manual/training curriculum, but did you give any thought to this issue as it is what tends to happen in most governments? 

[petagny]

Hi Marc,

Question # 7
You will have seen Napodano's reference to the recent article by Dirk Kraan on the IMF PFM Blog. He draws attention to a number of potential difficulties with performance budgeting, but I would be particularly interested in your thoughts on the incentive problems he highlights, namely the issue of perverse incentives and asymmetric information between budget holders and the finance ministry. His comments reminded me of John Kay's book 'The Truth About Markets' in which he is very negative about New Public Management, particularly the New Zealand version where there has been a heavy reliance on 'contractualising' performance. He labels the incentive problem 'incentive (in)compatibility'. Here's a flavour of John Kay's argument:

'Whenever competitive mechanisms are not available, or not used, there are potential problems of incentive compatibility. The common response is to set targets, and reward or punish by reference to the targets. This is the system that worked so badly in the Soviet Union. It badly for two main reasons. One was that the centre did not have sufficient information to set targets effectively. The other was that the targets could only imperfectly reflect the centre's real objectives. These problems arise whenever target setting is tried, in public sector activities such as health and education, in the regulation of utilities or in encouraging executives to maximize shareholder value. The same fundamental difficulty arises in each case. If the target setters had enough information to set targets appropriately, they, not the people on the ground, would be the effective managers of the business. So managers aim to meet targets, not the objectives of the targets. 'Plan bargaining', strategic negotiations between the various parties, and the proliferation of more and inconsistent targets cause cynicism and demoralization.'

[Marc Robinson]

Performance measurement difficulties and performance budgeting

Ian has raised some very important issues about how to link budgeting to performance when performance is difficult to measure. I think this gets back to the basic point that performance budgeting is not just about budgeting based on performance indicators. It is certainly not just about what some – based on a private sector model – referred to as the use of "performance metrics". There are clearly many parts of government – in particular national government – where outcomes in particular are very difficult to measure. The outcomes achieved by the foreign affairs ministry is a classic example, as is the effectiveness of the armed forces during times of peace (obviously maintaining an army deters aggression from others, but how does one measure deterrence?). So pretending that budget funding can be simply linked to "demonstrated performance" in the sense of clear and unambiguous measured improvements of performance is absurd. But this is not what sensible performance budgeting is about.

What performance budgeting, properly conceived, is saying is that budgeting should be as evidence-based as possible. In other words, we should do our best to assess the efficiency and effectiveness of expenditures, and integrate that information systematically into budget preparation. In this context, the program logic framework itself is extremely useful, because a key starting point is simply being clear about what we are trying to achieve with public expenditure.

I go back to the point I made yesterday, that we should not think of performance budgeting so much in terms of inputting performance indicators directly into budget preparation, but rather in terms of using good performance indicators as the basis for much improved expenditure analysis (outcome evaluation, process evaluations, policy analysis etc), and then using this expenditure analysis in budget preparation.

[Marc Robinson]

OUT-OF-CYCLE NEW SPENDING PROPOSALS

Gordon Evans raises the point about governments' tendency to make new spending decisions "on the fly" – in other words, for many decisions about new spending initiatives to be made not in the annual budget preparation process, but at some other stage during the year. He is absolutely right in saying that this is a really major problem for rational budgeting. If new spending proposals are decided on an ad hoc basis at any time, the possibility of comparing alternative proposals in deciding which offer the best value to society within an affordable overall  aggregate expenditure limit is lost. For this reason, a very important theme of budget preparation reform is the importance of absolutely minimising the presentation of new spending proposals outside the regular annual budget preparation process. It will, of course, never be feasible (or even desirable) to entirely ban the presentation of new spending proposals during the budget execution phase of the budget cycle. Unexpected developments which require expenditure response inevitably do arise, and it is not always possible to deal with these entirely through the use of inbuilt budget flexibility (e.g. reserves under the control of the Minister of finance). However, government should develop clear criteria which must be met prior to the consideration of any new spending proposal outside the ordinary budget preparation cycle – including urgency, and inability to foresee the policy problem. The relevant central agencies – either the Ministry of finance, or the president/Prime Minister's office – should then be given a mandate of vetting all "out of cycle" new spending proposals to assess whether they meet these criteria or not. If they do not, they should be rejected. In practice, this is a problem which some countries have dealt with quite effectively – for example, Australia and the United Kingdom – whereas other countries basically permit almost total freedom for spending ministries to put forward out of cycle proposals.

[Marc Robinson]

TARGETS AND PERFORMANCE BUDGETING

Petagny's thoughtful comments raise the familiar issue of the relationship between target-setting and performance budgeting. The first thing to say here is that performance budgeting does not necessarily involve the setting of budget-linked performance targets. It is an empirical question as to how far it makes sense to use performance targets of this type. This is important to emphasize because in program budgeting models which one sees around the world, it is really common to find an assumption that for every program performance indicator, one necessarily must have a performance target. For example, if one uses the new infections rate as an outcome measure for an infectious diseases program, it is supposedly essential to have a target for reductions in the new infections rate over the next 12 months. But this is entirely wrong. There are some performance indicators for which it makes sense to set targets, and other indicators which – although informative about performance – are entirely unsuited to target setting. Outcome indicators are rarely suited to the setting of annual performance targets because it typically takes years to achieve intended outcomes. In some cases, it is not even appropriate to set four or five-year targets for certain outcomes, because government has very limited control over the outcomes concerned – I'm reminded here of the, in my view inappropriate, targets set by the former British Labour government for the reduction of rates of childhood obesity and rates of teenage pregnancy.

Does this mean that target-setting is damaging and misguided? The most vibrant debate on this topic in recent years was conducted in Britain under the Labour government because of the extensive use of budget-linked performance targets under that government's so-called "public service agreement" (PSA) system. John Kay's comments are representative of the major school of opinion which condemned target-setting as the resurgence of Soviet-style economic planning, and which dusted off all of the amusing stories of the perverse effects of Soviet targets (usually drawn from Alex Nove's classic works). These critics also drew on the usual agency theory analysis of the consequences of information asymmetry in a principal-agent relationship. Okay, there is no doubt that they had a point, and that target setting can indeed induce perverse effects. However, it is not sufficient to just argue from economic theory – based, by the way, on naive "homo economicus" assumptions that everyone is simply trying to rip off the system to maxim personal advantage – and the economic history of communism. Nor is it sufficient to recount a few anecdotes about alleged adverse effects of Labour's target-setting. What was really needed was more empirical analysis of the impact of target setting. Unfortunately, there has been too little of this. However, such analysis as has been done suggests that the whole story of perverse effects has been grossly exaggerated, and that target setting did not do anything like the amount of damage that the critics alleged. (My manual refers to some of this literature, but is also discussed in my chapter with Elisabeth Paul in my 2007 book). And if you look at the track record of improvements in measured performance against the PSA targets in the UK, it is actually quite impressive. In my view, target setting under the Blair/Brown government in the UK was, on balance and as far as we can judge based on imperfect information, a success. The critics got it wrong in part because they ignored important effects such as the "public service motivation" of government employees who in so many cases really care about the services they are providing, or the outcomes that they are achieving, and do not necessarily respond to target setting by simply exploiting every loophole offered by imperfect performance measurement.

Having said all of this, I nevertheless advise governments to be cautious and quite limited in the use of centrally-imposed performance targets. Firstly, this is because– as noted above – not all performance indicators are suited to target setting. Secondly, because it makes no sense at all for the government to impose upon its ministries more performance targets then it can actually monitor and act upon.

[John Short]

Marc,

Thank you for your excellent contribution to the PFMBoard.

In his Guidance of Note on Sequencing PFM Reforms Jack Diamond has produced a paper, the aim of which in his words “is to assist in the successful design and implementation of public financial management (PFM) reforms by addressing the issue of appropriate sequencing”. The note is targeted at planners and decision makers in government and donor agencies.  It document summarizes work that began in September 2010, to review the rapidly expanding literature on sequencing public financial management (PFM) reforms in response to a request from OECD DAC for a Guidance Note (GN) to assist donors when sequencing PFM reforms.

The paper is downloadable from http://www.pefa.org/sites/pefa.org/files/guidance-note-on-sequencing-pfm-reforms.pdf  (and attached below)

The Note includes an extremely clear path showing the sequencing PFM reform with the attached chart: Overall Sequencing Strategy for PFM Development

Apart from mentioning the technical issues in delivering the various stages of the sequencing strategy, the paper also notes the systemic risks that may be encountered.  To quote:
"The technical specification of reform actions required to improve the PFM system, (i.e. what is needed), must be consistent both with what is possible (implementable) and what is wanted (politically supported). Bringing all three dimensions into line is the role of change management, a crucial factor in reform success.  The viability of any reform program should be determined by a systematic analysis of the risk and opportunities existing in the specific country environment, and those arising from the nature of the reform actions being contemplated.  External non-technical factors should be recognized as critical for sequencing and must be accommodated in any viable reform program. These non-technical factors can be analyzed by dividing them into three tiers:
1.   top-level conditioning factors, that determine the overall reform environment;
2.   PFM system institutional factors that determine how well reforms will be received and implemented; and
3.   at the lowest level, organizational factors and HR factors that often impose constraints on the extent of reform.
Each level should be analyzed to assess the risk and opportunities they pose for reform.  Based on this analysis, wherever possible action should be taken to reduce risk of reform failure and to enhance support for reform.
Choice of the type of reform action has an important impact on its likelihood of success. A review of the literature identifies five dimensions as influencing the risk of failure:
1.   the number of institutions involved;
2.   the time required for implementation;
3.   scope of reform actions;
4.   the degree of behavior change implied; and
5.   the degree of visibility in the final results.
All are highly correlated: reforms wide in scope usually involve more institutions and greater time for completion, they tend to involve a greater degree of behavior change and often their results are only visible, if at all, with a considerable time lag. Also a trade-off should be recognized: the higher is the risk level of different reforms typically the higher is their potential reform impact.”

I apologise for a long introduction, but in the context of my questions it will save participants in the Conversation from going to the original though it is a paper well worth reading.

While I completely agree with desirability of moving right down the sequencing road and gaining from the benefits of programme/performance budgeting, the sceptic in me suggests that the chart should be titled “The Road to PFM Nirvana” as synonyms for Nirvana include paradise and heaven but also illusion and fantasy!  The road to Nirvana is littered with many good intentions.

Question # 8

My question is twofold:
In your experience from your country work have the costs associated with the systemic risks outweighed the benefits from introducing fully fledged programme/budgets so it may be better to stop at the MTBP stage on the grounds that the pursuit of the excellent is the enemy of the good? (The stage the budget projections change from an information- to a decision-making tool, so next year's estimates have some policy status and would reflect cabinet decision-making on aggregate and line ministry spending ceilings over the medium-term in the Guidance Note.)
and
What time frame has been required to mitigate these risks to reap the benefits when the MTEF stage as described in the chart has been reached? (The MTBF becomes strictly linked to policy and anchored in sector MTEFs and it is presented on a more detailed level, with full integration of recurrent and investment costs of programs projected in the out years. The fiscal strategy may also include non-financial performance ("results based") information linked to program budgets in the Guidance Note).
I realise that not all countries’ experience have been the same, but some ballpark range would be useful!  In my experience the time scale of donor funded MTEF projects is vastly under what is needed and as a result are wrongly judged as they are almost programmed to fail from the start.

[Marc Robinson]

SEQUENCING AND PERFORMANCE BUDGETING

John Short's thoughtful comments have raised the crucial issues of: when a country is ready for performance budgeting, and how should performance budgeting implementations be sequenced relative to other reforms such as an MTEF. He asks about when do "systemic risks outweighed the benefits from introducing fully fledged programme/budgets so it may be better to stop at the MTBP stage"?

Addressing the first part of John's question, we have to do ask ourselves first what we mean by "fully fledged" program budgeting. In my view, to work effectively, a program budgeting system involves three elements – a program classification of expenditure which is used as the basis of legal appropriation of expenditure; the development of a useful body of program performance information (performance indicators, but not only performance indicators); and the use of this information to improve budget decisions about where to allocate limited funds. This constitutes the "bare bones" of a meaningful program budgeting system. It doesn't include any fancy add-ons, such as lots of program performance targets, or unit cost calculations of program funding requirements, let alone activity-based costing, accruals and all the rest. These are at best optional add-ons (and some cases, only to be used selectively or not at all).

While there is absolutely no doubt that there are many developing countries which are not yet ready for the implementation of such a "bare bones" version of program budgeting, it is also clear that such a program budgeting system is not so vastly complicated that only very advanced countries could expect to make it work. This is the reason why I have always rejected the proposition that performance budgeting is generally inappropriate for developing countries.

However, it is very true that we must always keep in mind that there are PFM fundamentals which are more important and more fundamental than performance budgeting, and which should be in place before anyone thinks of implementing performance budgeting. For example, the Ministry of Finance and government have to be able to enforce budget spending limits on ministries, and avoid undisciplined spending overruns or buildups of unpaid invoices. And the budget should not be on a fiscally unsustainable trajectory. One could point to a number of other fundamentals of this sort which should be given priority over performance budgeting in technical assistance to developing countries. I am often quite shocked at some of the countries which bilateral donor organisations seem to think should be introducing performance budgeting now – e.g. the Congo and Zimbabwe.

But the other question which John raises is the sequencing of performance budgeting verses an MTEF. I think it's easy to lose sight of how technically difficult an MTEF can be, and how few developing countries have successfully implemented one. Even the most fundamental elements of medium-term budgeting – good medium-term "unchanged policy" expenditure and revenue forecasts – are technically difficult to develop. But going beyond that to fixed medium-term ministry spending ceilings requires a vastly higher order of technical capacity and budget system development. It is one thing for countries like the UK and France to make commitments to spending ministries about the budgets that they will receive in two or three years time. It is a totally different thing for developing countries to make a commitment this sort, and I am generally speaking very sceptical about the wisdom of fixed multi-year ministry spending ceilings in these countries. Without the ability to make accurate unchanged policy expenditure forecasts, the risks of getting these medium-term fixed ceilings wrong – too low or too high – are enormous. Even more seriously, if the country does not have well-developed mechanisms for expenditure prioritisation, then making medium-term commitments to spending ministries can just make expenditure even more rigid, and make it even harder to reallocate spending away from the low priority to high priority areas.

Overall, it seems to me that implementing a basic program budgeting system is actually less technically forbidding than the development of a MTEF with fixed medium-term Ministry expenditure ceilings.

[FitzFord]

Marc,

The subtle distinctions you make between performance budgeting and performance measurement and program budgeting are important and useful. A couple of years ago I did a study for the World Bank specifically related to Aid programs (where some donors were arguing for resource allocation to recipient countries based on performance), and in this study I reviewed the use of performance indicators in the OECD countries - much of the material came from studies done by the EU. A quick summary of the story is that no OECD country used performance indicators as the basis for resource allocation across ministries or sectors. This despite:72% of the countries include non-financial performance data in thier budget documentation; in 44% of the countries, these data are available for more than 3/4 of the programs; in 71% of the countries, performance data included performance targets although there was a wide variation in program coverage; and, in 65% of the countries, these results are included in the main budget documents and/or the annual financial document. Only 18% of countries reported that they specifically link expenditures to all or most of their output or outcome targets (ie Performance Budgeting) and mostly in health and education. There was an acknowledgement that are good reasons for the reluctance to extend the use of outcome performance data into resource allocation decisions for (predominantly) technical, managerial and political considerations. However, perhaps the most important (but not explicitly acknowledged) rationale for this hesitancy was that this could be inherently counterproductive to the underlying objective of a program. The conclusion from experience was apparently that the best use was as a management tool  to engage with relevant stakeholders. Less than 10% of Cabinets and legislative committtes use the available measures in their decision making.

I think your comments on how and when to situate and apply these measures (which are expensive instruments as you know) in a public sector management system.


Fitz.

[Marc Robinson]

INDICATOR/BUDGET LINKS ONCE AGAIN

Fitz makes a good point. Again, one should not generally expect direct links between performance indicators and budgets. For example, in the case of outcome indicators, if the outcome indicators are getting worse, it does not necessarily make sense to cut budgets -- if, say, we are talking about the school system and literacy and numeracy indicators are getting worse, budget cuts are unlikely to be the appropriate response. So the connection between indicators and budget decisions should be more sophisticated than that. In relation to effectiveness (outcomes), the connection should instead be something like the following:

Step 1: Do the outcome indicators look bad?
Step 2: If so, one should try to establish -- e.g. using outcome evaluation -- whether the program really is ineffective, or whether the indicators are misleading (e.g. because of the impact of "contextual factors")
Step 3: If the program is ineffective, does it appear that it could be fixed (e.g. by redesign, or even via more funding)? Or does it look irredeemably ineffective?
Step 4: Only if it is irredeemably effective would one necessarily wish to abolish the program.

So again, the whole obsession with whether performance indicators are directly impacting on budget decisions is misplaced. Indicators are very important, but as the starting point for the analysis of effectiveness and efficiency, not as conclusive stand-alone guidance on funding levels.

(Note, however, the exception to this in the case of sectoral performance based funding systems -- such as output-based hospital funding -- in which funding is directly linked to (output) indicators.)

[Napodano]

 Hi, Marc;

In response to Glen (question #4) you mentioned UK (under Blair) as reference country which applied performance budgeting, wisely.

Question #9
Can you mention a developing country and an emerging country which can be taken as reference? Any MoF website which you consider worth consulting for indicators of performance?

[Marc Robinson]

EMERGING OR DEVELOPING COUNTRIES

Hi Mauro. Amongst emerging countries, South Africa has I think done pretty well in developing program performance indicators within the framework of a reasonably sound program budgeting system. Indeed, the South African Treasury's Framework for Managing Program Performance Information (http://www.treasury.gov.za/publications/guidelines/FMPI.pdf , also attached below) (2007) is one of the better program performance indicator guides which have been produced internationally.

The Framework document is particularly impressive for a conceptual clarity and readability which is so often lacking in this type of guide. And I would give 6.5/10 for the actual program performance indicators which have actually been developed by spending ministries, and which can be seen in the Budget Estimates documents (http://www.treasury.gov.za/documents/national%20budget/2012/ene/FullENE.pdf, also attcahed below). The great positive about these South African program performance indicators is that, for the most part, they avoid the mass of activity and input indicators which clog up program budgets in so many countries. At the same time, however, the big remaining problem with them is that there are far too few effectiveness (i.e. outcome) indicators – most of the program performance indicators are output indicators, and more specifically output quantity indicators (i.e. they tell you about the volume of services which are being delivered to the public). I think one point which this underlines is that the government should give spending ministries clear directions on the key types of performance indicators which they are expected to develop for program budget purposes – making it explicit that ministries are expected to develop effectiveness, output quantity, output quality and efficiency indicators.

South Africa's program classification is also reasonably good, if in need of some further work – in particular, to connect more systematically to outcomes as well as outputs.

[harnett]

Marc

Thanks for a truly enlightening discussion on our favourite topic!

To follow on from your response to a useful reference country, I'm looking for best reference work..... the South African example you cite is useful for us practitioners, but as you say - only 6.5/10.  In my work in Albania I was often asked to assist with the organisation of programmes within a ministry and also appropriate outputs (not outcomes which I suggested was often beyond the influence of a single ministry!) and performance indicators to satisfy the requirements of the MTBP submission.  I spent many hours trawling through Australian, NZ, UK and other examples with varying degrees of success. As you indicate earlier in the fireside chat this is easier in some ministries than others (vaccinations easy, deterrence in the MOD difficult).  So...

Question #10

Allowing for the fact that all countries have different circumstances, can you point to any reference work which would provide a starting point for us practitioners when advising on programme structure, outputs and performance indicators

[harnett]

On a different note...

I'm not sure how familiar you are with the UK's PFM but the introduction of performance targets is often blamed for a reduction in goodwill among many public servants - the most prominent of which being education and health professionals (I admit a certain interest in this as an ex secondary school teacher).  During the last 20 years the degree to which teachers perform after school activities and nurses / doctors perform caring services above and beyond their technical duties has purportedly decreased significantly.  We are even now faced with a situation whereby nurses are now to be faced with "caring" targets (don't ask me how this will be measured!) to redress the imbalances of past targets.

Question #11

Any thoughts on how these issues of "goodwill" should be handled?

[Marc Robinson]

Programs, Outcomes and Outputs

A couple of points about Hartnett's interesting observations:
1. Yes indeed, the budget should be appropriating outputs rather than outcomes – in the sense that programs are groups of services delivered by ministries for which the budget is allocating funding.
2. However, at the same time programs should be based on outcomes as well as outputs, because what the groups of outputs (services delivered) which constitute programs primarily have in common is that they have a common intended outcome. For example, a "preventative health" program brings together a range of different types of preventative health services (outputs), but they all have in common that they are aiming to reduce the incidence of preventable disease and injury (an outcome).
3. It is for this reason that the debate about with the programs should be output-focused or outcome focused is beside the point. Program should be both output focused and outcome focused.
4. Program terminology used around the world just adds to the confusion on this point. In Australia, for example, programs are called "Outcomes", but they are still groups of outputs and the use of this terminology doesn't change what you are appropriating for.
5. It is, however, really important that we don't just focus on the outputs – and forget the outcomes – when we define programs and when we develop associated statements of program objectives. This is one of the problems with the South African system which I was getting at in my comments yesterday.

In terms of guidance material, I hope it's not in modest to suggest that my recent manual is quite a good guide on these points – which I think, unfortunately, are not all that well dealt with in manuals developed by most ministries of finance.

[Marc Robinson]

Targets and "Public Service Motivation"

Our colleague Hartnett that also raises the interesting issue of the allegedly negative impact of performance targets on the morale and performance of civil servants including front-line workers such as teachers, nurses and doctors. Yes, I'm very much aware that this is an issue which was much debated in the UK during the Labour years when performance targets were so extensively used. It is the same issue which gets raised in relation to performance pay, and is an issue which I looked at in detail in a chapter in my 2007 book. I would not in any way dismiss the reality of the issue. So-called "public service motivation" – the altruistic commitment of many public sector workers to delivering a good service, or to achieving social objectives which are important to them (e.g. protecting the environment) – is very important to the effectiveness of government, and it is important to manage personnel in such a way as to build rather than erode public service motivation. It is important for this reason not to introduce such a large component of performance pay in remuneration that it swamps other motivations and distorts performance. This danger arises, of course, from the inevitable imperfection of the performance indicators upon which performance pay is based -- some dimensions of performance are not captured in the indicators upon which performance pay is based, and these dimensions of performance could suffer.

However, I completely disagree that public service motivation is a reason for not using performance targets in government. For example, in education I very much applauded the setting by the UK Labour government of targets for raising literacy and numeracy levels – and I note that under the PSA regime measured performance in these areas improved very substantially. The argument that gets raised against this type of target setting is that it is supposedly damages performance in areas which are not measured (e.g. in the case of education in the teaching of values and of subjects other than reading and arithmetic), and this damage is supposedly so great that it outweighs any improvements in the dimensions of performance which are measured and targeted. In my view, this is an empirical proposition that has to be tested -- rather than just asserted on theoretical grounds -- and I do not read the available evidence as supporting it. Expressed differently, while I don't doubt that targets do have some perverse effects, it seems to me that the benefits of rational target-setting can -- and in the UK case, probably did -- outweigh the perverse effects. In fact, I think that part of the reason this is the case is that the existence of so-called "public service motivation" greatly reduces the extent of the perverse effects which may in principle arise from target setting. For example, even if you set cost reduction targets in hospitals (or achieve the same effect by linking hospital funding to treatment volumes), doctors and nurses are not going to immediately go out and sacrifice the quality of patient care in order to cut costs. They will not, in other words, behaved like ruthless "homo economicus", precisely because they do have a genuine commitment to the quality of patient care and patient outcomes.

[Glen Wright]

The example from South Africa is quite interesting.  I particularly like the structure and organization of this as one of the main problems I have dealt with in several assignments is how to develop a very concise, but comprehensive, budget presentation. 

With regard to this issue of outputs dominating outcomes, the main problems I see are that outcomes are not easily counted, often can only be realized several years later that the budgeted funds for them, and are highly variable on a unit cost basis.  Outcomes across programs are not easily comparable for budget decision making, but outputs, based on personnel cost, etc, are more easily counted and compared. 

[Marc Robinson]

OUTCOMES AND BUDGETS

Glen makes excellent points about the measurement difficulties which often arise with outcomes, and the fact that outcomes are often achieved in the medium (or even long) term rather than immediately.However, there are two separate issues we need to discuss here. The first is the question of outcome measurement, and the other is the question of the link between programs and outcomes. The fact that outcomes may be sometimes difficult to measure does not change the need to clearly define the link between each program and outcomes.

For example, school education programs should be defined not just as programs for teaching kids, but as programs for raising the level of knowledge of children. It is not good enough, as I find that education ministries around the world tend to do, to have a statement of program objective which read something like "the delivery of quality education to the nation's young people". Because the failure to explicitly link the program to outcomes, and the tendency to mention only the outputs which the program delivers, leads to a focus in the program performance measures exclusively upon output measures to the neglect of outcome (effectiveness) measures. I am astonished at how few education ministries under program budgeting systems have included basic effectiveness indicators such as literacy and numeracy levels amongst their program performance indicators. They seem to consistently prefer to rely exclusively on output measures such as enrolment rates. Yet outcome measures are enormously important, not just in the most obvious ways but sometimes in ways that are not generally understood. For example, in respect to gender equity in schools, education ministries tend overwhelmingly to focus on measures such as enrolment or attendance rate differentials (i.e. the ratio of female school enrolment rates/male school enrolment rates). Well, this is a useful measure. But an outcome equity measure – namely, the literacy level of female school students/male students and similar sorts of indicators – are more useful and important. So leading country such as France and the UK use precisely these types of outcome-based equity measures.

So instead of the objective of the school education program being defined as "the delivery of quality education to the nation's young people", it should be defined and outcome-focused terms as "well educated and socialised young people". (Incidentally, the widespread inclusion of references to "quality" education in school education program objective statements does not make these statements outcome-oriented. Quality is an output concept, not an outcome concept).

The other issue to which Glen draws our attention  is the impossibility of budgeting on a unit costs basis for outcomes. He is absolutely right – there is no stable unit cost for, for example, the number of lives saved in malaria treatment. However, I would draw attention once again to the limitations upon unit cost budgeting even in the context of outputs, which I discussed earlier in this conversation. In short one can only calculate budget requirements based on output unit costs for types of outputs which are either quite standardised (every client gets pretty much the same service, at the same cost) or where variations in client costs average out over a large number of clients (which is broadly the case of many types of hospital treatments). At the national government level, only a minority of services delivered by government meet this criteria and are therefore candidates for budgeting based on unit costs. It is in my view an unfortunate element of the technical assistance which has been given to many developing countries that these countries have been told that they should use unit cost methodology across the board to estimate the budgetary requirements of their programs.

[FitzFord]

Marc,

I like where this discussion is going. I don't know if you have discussed this elsewhere: it seems to me that it would be useful to separate the timeframe for assessing outputs and outcomes, but nevertheless, link them. If we continue with the education example, as you and my colleagues in this discussion have noted - education has mutiple outputs and outcomes and they manifest themselves in different ways and timeframes.

Question #12

A more complete system would be to specify the outputs and timeframes and the (hypothesized) outcomes and timeframes. In this framework, the evaluation systems could be more appropriately linked, planned and applied, and we would learn more efficiently and accurately what is working and why. Has this approach been applied anywhere? To any sectors?

Fitz.

[Napodano]

With the last post from Fitz, we conclude the conversation with Marc.

Marc,
please answer the last question and I will lock and archive our conversation.
I take this opportunity to thank you on behalf of all PFM Board members. Please come back to the Board to share your knowledge whenever you feel like it.

Let me remember our members to follow Marc on his blog http://blog.pfmresults.com/wordpress/

All the best to you all.

[Marc Robinson]

Fitz, a good question. I think it's something we all need to work on. And in differentiating the timelines of outputs and outcomes, we need also to bear in mind that -- in general -- higher-level outcomes are realized over a longer time-frame than intermediate outcomes. In education, student literacy and numeracy levels are intermediate outcomes which are realized relatively quickly (over several years), whereas stronger economic growth and lower unemployment are higher-level -- and longer-term -- outcomes of education.

In signing out, let me thank Mauro for organizing this excellent discussion, and for doing so a great job with PFM Board. PFM Board is a wonderful contribution to the PFM discipline.

[John Mercer]

One of the key points discussed here is that the inclusion of performance goals and results in a performance budget is just the starting point, and that to be meaningful that data has to be a basis for more in-depth performance analysis. I agree, because I think the real question then becomes, “Why?” This question should apply whether or not a program’s goals were met. It is important to understand the underlying reasons as to why a program is or is not operating efficiently and effectively in achieving the desired results.

This may be easier at the output level, where measures of program activity and their direct results are often used – e.g., number of safety inspections conducted. If a certain number of work hours of inspection are engaged in, then the goal for number of inspections should be the direct result. It is more challenging (and often much more so) at the outcome level, especially when you are addressing end outcomes – e.g., reduction in the number of accidents and their seriousness. Whether or not the target for the number of accidents was met by a safety inspection program, you still need to know to what degree there is a meaningful relationship between the number of inspections conducted and the number of accidents that occur. This then can raise questions about the nature of the inspections themselves (e.g., how thorough and complete), as well as requiring an examination of other factors that impact the number of accidents – which are often outside the control of the particular program or even of the government itself.

When I originally drafted the Government Performance and Results Act of 1993 (GPRA), I focused on requirements that US federal government agencies develop and publish annual performance goals and the associated results. I also included a requirement that when a goal is not met, the agency must explain why not. However, there was no specific requirement that this information actually be used to shape policy, formulate budgets, and manage programs. Silly me, I actually thought this would be unnecessary to require because of course it would happen routinely.

But it did not happen. In fact, several years ago we started hearing complaints by the Office of Management and Budget (OMB) that while federal agencies had been getting better at complying with GPRA requirements for outcome and output measures, they were not actually using the resulting data to affect policy and managerial decisions. So OMB recently worked with Congress to develop and enact an update to GPRA, called the GPRA Modernization Act, which includes a requirement for each agency to conduct Quarterly Performance Updates that specifically address the performance data relating to its priority goals. Sometimes what may seem like just common sense to some people apparently requires a statutory mandate to move others.

[John Mercer]

Another of the issues addressed here is whether performance measures should have targets attached to them. In other words, should it be sufficient to define the performance indicator (e.g., “The number of accidents”) and then simply report the result each year, or should there be measurable target amount attached as part of the goal (e.g., “50 or fewer accidents”).

From my own experience, I would say that defining the right indicator (or set of indicators) and generating accurate performance data is far more important than identifying a performance target level. If you get those two items right, then at least over time the organization will probably show improved performance – particularly if it is subject to regular oversight/scrutiny. And this is especially the case if incentives for improved performance are provided – e.g., better personnel performance reviews for program managers and staff, greater ease in getting budget requests approved, standing out in comparison to similar organizations, etc.

However, it is probably unrealistic to argue that when money is appropriated to a program, there should be no indication of what will be achieved with it. “What will we get for our money?” is not an unreasonable question. Legislators will ask it and the public may come to expect it. Nonetheless, it is the setting of the performance target itself that often generates real fear of and resistance to PBB within many organizations. And I have seen instances of where the actual target was seemingly “picked out of the air” (such as in Congress a few years ago, for the purpose of embarrassing the Administration when it would not be able to meet the statutory target of a 50% reduction in illegal drug use.)

One way to “compromise” this issue would be to not set targets for some (or most) measures during the initial years of PBB, and instead wait until after the first several years so that the organization gains some experience and comfort with program performance measurement. And at that point, with the trends in program performance having been identified, the setting of longer-term and annual targets linked to spending levels can then be done perhaps a bit more realistically and effectively. I recognize that I have just described may be the “ideal” in some respects, but it may not reflect political reality in many situations.

I want to add another thought relating to measuring improvement in program performance. It is that there are at least two ways to think of how to measure improved performance. One is that for the same amount of money expended, the level of performance is increased the next year. Another is that for the same level of performance, the amount of money expended is decreased the next year. This choice (or a blend of the two) can also relate to the setting of a performance target – i.e., “This level of performance is already high enough to be satisfactory, but now get you get it for us for less money?”

[Napodano]

Thanks, John;

just a bit of background for the other members.
John made these comments on a LinkedIn group (http://www.linkedin.com/groups?gid=4518607&trk=myg_ugrp_ovr )where I posted the link about this conversation. I asked him to share them with us at the Board and here they are.

Thank you again, John and welcome to our community.

[Marc Robinson]

Dear John,

Firstly, my apologies for the delay in responding to your thoughtful comment.

I agree with you very much in respect to your core point about not rushing the setting of performance targets. It is crucial that, before the setting of any targets is contemplated, a performance indicator is in place for some time so as to permit the establishment of a robust performance baseline and, as one element of this, to ensure that the indicator is being measured reliably. Indicators are, as statistics, subject to some degree of randomness and only with several years of experience is one able to distinguish a baseline from any degree of random "noise" which might be present.

However, I remain firmly convinced that – even in the long term – it is inappropriate to set performance targets for all performance indicators. In particular, it seems to me that there are some outcomes which it is extremely important to measure but over which our control is so imperfect that it is simply self-deluding to set think that we can set a target such as a 30% improvement over five years. I think of examples such as cancer rates in childhood obesity rates. Of course the objective is to produce these disease rates, and to reduce them substantially. But are we really in any position to set precise quantitative targets for this? I know that some countries do this (an example being the UK under the former labour government), but I think that such targets are inescapably arbitrary and setting them serves little purpose.

A further point I would make concerns centrally-imposed targets, as distinct from targets which spending ministries or other government agencies may set internally for themselves. I believe that there is a strong case for strictly limiting the number of centrally-imposed targets because, if too many such targets are set, it becomes impractical for central agencies (e.g. the finance Ministry or President/Prime Minister's office) to monitor performance against these targets and to intervene appropriately when there is an unjustifiable failure to meet the target. Far better to follow the UK Public Service Agreement example of limiting centrally-imposed targets several hundred than to have thousands of performance targets in the budget papers – as some African countries do – but to have no follow-up for them. This suggests limiting centrally-imposed targets to a selection of the most important performance indicators.

One final somewhat different point: another common practice in developing countries to which I take strong exception is the notion that for every performance target, targets should be set on an annual basis. In other words, if one sets a target for something, then there should be a target set for the variable in twelve months time, twenty-four months time et cetera. Again, it seems to me that this is part of a mentality which exaggerates greatly our degree of control over many of the variables concerned.

Overall, while I personally accept that target setting is a valuable performance management tool, I think we have to avoid delusions of a Soviet-planning approach on the matter.