Public sector performance: efficiency or quality?

Citation

Micheli, P., Mason, S., Kennerley, M. and Wilcox, M. (2005), "Public sector performance: efficiency or quality?", Measuring Business Excellence, Vol. 9 No. 2. https://doi.org/10.1108/mbe.2005.26709bab.001

Public sector performance: efficiency or quality?

Public sector performance: efficiency or quality?

Over the last three decades, in what has become known as the “new public sector” (Hood, 1991) services in advanced economies have been reformed to provide better quality, while reducing demands on taxpayers. Concepts once belonging almost exclusively to a private-sector environment have been transferred to a public-sector context, e.g. economy, efficiency, effectiveness, transparency, accountability, choice and value for money.

In the UK, the New Labour administration has put great emphasis on the provision of public services. A number of influential reports have been recently published in this area and have created a lively response from official bodies and the media.

This research article discusses three reports:

1. 1.

   “Public aervice productivity: health” (Office for National Statistics, 2004), published in December 2004 on National Health Service (NHS) productivity, issued by the Office for National Statistics;

2. 2.

   The Atkinson Review (Office for National Statistics, 2005), published at the end of January 2005 on the measurement of Government output and productivity for the national accounts;

3. 3.

   The Gershon Review (HM Treasury, 2004), known also as the “Efficiency Review”, published in July 2004 – an independent review of public sector efficiency.

Although these reports are very specific in their content, they received considerable media coverage, particularly in the case of the Gershon Review, which has been adopted to drive efficiency savings throughout the public sector. The analysis of the reports and the related press coverage raise the issue of how to disclose information regarding public sector organisations performance, if at all. In the case of reports, such as the ones this paper considers, there are issues very much related to the way information is communicated to the public. Therefore, it is important to realise the fundamental role played by the media in filtering and conveying the message to the audience.

First of all, the analysis of the press coverage associated to the reports shows that very often journalists quoted comments made by politicians, rather than citing the original source of information. Second, the article titles are written in order to draw the attention of the readers, rather than to convey relevant information. Third, it could be argued that the contents of the reports are usually not accessible by a wide audience, given their level of detail. Therefore, they should be directed just to specialists and civil servants or, at least, appropriately explained.

Although these three characteristics are almost ingrained in the way media operate, they seem to have been very much neglected by the people who commission and carry out these reports. As in the case of the report on productivity in health, the way the information is presented to the public should be considered as a determinant factor in the choice on whether to publish very sensitive documents or not.

The analysis of the reports also shows how relevant recommendations on methodology have been neglected. In October 2003, before any of the three reports was published, the Royal Statistical Society released a document (The Royal Statistical Society Working Party on Performance Monitoring of Public Services, 2003) where it provided very insightful comments on how to measure performance in the public sector and on related issues. The concern of the authors of this report was that “Performance monitoring done well is broadly productive for those concerned. Done badly, it can be very costly and not merely ineffective but harmful and indeed destructive” (The Royal Statistical Society Working Party on Performance Monitoring of Public Services, 2003, p. 2).

The objective of this article is to show how, taken together, the reports give a fuller picture of the situation and tell a rather different story than is taken from looking at them independently as has predominantly been the case to date. Furthermore, the main focus of these reports is on productivity and efficiency, whereas quality is often neglected or seen as a complication. Therefore we will argue for an alternative point of view, juxtaposing the present focus on efficiency with issues more related to quality of service, and broadening the context to provide a wider perspective. This is analogous to a well-established principle in the field of information technology – the productivity paradox –, which describes the need to consider the issues of quality and productivity together.

Public service productivity: health

The accurate measurement of the performance of national systems is fraught with difficulty, especially in the case of health, where conceptually complex outputs are delivered utilising a wide variety of inputs. The measurement methodologies used are often complicated, and analysis of the results requires careful consideration of the accompanying caveats and assumptions, making the unbiased reporting of any conclusions problematic.

In the past the Department of Health measured productivity on the basis of input-output analysis through the Cost Weighted Efficiency Index, which could not capture all quality improvements. In its report on productivity in healthcare, the Office of National Statistics (ONS) recognises that this approach is flawed and attempts to create a more accurate measurement methodology, questioning the concept that a health system can be assessed from a purely financial viewpoint. In this sense, the ONS report states that the “aim of most health systems is a variant of ‘improving and maintaining the health of the population served’” (Office for National Statistics, 2004, p. 41).

Hence, it is acknowledged that the output of a health system must be measured in a number of dimensions for any measurement to give a balanced and comprehensive view. These may include health outcomes, social outcomes, quality of life, and social equality, as well as financial measures. Clearly, using financial costs of outputs as a proxy for such complex constructs is inappropriate. Even if outputs could be defined and measured as a single number, it would be nonsensical to express it as a ratio of financial input measures. The ONS report recognises this and it is littered with alerts urging restraint in interpreting the results:

These [productivity] estimates should be interpreted with care. The output figures are based on a subset of activities in the English NHS and do not include changes in the quality of NHS output. The inputs figures are also not ideal, as direct quantity measures would be preferred, and the existing method involving measurement of current price expenditure and using indices to deflate to quantity measures for England only needs improvement (Office for National Statistics, 2004, p. 52).

However, despite all these caveats, the ONS study then pursues the very financial cost approach it criticises: to simplify the analysis purely financial elements are included in the productivity ratio produced by the report. Therefore the main conclusion of the report is that NHS inputs have grown by between 32 and 39 per cent while output has risen by only 28 per cent – a substantial decrease in efficiency. Even this oversimplified analysis faces significant challenges, as calculating reasonable estimates for the financial inputs and outputs of the NHS is also very complicated. However, this tag line has been seized upon by both the media and politicians, often without reference to the aforementioned methodological caveats.

Hence, if the previous conclusion on productivity is isolated from its context – i.e. all the warnings contained in the report are neglected – the title “The NHS is ‘wasteful and inefficient’” (BBC online, 19 October 2004) seems absolutely appropriate. The same could be said regarding the comment: “on every one of the measures chosen by the statisticians, NHS productivity has declined under this government” (Oliver Letwin, reported by the BBC online, October 2004).

Isolating the conclusion from the report, however, gives a misleading view.

Leaving criticism or support for the methodology aside, the question that is most pertinent here is whether it is sensible to publish a report around which there is so much controversy, both methodologically and politically. The danger of its publication is that it “badly” estimates the change in cost based productivity, in itself a coarse and misleading measure, in such a way as to provide political capital, while presenting the caveats to the analysis in a way that is easily overlooked by laypersons, despite the laudable strength of the ONS warnings.

This highlights the quandary of performance reporting in both the public and private sectors: aggregated measures of complex system performance, especially single ratios, can never capture the “reality” of performance. In the case of the report on health productivity, using what is available or easy to obtain is worse than no analysis at all.

The Atkinson Review

The Atkinson report clearly states its objectives: “This report is about methodology and does not contain any new figures with regard to government output or productivity; the aim of the review is to establish the future strategic direction for work in this area” (Office for National Statistics, 2005, p. 2). The issues identified in the previous section on productivity in health show how important it is to develop appropriate measures for government outputs and productivity. The Atkinson Review begins to address them, using the following terms of reference (Office for National Statistics, 2005, p. 1):

To advance methodologies for the measurement of government output, productivity and associated price indices in the context of the National Accounts, recognising:(a) The full scope of government outputs;(b) Differences in the nature and quality of these outputs over time;(c) The relationship between government outputs and social outcomes;(d) The need for comparability with measures of private sector services’ output and costs;(e) The existing work of the Office for National Statistics (ONS); and(f) The appropriate measurement of inputs, including quality and the distinction between resource and capital, so that, together with the measurement of output, light can be thrown on developments in government productivity.

The report presents a wide range of recommendations and principles to improve the way government output is measured in the UK. Quality and productivity, and their measurement, are central issues in the document:

… we are firmly of the view that, in principle, measures of output growth should take account of quality change. Quality has many dimensions, and some will prove elusive, but there are several possible ways forward … If quality adjustments cannot be comprehensive, they should be representative of the range of dimensions (Office for National Statistics, 2005, p. 183).

As commented by the press, the Atkinson Review concludes that the “Office for National Statistics needs to make big improvements to the way in which it measures public services if the government is to find out whether the billions of pounds it has pumped into them have been well spent or wasted” (The Guardian, 1 February 2005).

Although this review has been written mainly for statisticians, it could have a strong impact on the whole public sector in the medium-long term, if its recommendations are taken into account. However, despite its relevance, it has not been sufficiently communicated to civil servants.

Even though the report seems a technical and somewhat dry statistics paper, it received extremely passionate comments. Shadow chancellor Oliver Letwin, for example, expressed strong criticism both at the interim report published last summer and its recent final version.

He firstly said “the idea that quality improvements in the public services can be objectively measured and that such a fuzzy concept should be captured in the measurement of national output is absurd. The role of the Office for National Statistics is to provide reliable and robust economic statistics in which the public has trust, not to act as a propaganda tool for the government” (The Guardian, 20 July 2004).

Subsequently, he reinforced his opinion as follows: “It doesn’t really matter whether Atkinson’s recommendations are intrinsically good or bad. Labour has a dismal track record of waste in government and no one is going to believe a newly created statistic that tells them otherwise” (The Guardian, 1 February 2005).

The discrepancy between the nature of the original document and the comments it received is astounding. As the Royal Statistical Society, the productivity in health, and the Atkinson reports show, the measurement of efficiency, quality and productivity is extremely difficult and potentially harmful. The Atkinson report argues for more accurate and comprehensive ways of measuring public services, and for a shift from purely cost-centred considerations to more comprehensive ones. As a consequence, it also advocates a political debate, which is informed by more reliable data on performance and that takes into account the complexities associated to the delivery of public services.

The Gershon Review

The Gershon Review (or the “Efficiency Review”) was issued in July 2004 – at the same time as the interim Atkinson Review and almost one year after the Royal Statistical Society report. This review focuses on the release of “major resources out of activities, which can be undertaken more efficiently into front line services that meet the public’s highest priorities, and on the need to help inform the results of the 2004 Spending Review” (HM Treasury, 2004, p. 3). In terms of outcomes, it identifies:

• •

  efficiency gains of over £20 billion in 2007-2008 across the public sector, over 60 per cent of which are directly cash releasing; and

• •

  a gross reduction of over 84,000 posts in the civil service and military personnel in administrative and support roles.

The report focuses almost exclusively on efficiency. Nevertheless, it is argued that the efficiency gains will not go against the quality of the service. While discussing the Efficiency Review with the Centre for Business Performance, a government official affirmed that no civil servant could stand up and say that he/she cannot achieve the 7.5 per cent efficiency gains in the next three years, as prescribed by the review. However, both departments and local organisations believe that these savings may not be so easy and quick to attain.

This report on efficiency in public services is now being operationalised by different teams across the country and will have a strong impact on the whole public sector. However, various comments reported by the press challenged the promises contained in it, as the following quote shows:

The director-general of the Confederation of British Industry has cast doubt on the government’s ability to deliver £21.5bn annual savings promised by 2008 (The Guardian, 7 March, 2005).

This scepticism is rooted in the failure of local authorities to meet the £6.75 billion savings target set in 2000: just 5 per cent of the best value efficiency target is currently being achieved. However, the Treasury seems confident that the efficiency savings set out in the Gershon Review will be delivered.

One of the important features of this review is that councils have been “invited” to “offer more contracts to small, local organisations with less bargaining power to help to achieve Gershon Review efficiency savings. The Office of the Deputy Prime Minister wants every local authority to sign a code of practice pledging to make contracts more accessible to small and medium-sized enterprises” (The Times, 8 March 2005).

As to this, it may be argued that local authorities should be empowered in order to build strategic partnerships, rather than having them ordered from above. Although this report will have an impact on all public sector levels, it seems that the centre will always steer what happens locally. This argument is particularly strong in England, where decentralisation and devolution of competencies are mostly theoretical concepts (Eurostat, 2004).

However, the main issue is that this review will be extremely influential, but there are still few operational indications associated to it. Furthermore, as a consequence of the strong focus on efficiency, despite the claims made in the review, the quality of service delivered could suffer from it.

Conclusions: a broader perspective

The reports analysed in this paper raise several issues, many of which are very much related to performance measurement in the public sector.

First of all, there is great concern about the disclosure of information regarding the performance of public sector organisations. Although it is perfectly sensible to inform the citizenship about services delivered, it is also important to note that their evaluation is not without problems. It is difficult to communicate public sector performance and the countless issues related to it, e.g. performance indicators used, reporting of information, statistical attributes of the data, methodological issues, collection, analysis and grouping of data etc. Furthermore, it should always be considered that people usually receive information through the media, which filter it in order to simplify complicated concepts, to phrase sexy titles or to pursue private agendas. As concluded in the case of the health productivity report, sometimes it is better not to publish reports that could be relevant, but also incomplete, skewed and which political parties could purposefully manipulate. In this respect, a recent report by the independent think tank Demos, argues “[T]he press is ‘manufacturing dissent’ in a way that resonates with public concerns and challenges the legitimacy of elected politicians … [D]aily newspapers, which have interests of their own, are not the ideal places to look for dialogue and compromise” (Demos, 2005).

Secondly, the main focus of these reports is on productivity and efficiency. Quality is often neglected or seen as a complication. On the contrary, it warrants description and interpretation and should be treated as a subject for debate, taking into account the issues related to it. As Carl Emmerson, deputy director of The Institute for Fiscal Studies asserted, it is very difficult to measure the output of the public sector, because the quality of services it provides can be measured in many different dimensions. Even if the Government is promising an improvement on productivity in public services of around 2.5 per cent a year for the next five years, in 2010 it will be almost impossible to assess whether this has been achieved (speaking on The Politics Show, BBC1, Sunday 20 March 2005).

In the private sector, quality has been crucial in the last few decades. The objective of this paper is not to provide an in-depth analysis of this concept, but it is at least worth remembering an analogous concept that is well established in the field of information technology, the productivity paradox, i.e. the failure of great investments to boost productivity growth. As the Nobel Prize-winning economist Robert Solow said, “you can see the computer age everywhere but in the productivity statistics”. There are four main explanations that have been proposed to explain this paradox (Brynjolfsson, 1993): “(1) mismeasurement of outputs and inputs; (2) lags due to learning and adjustment; (3) redistribution and dissipation of profits; (4) mismanagement of information and technology” (Brynjolfsson, 1993, p. 73). The first argument is that most of the benefits of IT come in the form of enhanced product or service quality, time savings and convenience. This is entirely consistent with the issues raised relating to public sector performance. If firms were to look just at statistics on productivity alone without considering quality of products and services, the massive investments made in the last twenty years in IT would have not taken place. Similarly, it could be argued that focusing just on productivity in public services could provide just part of the picture, neglecting very important aspects and driving wrong behaviours. As Erik Brynjolfsson, one of the most eminent academics to study the productivity paradox, stated: “A review of the IT productivity research indicates an analogous opportunity to rethink the way productivity and output are measured” (Brynjolfsson, 1993, p. 76).

Thirdly, further insights on the performance of public services could be gain by examining international reports, e.g. statistics published by the European Union (Eurostat, 2004) and reports issued by the World Health Organization (2000). Although it is certainly difficult to make comparisons between countries, if league tables and rankings are considered, this should be also in the light of these documents, which could provide a broader picture.

Analysed together the three recent reports concerning performance measurement in the public sector highlight the considerable shortcomings in current approaches. A multi-dimensional view of performance is necessary as well as a more knowledgeable debate on public services, where information is treated with caution and citizens and media are actively engaged.

Pietro Micheli, Steve Mason, Mike Kennerley, Mark WilcoxBased at the Centre for Business Performance, Cranfield School of Management, Cranfield, UK.