A second key question in designing performance measurement systems focuses on the best place to measure performance in the complex system that transforms inputs granted to the organisation into the production of desired results through a particular set of policies, programs, and activities. One useful way to think about this production process is to describe it as a value chain that consists of several distinct stages of the production of public value. (See Figure 2 below)
Obviously, the concept of a value chain is not rocket science. The value chain describes an organisation that is receiving inputs (in the case of a public organisation, money and authority from the state), which are deployed in particular processes, procedures, and programs which produce outputs. Many of those outputs involve an important engagement with people, or clients – and by clients I mean people who make individual transactions or have individual encounters or relationships with the organisation. One of the important consequences, or products, of those client transactions or encounters or relationships, is some form of social outcome. So the value chain gives us a general picture of organisational production.
But the value chain, as it is drawn above, also reminds us to focus on the actions of partners and co-producers who can be instrumental in the production of client satisfaction and social outcomes. Contributions from partners and co-producers could emerge spontaneously without any encouragement or guidance from the organisation. Or, they could emerge as a consequence of deliberate efforts made by the organisation to mobilise co-production activities. The organisation can write contracts with partners or it can seek to use moral persuasion of various kinds to motivate private individuals and organisations to contribute to public goals. It can even use the authority of the state to require others to contribute to socially desired outcomes.
In principle, we could take measurements and demand accountability for performance at any step along the value chain or the chain of partners and co-producers. We could focus on inputs to make sure that our managers have firm control over the equipment and inventory, that they are doing a good job of cash management , and so on. We could focus our attention on activities, processes, procedures and programs through a ‘compliance audit,’ where independent evaluators would go into government organisations and see whether people were following the policy and procedures that were laid down for them.
We can all see the limits of a compliance audit: it does not tell us very much about the quantity or quality of outputs we are producing, or whether the outputs are judged to be valuable in the sense of satisfying clients or producing desired social outcomes. However, if we had already embedded in our policies and procedures the best available technology for accomplishing results, then a compliance audit would be tantamount to an audit of efficiency and effectiveness. If our procedures were known to be efficient and effective in producing desired results, then all we would have to do is monitor and demand performance with respect to following policy and procedures and, arguably, we would have achieved our purpose. If the policies and procedures are based upon ‘best practices,’ we can effectively guarantee operational performance by adhering to these policies and procedures rather than by guaranteeing results. (This is similar to the approach that McDonald’s takes in trying to ensure that each restaurant produces the desired products and services. The judgment about whether this particular product and service can be delivered in a profitable way is made elsewhere.)
But there is another reason to adhere to strict policy and procedural regimes: namely, to assure not efficiency and effectiveness, but consistency. This is a desirable characteristic of government operations quite apart from questions of efficiency and effectiveness. One of the characteristics we want in a government organisation is a guarantee that like cases will be treated alike. This is part of what we mean by fairness, and it constitutes a virtue independent of how effective a particular government process turns out to be.
So, our options are: (1) concentrate on managing performance measurement at the level of activity, processes and procedures; or (2) monitor outputs, workload measures, and measures of productivity. Increasingly, however, government organisations are being pushed to measure performance even further down the value chain in the direction of either client satisfaction or outcomes. I think the reasons for that are pretty clear. But, as you know, the great debate goes on about whether we should measure outcomes or outputs in the public sector.
My observation is that the current debate is shifting heavily in the direction of measuring outcomes. The reason for this is that an outcome is a direct measure of the public value that we are trying to produce. If we need to be able to demonstrate that we are producing something of public value, then we would have to be able to measure outcomes. Once we devise a way to measure the outcome we are trying to achieve, we can find out whether the technical means that we are relying on represent the best technology for accomplishing that purpose, or whether there is a better alternative we can use to test the value proposition offered by the organisation, so that when we say, ‘Well, we are going to produce public value by engaging in the following activities,’ we can track whether our theory is correct.
If it is so clear that outcome measurement is valuable, why do we so rarely do it? Why do we not measure outcomes? Well, it turns out there are a lot of reasons why measuring outcomes is a difficult device for managing organisations. One reason is that measuring outcomes is incredibly expensive.
In a private sector firm, revenue is earned by the sale of products and services. Revenues earned by the sale of products and services to willing customers are easy pieces of information to collect. They are a terrific thing to use in measuring performance because they represent a reasonably objective, direct measure of value. What makes them a direct measure of value? Individuals’ choices to spend their own money on a given product or service: people plunk their money down to buy the product or service. In doing so, they give objective information that they value whatever good or service the firm is producing.
The second attractive property of revenue earned through the sale of products and services (as an outcome measurement) is that the information is collected right at the boundary of the organisation at the moment of transaction. We don’t have to go beyond this transaction to measure something that takes place at a relatively remote place and time. Nor do we have to wait for the result to occur. We get the judgment of value right at the point of delivery.
A third property of revenues earned through sales is that one can compare apples and oranges. Individuals show us how much they value these distinct products by paying for them in a given currency.
So, earning a revenue from the sale of products and services turns out to be a really wonderful way to measure the value of organisational output. If we have such a measurement, managers (and others) are in a good position to judge the value of what has been produced. If we do not have revenues earned from the sale of products and services to voluntary customers, we are in big trouble.
To see the significance of the loss of this kind of revenue measure, consider this problem: if I were the manager of General Motors and somebody said to me: ‘You can have all the information you want about the cost of producing automobiles, but you cannot have any information on how much you have earned by selling them,’ what would I do? How would I know whether I was creating something valuable?
I think there are only a limited number of answers. One of them would be to survey your customers to see whether they liked the vehicle. Another possibility: you might observe the outcome and see how often people drive and how well the car stands up to use. A third possibility: you might try to get engineers to tell you whether they think the car that you designed is a good car – is it better than previous or comparable models?
These would be all the same methods we would use in the public sector to try to ascertain whether we are producing something of value. We survey our customers; we evaluate outcomes; we check on the physical characteristics of our product. While such measures might do some managerial work for us in helping us determine whether we are producing valuable results, none of these measures of value would stack up well against the information we would receive from measuring the revenue that we earned from the sales of products and services.
So, one of the difficulties with relying on the measurement of outcomes is that outcomes are very expensive to measure: we have to go out into the world to find the information and, often, information comes in late.
In the case of a welfare-to-work program, for example, we are interested in trying to spend public dollars to cause clients who use the program to move from a dependent life on welfare to an independent status of job-holding and an active life in the community. An interesting question is: When do we decide that we have accomplished that outcome? And the answer to this question is: We never have achieved the outcome – at least not for a large group of people. What we are trying to do is to maintain a certain level of functioning for a group of people who, ideally, rely less and less on public support in the form of both dollar benefits and social services. The benefit of public support comes from the fact that people improve in their independence and social functioning continuously over a long period of time. So, when do we decide that we have accomplished the goal?
Perhaps the best way to think about it is that, each month, we have kept X number of people at a higher level of functioning than they would have been without intervention. That number represents the value but, in order to get it, you have to have continuous monitoring of the whole target population over a long period of time – a very difficult and expensive process.
Further complicating things, because the information about outcomes comes in late and public value production happens far down a causal chain, the effect is often hard to attribute to public managers or public organisations. Crime statistics may have gone down under Bill Bratton’s watch, but we actually do not know whether or to what degree that was the consequence of any given action that Bratton took.
While we talk a great deal about the ultimate importance of focusing on outcomes in the public sector, and while the challenge to think about and measure the ultimate outcomes is certainly valuable, the reality is that we are not only interested in the social outcomes produced by government organisations. We are also interested in the processes and procedures we use to achieve the outcomes. We want to make sure they have certain characteristics – including fairness and economic and technical efficiency – and that means that we will want to measure some characteristics of activities and outputs as well as outcomes. So, while I understand and support the argument for measuring outcomes, there are real problems with relying exclusively on outcome measures.
So then, what about measuring outputs? Measuring outputs is cheaper and easier than measuring outcomes. In addition, measuring outputs can contribute to improving the performance of midlevel managers because, unlike outcomes, outputs are mostly within managers’ direct control. We can make interesting calculations about productivity and – without necessarily making claims about value or effectiveness – we can at least see whether we are driving down the cost per unit of output. This helps to focus the manager’s attention. We get improved consistency in operations, and, if we have an idea about what kinds of practices constitute value, we can make sure that those best practices are being used widely throughout our organisation.
These observations add up to some important reasons to measure outputs rather than outcomes. So much so that it might be tempting to rely only on the measurement of outputs. The problem with that solution, however, is that organisational outputs are not quite the same as the production of something of public value – not even in terms of producing client satisfaction, let alone in terms of the achievement of desired social outcomes.
If both outcomes and outputs are important and useful to measure, and each has limitations, it seems to me that the right conclusion is that one ought to measure both.
This conclusion is reinforced by the observation that private sector organisations are increasingly managing according to 12 or 13 or 14 different measures of performance. They do not limit themselves to one ‘bottom line.’ They exist in a ‘data rich environment’ where they can make judgments about the degree to which their current strategies are succeeding and, at the same time, search for clues about how they might improve their performance.
The bad news about measuring multiple dimensions of performance is that it makes it harder for managers to focus. The more things you are trying to manage, the more complicated the job becomes. Another way to think about it, however, is that if we are managing 12 different dimensions of performance, then we have an opportunity to engage the organisation in a conversation about what it is being produced. A conversation about how to get all of those different dimensions of performance moving in the right direction would be more interesting for those inside the organisation than a simple thumbs up or thumbs down test of how well the organisation and the individuals in it are performing. The organisation might actually be able to learn and feel motivated to find better ways of achieving difficult results, rather than simply censuring people for failing to meet more or less arbitrary standards.