Reporting on Outcomes: Setting Performance Expectations and Telling Performance Stories

Introduction

Historical background

The results-expectations chart

Setting clear and concrete expectations

Telling a Performance Story

What can Parliament and the public expect?

Conclusions

References


Introduction

"Results" and "performance" are becoming mainstays of discussions of public management. Managing for results and reporting of results are part of the lexicon of public sector management. Public managers are being asked to produce results and in an economical way with the public money entrusted to them.

The basic elements of managing for results, or results-based management, can be set out quite easily. In the context of programs, managing for results requires (Auditor General of Canada 1997, p. 11-12):

  1. Fostering an organizational climate that encourages managing for results,
  2. Agreeing on expected results,
  3. Measuring results to improve performance, and
  4. Effectively reporting performance.

Managing for results, however, has proven to be quite difficult to implement. Measurement of results in the public sector is often thought of as the biggest challenge. Moving to a results-focussed culture is also seen as a huge challenge, and good performance reporting seems to be painfully slow (Auditor General of Canada 2000). Agreeing on expected results is often considered a challenge but rather straightforward. Experience with managing for results, however, is showing that setting performance expectations may be the most difficult aspect of managing for results to accomplish.

Wholey (1997) has pointed out "The most important initial step in performance-based management is getting a reasonable degree of consensus on key results to be achieved …" (p. 100).

It is essential to be able to set out clear statements of what is to be accomplished. The concept of "performance" requires a comparison of what was expected with what was achieved. It is not possible to assess performance either for managers or for the public without knowing first what level of performance was expected.Footnote1


The purpose of this paper is to

Terms

Before beginning, we need to define the terms we are using. Government programs undertake a number of activities that produce a variety of results. Programs deliver two kinds of results: outputs, the direct products and services produced by government activities, such as an unemployment cheque or some requested information; and outcomes, the consequences (both intended and not) of those outputs on Canadians and our society. Outputs are results that managers can largely control, while the outcomes that managers are trying to accomplish are influenced by factors outside their programs.

End outcomes (sometimes called long-term, final, or ultimate outcomes) are the end results sought, such as general improvement in the well-being of Canadians, the economy, or the environment. Between the outputs and the end outcomes, there is a sequence of immediate and intermediate outcomes that are expected to lead to a desired result but are not ends in themselves, such as changes in the actions of program clients. Immediate outcomes are more easily linked to the activities of a program than are end outcomes. The results chain (Exhibit 1) is this logical sequence of outputs and outcomes that occurs as a result of a program's activities.

Articulating what a program is intended to accomplish is critical to good results management and reporting. A wide range of terms are used to describe these normative statements: objectives, goals, strategic outcomes, expected results, planned results, targets, and expectations to name a few. Among these, there are two important concepts to distinguish:

Both types of statements of intentions are needed. The first type (objectives) Footnote2 sets out, at a high level, the general direction and end state sought, but it often does not specify the extent of results sought or the time frame to accomplish them. Objectives link to the mission and vision of an organization and set the stage for the second set of statements, the more concrete performance expectations. Performance expectations define the specific results expected, the extent (how much is expected), and the timeframe. Good performance expectations allow one to know and determine if what has been set out to be accomplished has been achieved. Having clear statements that specify what is expected is essential to performance information. Without them, all one has is results information.

Historical background

In the past, setting expectations for performance has normally involved setting expectations for outputs. Outputs are well understood since they represent the direct result of activities. They are usually quite visible and measurable, and we can comfortably talk about being accountable for them because we control them. The performance reporting regimes of a number of jurisdictions have been heavily based on setting specific output targets and reporting on how well outputs have been produced.

For a management and reporting regime focussing on outcomes, the situation is quite different. The production process for outcomes, be they immediate, intermediate, or end outcomes, is usually not as well understood as for outputs. The linkages between various levels of outcomes may not be well known and the measurement of the outcomes themselves may be quite a challenge. By definition, we do not control outcomes but rather seek to influence their occurrence by carrying out certain activities and delivering certain outputs. As a result, we are much less comfortable with being accountable for outcomes because we do not control them. Yet, despite these different conditions, the usual approach to setting expectations for outcomes has simply adopted that used for outputs by attempting to set numerical targets for each outcome specified.

This approach has proven to be less than satisfactory because it doesn't take into account the fact that outcomes can be difficult to measure, links between outputs and various levels of outcomes can be difficult to establish, and being accountable for single specific numerical outcomes is often not very realistic nor, more importantly, useful. For example, it may be easy to set targets for literacy and numeracy, but improvements in those areas may be at the expense of skills that are more difficult to measure, such as creativity (The Economist 2001). In this paper, we suggest an alternative approach that recognizes that outcomes are not like outputs.

The results-expectation chart

When considering the performance of a program, a logic model or results chain is often developed. This is usually in the form of a diagram (see Exhibit 1) of how the program is supposed to work. It describes how activities undertaken produce a variety of outputs, which in turn result in a chain of subsequent outcomes that are expected to occur.

We have previously called for clear and concrete expectations to be set and reported against (Auditor General of Canada 1997, 2000). Good performance reporting tells how well a program has done at what cost. Good management requires that a manager know when expectations have been accomplished and be able to demonstrate it. Clear and concrete expectations enable a manager to do this, and to know how successful a program has been and where improvements are required.

In our view, a results chain is at the heart of setting outcome expectations for a program. We suggest that setting sensible performance expectations requires the results chain to be developed and set out. We expect that not just one or two specific outputs and outcomes will occur, but rather that the whole chain of events presented in the results chain will occur. Thus, setting expectations entails setting out a results-expectations chart.

A results-expectations chart requires

A results chain is like a logic model, which can be developed to show, usually in some detail, the causal sequence of outputs and outcomes. But the results chains suggested here do not try to provide that level of detail. They are based on the espoused theory of the program. As a result, they do not necessarily require confirmation about causality, which is often associated with developing a program theory on which a logic model is based. Results chains are suggested here as a structure for describing the expectations of a program (and as a basis for reporting the subsequent performance story). Of course, the better understood the theory of the program is, the more realistic the expectations set out will be.

Where the results chain describes a program, it should include the planned expenditures. Where efficiency is an important aspect of performance, the expected units costs of the outputs produced should be part of the results chain description of outputs.

A complete picture of performance expectations, in addition, would discuss the legitimacy of the expectations — how they were set, their consistency with mandate and mission, their reasonableness, and their significance.

Exhibit 2 outlines a generic results-expectations chart, while Exhibit 3 provides an example of a results-expectations chart for a unit in a national audit office working to improve the practice of accountability for results. The example is typical of most government programs in that the outcomes sought occur well outside the program in question, and are at best influenced by the activities of the program.

Exhibit 3 sets out both the underlying results chain (the espoused theory of the program) as well as the results-expectations chart. Depending on the specific case, this may or may not be a useful approach. The results chain shows the logic of the program, while the results-expectations chart sets out more specifically what is expected to be accomplished by the program, using the results chain as a reporting structure. Note also that the results-expectations chart is a chart of intentions, not facts. Measurement is required to determine the extent to which the expectations listed in the chart do in fact occur.

A number of observations can be made:

Setting out performance expectations in this form requires more information, largely because outcomes are more challenging to deal with than outputs. Public programs are usually complex interventions trying to alleviate a problem or maintain a condition. Certain aspects of the program might work and be successful (certain outcomes might have been achieved) while other aspects might not (other outcomes might not have been achieved). For still other aspects, it might not be clear (perhaps the links between aspects of the program and the outcomes sought cannot be demonstrated). Of course in many programs, certain key outcomes may be identified and success readily determined in those terms. But even then, success needs to be further discussed by setting out the likely extent of the contribution made by the program to the expected outcome.

Developing a results-expectations chart would have many of the advantages often identified with developing a results chain, such as the following:

Limitations also need to be recognized, including the following:

Benefits of setting good performance expectations

Setting out a results chain and a results-expectations chart requires some hard work but is essential for managing for results. Managing for results requires a focus on the whole results chain/expectations chart. Too often, managing for results focuses only on the end expected outcome. But in most programs, there are many assumptions that must be realized and events that must occur if that final outcome is to be achieved (or contributed to). Without an understanding of the intended chain of events a program is trying to influence, focussing on the end outcome is not likely to be very useful.

An additional benefit of using a results-expectations chart is that, rather than the more traditional one or two indicators, the concern about being accountable for outcomes should be reduced. We have discussed elsewhere the need for a new perspective on accountability (Office of the Auditor General 2002). We argued that accountability for outcomes should mean being able to demonstrate

This perspective recognizes that outcomes are influenced by other factors and looks to managers to maximize the possibility that the services and products they do control adequately contribute to the end outcome sought. Setting out expectations in an results-expectations chart emphasizes what is reasonable and thus what reasonable accountability expectations ought to be, and what needs to be measured. From this perspective, good performance expectations require a clear and concrete results-expectations chart.

Setting clear and concrete expectations

There is often resistance in organizations to setting out clear and concrete performance expectations. Sometimes this is just a reluctance to have performance measured and compared with expectations. This is the cultural barrier that most organizations have to overcome if they wish to move toward managing for results (Auditor General of Canada 1997, paragraphs 11.41-11.66). In other cases, the reluctance is due to not knowing what the level of results should be. This can be due to difficulties in measuring the results in question as well as considerable uncertainty about how the activities of the program influence the intended results. Clear and concrete expectations are needed to determine and demonstrate when expectations have been accomplished.

Two concepts of expectations. There are two types of performance expectations. More traditionally they are seen as predictions (targets) of what is hoped for in the future given the resources available, with the expectation that they will be met most of the time. Alternatively, expectations can be seen as challenges to seek. These are often referred to as stretch targets, set to provide clear direction on where the program is striving to be in the future. Stretch targets are set at a high enough level so that they cannot easily be met with existing resources, and thus with the knowledge they will often not be met. Their purpose is to set results to be strived for rather than results to be met. There is an expectation that in stretching, innovation will be encouraged and opportunities to learn and improve will be sought.

Performance expectations set as predictions (targets) can play a positive role, but they also have a number of shortcomings:

The predictor model of setting expectations probably works better for output-based reporting rather than outcome-based reporting, since considerably more certainty can be established about the levels and quality of outputs produced.

We would argue that reporting on results (outcomes) as part of performance reporting must mean using expectations as challenges — that is, stretch targets. This type of expectation reflects the learning model underlying the concept of managing for results. One of the many advantages of using stretch targets as expectations would be the recognition upfront that many of these expectations would probably not be met, but that much learning would have taken place. This learning should be reported as achieved performance.

Setting expectations is evolutionary. An essential element of the managing for results framework is the need to review and adjust over time as more experience and understanding are acquired. It is unrealistic to expect that at the outset one will be able to identify the perfect set of measures and corresponding performance expectations, and then set out to implement a performance measurement system that will last over time. Experience everywhere clearly shows that the process is evolutionary and advances through a lot of trial and error. Furthermore, the environment within which a program operates is constantly changing, and thus ongoing planning and consequent revisions to performance expectations are needed. Clear and concrete expectations evolve over time from the objectives.

The implication is that we should see a results-expectations chart as an evolving construct. Over time, the chart

This evolution should occur in a deliberate manner, rather than as random trial and error. There should be a very visible built-in review and adjustment mechanism that identifies the "fittest" measures and expectations — that is, those that turn out to be useful to the organization for managing and reporting. This again reinforces the importance of deliberate learning based on past experience, the hallmark of managing for results. Reporting simply on the gap between expectations and actual performance encourages a focus on meeting targets rather than learning.

If we put this approach together with the use of challenge-expectations, then reporting on performance comes naturally, with a focus on reporting the learning that is occurring rather than on variances from predictions. It also means that good performance reporting should include a description of how the results-expectations chart and its components are expected to evolve, how current expectations have been set, and what is hoped to be achieved with them as experience is gained.

Outputs still matter. The focus here is on setting outcome expectations and using them in planning, managing, and reporting. But in no way are we suggesting that outputs are not important. Setting output expectations and measuring progress toward them and tracking the expenditures associated with those outputs are essential for day-to-day management. Although managers cannot track outcomes on a daily basis, they need to keep track of their day-to-day activities and the resulting outputs and expenditures. Managing for outcomes means that from time to time managers need to have information on how well their results chain reflects reality and thus be able to modify their activities and outputs as required.

Strategies for developing expectations. In keeping with the idea of an evolving results-expectations chart, there are several approaches that can be adopted to move toward the goal of clear and concrete expectations:

Good performance reporting would report on these strategies for developing expectations and explain where the expectations came from and how they were determined.

Clear expectations. At a minimum, stated expectations must be clear (well-defined and not ambiguous). The reader should be able to tell what aspect of performance is being reported.

Concrete expectations. Where possible, expectations should be concrete. However, they may vary as to the type or extent of concreteness. The ideal concrete expectation has a number of characteristics:

A concrete performance expectation allows one to know when the expectation has been attained.

Targets can be both numerical with time frames or non-numerical. A variety of targets that are non-numerical or not a single number yet still concrete are possible, including the following:

For a number of reasons, setting out a concrete target at the outset might not be feasible or useful. In moving toward this ideal, one might adopt a number of approaches, such as the following:

In both cases, the expectation could be made more concrete once more experience is gained. The reasons for using either of these approaches should be reported.

Selecting performance measures. Key to setting clear and concrete performance expectations is selecting good performance measures to represent those expectations. Good measures produce measurements that are accurate, reliable, and valid.

Accurate measurements are those that correctly represent the reality measured. Reliability implies that if others use the same measures, they would produce the same results. Valid measures measure the performance characteristic of interest rather than something else. For example, an IQ test may be an accurate and reliable measure but as a measure of intelligence it has poor validity. Similarly, the time taken to go from 0 to 100 kilometres per hour is a valid measure of the acceleration of the car, but not a valid measure of the performance of a car.

Often, good measures that provide useful information on performance are not evident until after measurement has been underway for some time. Reviewing and revising the usefulness of the measures is good practice.

Being selective. For most programs, it is easy to identify quite a large number of performance measures from a results chain and correspondingly to set a large number of expectations. This is normally not useful. One needs a manageable number of expectations, perhaps four or five for a particular span of management control. Too many measures and expectations cannot be managed and will likely end up just feeding information systems.

Alternatives to concrete expectations. As suggested, in some cases it may not be practical or feasible to set out concrete expectations. Where expectations are clear but not very concrete, one approach would be to assess the achievement of the expectations using an external expert review panel with a qualitative scale. The National Science Foundation (2000) in the US has used this approach in its performance reporting. Each of its programs is reviewed every three years by groups of independent external experts for their success in meeting targets. They use a two-point rating scale: "successful" and "minimally effective." Exhibit 4 illustrates the results for the goals and targets for one program.

The credibility of this approach clearly depends on the credibility of the experts used. But it is one way of dealing with the problem of expectations that are hard to quantify. Over time, there may be agreement on what constitutes success in these areas.

"Gaming" expectations. Setting expectations so that you know they will be met is one way of "playing games", of not managing for results. In presenting the practice of developing expectations over time, another obvious game could be played: continue to change expectations every year so that over time, progress cannot be assessed. We are not suggesting this. We are suggesting a deliberate strategy to improve expectations, with many of them becoming quite stable over time (unless the underlying activities change). The initial evolution of expectations should be over a period of several years, not as long as a decade.

A final word. Although we have been discussing setting expectations for a specific result to be accomplished, it is important to recall that this is all being done within the bigger picture of the results-expectations chain. From that perspective, it might be appropriate for some of the expectations to be quasi-concrete — perhaps for the foreseeable future — as long as others were moving toward being concrete. Further, the number of concrete outcome-expectations needed will depend to some extent on the strength of the logic chain for the program. The stronger the evidence is that the program logic works, the less may be the need for concrete, higher-level outcomes. Thus, for example, an anti-smoking program has as its ultimate outcome the improved health of Canadians. On the surface, this is an outcome quite far removed from the activities of such programs. However, the link between reduced smoking and improved health has been strongly established. The program is therefore safe in assuming that reduced smoking resulting from its program activities will improve health.

Telling a Performance Story

Reporting on performance requires reporting on what was achieved in relation to what was expected, as well as on what was learned. Reporting on outcomes therefore involves reporting on what was achieved in relation to the results-expectations chart set out. It involves more than simply reporting against several specific performance measures or targets previously set out. It involves telling a credible performance story about the results-expectations chart — that is, presenting evidence on the extent to which the results-expectations chart in fact reflects reality. Of course, if in addition, unintended results are observed, these too should be reported. Indeed, it is good practice to ensure that the strategy for measuring program results includes room for addressing unintended results associated with the program.

Presenting the story. There are a variety of ways to present a performance story. All involve a mix of quantitative evidence that certain outputs and outcomes have occurred as well as narrative discussion and further evidence of the contributions made at various points along the results chain, all described within some context. A performance story sets out to convince a skeptical reader that the activities undertaken have indeed made a difference — that the expectations chain has, at least to some extent, been realized, along with any significant unintended results.

Exhibit 5 identifies the main elements of a performance story and illustrates further detail on each element. These elements could be used to present the story in a narrative form, with accompanying evidence on the specific results being reported and with reference to the expectations chart. One example of this approach can be found in the performance report of the Office of the Auditor General (2002).

Reporting on the contribution made. One of the more challenging aspects of reporting on performance is to credibly report on the contribution the program has made to the expected outcomes. The aim is to show that the program has made a difference. Practical approaches to undertaking a contribution analysis have been discussed elsewhere (Mayne 1999). Here we suggest ways to report on the contribution made, in particular, without the availability of a robust evaluation that addressed the contribution issue. In this case, the aim is not to definitively prove that the program has made a difference but to build over time a convincing case of plausible association between the results observed and the activities and outputs of the program, as more and more evidence is gathered. Presenting a case that is reasonably accepted is a more realistic aim.

The key component is the results-expectations chart, especially its logic. The logic model sets out what is supposed to happen as a result of the program activities. If this is not convincing, then the likelihood of a convincing contribution story is low.

While essential, the logic story only says what is supposed to happen — why one believes that the program is supposed to make a contribution. The real question is has it? A variety of data and information could, and indeed should be collected to build the evidence (Mayne 1999), such as the following:

The example in Exhibit 4 uses groups of outside experts to conclude that the program of the National Science Foundation has made a contribution. Does it present undisputable evidence? Probably not, but quite a strong and reasonable case is made.

Multiple lines of evidence. It is likely that no single piece of evidence gathered will on its own be enough to build a credible case concerning a result achieved or a contribution made by a program. As discussed elsewhere " While no one piece of evidence may be very convincing, a larger set of different and complementary evidence can become quite convincing" (Mayne 1999). It is the totality of the evidence gathered — some of it strong some perhaps rather weak —that builds a credible performance story. The measurement strategy used ought to anticipate the need for multiple sources of data, information and analysis.

A narrative performance story. The most common way a performance story is told is probably through a narrative describing the performance accomplished in light of what was expected. If the results chain for the program is used, the story can be expressed as discussing the extent to which the results chain actually reflects reality, giving structure and logic to the story.

A chart of performance accomplishments. One alternative approach to telling key elements of a performance story would be to prepare a chart of performance accomplishments or a performance story chart paralleling the expectations chart. Exhibit A.1 in the Appendix presents a generic performance chart. The left hand column presents the basic results chain against which the right hand column is reporting. This is needed to remind the reader of the basic theory and assumption underlying the program. The right hand side — the substantive performance story — can cover a number of elements:

There is even room in the performance chart for anecdotal evidence. Anecdotal evidence on its own is usually not very credible. It may be quite selective and not representative of the true picture. However, presented in a performance chart, the anecdote has context and serves to illustrate that at least some cases are conforming to the expectations chart. The anecdote is anchored somewhere.

Exhibit A.1 also illustrates how the learning that has occurred can be reported, by setting out a summary of the implications drawn from past performance.

Exhibit A.2 in the Appendix presents a performance chart for the results-expectations chart shown in Exhibit 3.

There is not much experience to date with using performance charts to tell performance stories. The exhibit illustrates a number of techniques that might be used to present parts of the story. The aim is to find ways to succinctly present a meaningful story.

What can Parliament and the public expect?

Setting out clear and concrete performance expectations and reporting progress against them is essential for good performance reporting to Parliament and the public. Without good statements of expectations, it is not clear whether the results achieved represent good performance.

Clear and concrete expectations. Several things are required to set out clear, concrete, and credible expectations. Good reporting should

In a results-expectations chart, a good statement of performance expectations would articulate

A credible performance story. Good reporting to Parliament and the public ought to set out credible information on each of the elements of a performance story, as outlined in Exhibit 5, should set out

Conclusion

Managing for and reporting on outcomes instead of outputs requires new approaches to setting outcomes expectations and telling performance stories. We have suggested a number of ways to approach this problem, including

References

Auditor General of Canada (1997). Moving toward Managing for Results. Report to the House of Commons, Chapter 11. October. Ottawa.

Auditor General of Canada (2000). Reporting Performance to Parliament: Progress Too Slow. Report to the House of Commons, Chapter 19. December. Ottawa.

Auditor General of Canada (2002). Modernizing Accountability in the Public Sector. Report to the House of Commons, Chapter 9. December. Ottawa.

Mayne, J. (1999). Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly. Office of the Auditor General, Ottawa.

National Science Foundation (2000). Accountability Report FY 1999. Washington.

Office of the Auditor General of Canada (2002). Performance Report for the period ending 31 March 2002. Ottawa.

The Economist (2001). "The trouble with targets". April 26. http://www.tbs-sct.gc.ca/rma/account/OAGTBS_E.html

Wholey, J. (1997). Clarifying Goals, Reporting Results. In Debra J. Rog (Ed.). Progress and Future Directions in Evaluation: Perspectives on Theory, Practice and Methods. New Directions for Evaluation, No. 76, Winter. Jossey-Bass Publishers.

1 An exception might be cases of programs that are experimental in nature—usually pilots—where the intervention is expected to be beneficial but there is perhaps little experience to date and hence concrete expectations may not be practical, or are expected to emerge as understanding of the program's contribution is gained. Over time, concrete expectations could be developed. Return to reference 1

2 In July 2001, the Treasury Board Secretariat issued a lexicon of performance reporting terms and asked departments to use the term strategic outcomes for what we call objectives in this paper. Return to reference 2

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: