Criteria for the Assessment of the Fairness and Reliability of Performance Information

The purpose of this document is to provide information on the criteria the Office uses to assess the fairness and reliability of performance information in annual reports. Assessments by the Office for the Parks Canada Agency, the Canadian Food Inspection Agency and the Canada Customs and Revenue Agency can be found in the annual reports of those agencies.

The Office is required to undertake an assessment of the fairness and reliability of performance information reported against corporate objectives provided to Parliament by certain agencies. In developing criteria to use in the conduct of this assessment, the Office reviewed practice in other legislative audit offices, examined the literature on performance reporting and considered what parliamentarians would want to know about the performance information presented to them.

Fairness and reliability relate to good and honest reporting. Fairness deals with honesty in reporting the telling of an accurate and complete story. Reliability deals with information you can count on and trust as part of the performance story. Fair and reliable performance information is not misleading.

The criteria that follow reflect these considerations. They are also consistent with draft reporting principles developed by the legislative audit community in Canada. Performance information with respect to objectives is fair and reliable to the extent that it meets the criteria below:

Fairness Criterion

Relevant

The performance information reports in context, tangible, and important accomplishments against objectives and costs.

  • Program context includes
    • the mandate, strategic outcomes and objectives, as part of a clear and complete results chains, which are linked to government priorities.
    • program structures
    • key horizontal initiatives and partners used to deliver the results, and
    • a discussion of the external environment and the key risks faced
    • A description of intended users
  • Reported results are focused on outcomes with related program activity types and outputs identified. (Results Chain)
  • New initatives for improving outcomes are described
  • The measures used to support the results chain are valid and complete
  • The public performance report should link financial and non-financial information to show how resources and strategies influence results, including relating costs to outputs and outcomes. Where possible, cost information includes both direct costs and allocated indirect costs of programs and services to present 'full cost' data.

Meaningful

The performance information describes expectations and provides benchmarks against which performance is compared.

  • Expectations are set out which are
    • clear, concrete, and measurable, identifying the amount and direction of the change, the target groups, and timeframes,
    • focused on outcomes with relevant activities and outputs identified, and
    • consistent with the strategic outcomes.
  • What is reported is in accordance with what was expected in the corporate plan or RPP, including timeliness of data
  • Comparisons are provided between reported accomplishments (actuals) and the expected performance with a realistic interpretation of the gap between the two
  • Changes in performance measures and targets are identified and explained
  • Comparisons are provided with relevant benchmarks, such as similar activities, programs or organizations, or trends over time, and their significant explained.
  • Key lessons learned about past performance and any resulting actions are discussed.
  • Performance information is reported in such a way that it is understandable and clear.

Attributable

The performance information demonstrates why the program made a difference.

  • The contribution that has been made by the program to the reported results is demonstrated, including evidence regarding attribution, where available (such as evaluations).
  • The contribution of key partners and other external factors is discussed.

Balanced

A representative and clear picture of performance is presented, which does not mislead the reader.

  • The explanation of variances would include both positive and negative aspects of performance, as well as major challenges identified for programs and services, in order to provide a complete picture of performance
  • Significant unintended impacts are reported
  • There is coverage of all key objectives and only performance information that is significant to the organization is reported.
  • There are no distortions of information through presentation or tone, or through omission of information or context.
  • The emphasis on information presented is proportional to its importance and materiality.

Reliability Criterion

Reliable

The performance information adequately reflects the facts

  • Valid and consistent/comparable measures of performance are used.
  • Performance information is based on data that can be readily replicated to produce similar results.
  • The data that supports the performance measures are complete
  • Analyses and conclusions about performance are supported by adequate evidence
  • The basis for confidence in the reliability of the performance information (including third party data) being reported is disclosed, including
    • methods of data collection
    • explanations for any limitations
    • sources of information are reported