Performance Audit Manual

Each audit should result in a report that clearly communicates to the reader: the objectives, nature, time period covered by the audit, and scope of the audit, including any limitations; the professional standards and policies used; the level of assurance provided by the report; a description of the program or activity that was audited, including management's responsibilities; the criteria used, their source, and any disagreements with management on their suitability; (May 2006) the observations made; the recommendations made to point to the direction in which positive changes can be made (may not apply to all audit notes); management comments (if provided) including planned action in response to the audit and any differences of opinion; and the conclusions reached against each audit objective including any qualifications, where applicable.

5.1 The reputation and credibility of the Office depend to a great extent on the quality of the reports of the Auditor General and the Commissioner of the Environment and Sustainable Development. The reports are what the client, media and public see of the work of the Office. Consequently, they have to meet the highest attainable standards for content and presentation. In preparing the report, the audit team should keep in mind:

• the end use of the report — that is, the use made by the parliamentarians in their scrutiny of government operations; and
• the scope of the Office mission to promote accountability and best practices in government operations.

5.2 The purpose of the report is to achieve positive change. The requirement for clear communications means that messages need to be:

• clear and precise and written in plain language to ensure that the reader will understand what the report is trying to achieve;
• convincing and their importance highlighted for the reader;
• fair and presented in an unbiased tone, noting where management has taken actions to correct the deficiencies and pointing out exemplary performance; and
• only dealing with matters of significance.

Key contents

5.3 The following information sets out the key contents of the report and provides an explanation for their inclusion:

Objective

5.4 To clearly set out the key questions about performance that the audit sets out to answer (such as "to determine whether the program was cost-effective...") as well as the issues related to non-audit objectives (such as "to provide information on...").

Timing

5.5 To inform readers of the period of time for which assurance is being given in the chapter and to assure them that the report is dealing with issues of current interest.

Nature and scope

5.6 To set out what was audited, the extent of audit and any limitations. When the objective of the audit is to conclude on whether an entity has complied with specified authorities or whether its transactions were carried out in compliance with specified authorities, the scope of the audit will state the authorities against which compliance is being reported.

Professional standards

5.7 To provide confidence that the audit was conducted in a professional manner. Accordingly, all our performance audit reports to Parliament state the following:

"All of the audit work in this report was conducted in accordance with the standards for assurance engagements set by the Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, we also draw upon the standards and practices of other disciplines."

Level of assurance

5.7.1 To state the level of assurance that is being provided by the work. The work reported in performance audit chapters and audit notes is performed at an "audit" level of assurance as opposed to a "review" level of assurance. An audit level of assurance provides a high but not absolute level of assurance and allows the audit team to make a conclusion on the subject matter against the audit objective(s) with a high degree of confidence. Audit level of assurance is obtained by designing procedures so that the risk of an inappropriate conclusion is reduced to a low level through procedures such as inspection, observation, enquiry, confirmation, computation, analysis and discussion.

For a review level of assurance the procedures to reduce risk to a moderate level are normally limited to enquiry, analysis and discussion. When the Office conducts any work at a review or moderate level of assurance, the team would provide a lower level of assurance by indicating that, based on the procedures performed, nothing has come to their attention that causes them to believe the subject matter does not conform in all significant respects with the criteria.

The level of assurance for audits is indicated in the "About the Audit" section of the audit report chapter by stating an audit has been conducted that provides a high level of assurance on the observations and conclusion. In a like manner, the level of assurance for a review would be indicated in the "About the Review" section of the review report by stating a review has been performed that provides a moderate level of assurance on the observations and conclusion.

Description of the program or activity

5.8 To provide context and background material to allow the reader a sufficient perspective on the audited activity to understand the issues.

Management's responsibilities

5.9 To advise on management's responsibility for performance and results in the audited area.

Criteria

5.10 To point out the basis of measuring performance and the source of the criteria as well as any disagreement with management on the suitability of the criteria chosen.

Performance observations

5.11 To report the extent to which performance satisfied the criteria, and to present sufficient, relevant and appropriate analysis and information to ensure an understanding of the issue. The observations point out the significance of the issue by describing its impact on the quality of performance or by quantifying the problem. They also point out, wherever possible, the effect on results. The issue is to be presented in a convincing but fair way. The underlying cause of the problem is described and visual aids are incorporated, wherever possible, to illustrate the nature of the problem.

Recommendations

5.12 To point to the direction in which positive changes can be made when there are most serious deficiencies.

Management comments

5.13 To include the pertinent views of management on the report observations, conclusions and recommendations and to point out what actions are being taken to correct the problems. Any disagreements are to be noted.

Conclusions

5.14 To point out the assessment of performance against each audit objective, which may include qualifications.

Reporting non-compliance with authorities

5.14.1 As noted in 5.6, certain audits may be designed to express an opinion on specified authorities. In other cases, an audit may report only instances of non-compliance with authorities that the audit team observed in the course of discharging their audit responsibilities. The instances of non-compliance may be reported to Parliament in an entity or government-wide report chapter, or in an audit note. In either case, when reporting instances of non-compliance with authorities, there are specific requirements that the audit team should include in their report:

1. describe the context in which the instances of non-compliance with authorities are being reported by:
   
   
• setting out the requirements of the audit mandate as set out in the audit scope;
• identifying the entity or portion thereof being reported on;
• describing the approach followed by the auditor in selecting matters to be audited; and
• stating that the audit of the matters reported was performed in accordance with appropriate professional standards;
  
  
2. caution the reader against drawing conclusions as to compliance or non-compliance with respect to matters not reported; and
   
   
3. for each reported instance of non-compliance:
   
   
• describe the matter being reported together with, if relevant and practicable, the monetary effect;
• specify the authority or authorities not complied with; and
• state that, in the auditor's opinion, the matter was not in compliance with the authority or authorities specified.

A high-quality report on time

5.15 From the start, the audit was designed to produce a high-quality report on time. The examination plan identified the timing of the audit, key milestones and control points. It was developed to allow the quality of the key audit decisions and progress of the audit to be monitored throughout the audit. The following steps occur in the process of finalizing the audit chapter:

Clearance of field work

5.16 The audit team initiates clearance of audit facts and results of tests.

Internal draft

5.17 The draft chapter is prepared on completion of the fieldwork. It is used to:

• obtain views of the audit advisory committee, Assistant Auditor General (AAG)/Commissioner of the Environment and Sustainable Development (CESD) and Quality Reviewer on the significance and ordering of the issues and whether the report message "hits the mark";
• start the edit, translation, textual and presentation review by the Reports Group; and
• obtain approval of Legal Services for initiating report clearance with entity staff.

The vehicles used for this purpose may be fact sheets, point form report, working papers and the initial draft. The purpose is to confirm the facts, obtain management's reactions to the observations and views on corrective actions, and to ensure that the report contains no surprises. (October 2004)

The Principal's draft

5.18 The Principal's draft is to be as close to the final chapter or audit note as possible. It is used to obtain:

• AG's comments;
• AAG/CESD approval;
• additional challenge, advice and counsel from the audit advisory committee or audit notes committee;
• Quality Reviewer's review, and FRLs', PL-Performance Audit, other PLs (i.e. for audit notes resulting from work on other Office products) and SMEs, as appropriate, and regional principals' review(s) and approval of matters within their area of interest;
• approval from Legal Services with respect to mandate and third party;
• substantive review and edit from the Reports group;
• third party clearance, where applicable (see below); and
• entity comments.

The audit team will provide the Principal's draft for the purpose of clearance in the official language(s) requested by the entity in accordance with the OAG Policy on Bilingual Drafts of Performance Audit Report Chapters. (October 2004)

Chapter Main Points

5.19 All chapters contain a Main Points section that summarizes the key messages from the chapter. The Main Points should:

• serve as a stand-alone summary for the benefit of parliamentarians and other busy readers who may not read the entire report;
• highlight the most important information about an audit in a way that is accurate, clear, coherent, and concise;
• be stated as clearly in the PX Draft as they will be in the media release; and
• allow the reader to easily grasp what we examined, why it's important and what we found. (October 2005)

5.20 The Main Points are divided into four parts:

• What we examined to summarize in a few clear sentences what the audit looked at. In certain exceptional circumstances where there might be confusion, this section should also explain what we did NOT look at.
• Why it's important to make the case for the relevance and significance of the audit. Readers should be told why they should care about how well the government is doing, and why they should care about the overall topic.
• What we found to state the most significant findings of the audit, which should rarely exceed four bullet points. Bullets are not preceded by a general, umbrella finding—each bullet sets out a specific finding. N.B. This should be expressed in terms of the facts we found, not as something that needs to happen or what the entity could be doing better.
• The auditee has responded. A brief reference to the entity's acceptance of recommendations and any action planned or under way. Any area where the Department disagrees is indicated here (the topic, not the nature of the disagreement). (October 2005)

Further guidance on chapter Main Points can be obtained from the FRL Communications and Guidelines issued by the Office.

5.21 Main Points are an integral part of the chapters, and are submitted to the entity at the PX Draft stage, with the body of the chapter. Its maximum length is 2 pages. While the Main Points are discussed with the Department, we retain the right to phrase them as we see fit.

5.22 The writing of the Main Points benefits from the writing of the press release: experience has demonstrated that in itself, the process of telling the story for the press release exposes any ambiguity, and forces the authors to think clearly and to use simple words.

Transmission draft

5.23 After the Principal's draft has dealt with entity comments and is signed off by the designated reviewers, it is submitted to the Deputy Minister/Head of the audited organization as a Transmission draft chapter or audit note.

The Transmission draft will be provided to the entity in both official languages simultaneously or within a week of each other in accordance with the OAG Policy on Bilingual Drafts of Performance Audit Report Chapters.

It is the audit Principal's responsibility to ensure that the Quality Reviewer is consulted on a timely basis and receives the information needed to perform his/her review. The work of the Quality Reviewer should always be completed before the Transmission draft chapter or audit note is forwarded to the entity.

The transmission draft is used to:

• obtain Deputy Minister/Head of the audited organization comments, planned corrective actions and any disagreements;
• incorporate departmental comments in the chapter or audit note (may not apply to all audit notes); and
• obtain sign-off from the Principal, Quality Reviewer and AAG/CESD on quality.

5.24 The Performance Audit Management Committee is kept apprised of progress and potential quality problems throughout the audit by the Principal and the Report Tracker. The audit teams are responsible for preparing report chapter substantiation binders for use in verifying information in the report chapters. The binder can be used by the Reports Group to check numbers, tables, organizational information, terminology and other data contained in the chapter.

Third party clearance

5.25 "Third party" is defined as any organization or person outside of the department or agency that is the subject of the audit report. For greater certainty, this includes other government departments, central agencies, Crown corporations, suppliers or beneficiaries of government programs, or any other organization, individual person or group of persons mentioned in the report. The Office owes third parties a duty of care to ensure accuracy and fairness of references. Further guidance is provided under Third Party References in Chapter 8.

Confidentiality and security

5.26 All material related to audit reports should be kept confidential and secure in accordance with the Office's Security Policy. Draft chapters and other material used to disclose audit findings to departments or agencies should be marked "Protected Draft for discussion purposes only" and clearly indicate that they are the property of the Office of the Auditor General. Further guidance is provided under Security of Information in Chapter 8.